Object detection device and method

Abstract

This object detection device detects an object from each of a prescribed number of surfaces for each frame from a divided image obtained by dividing an image of individual frames included in a video into a plurality of images and a surface corresponding to individual reduced images obtained by reducing the entire image, synthesizes object detection results for each of the prescribed number of surfaces, and determines a prescribed number on the basis of an index related to the difference between the input frame rate of the video and the frame rate of the object detection process.

Description

Object detection device and method

[0001] The disclosed technology relates to an object detection device and an object detection method.

[0002] An object detection device is a device that detects objects contained in an input image from the image and detects metadata including location information such as a rectangular frame indicating the area of ​​the object, attributes indicating the type of object (such as a person or vehicle), and the reliability of the detection result.

[0003] In recent years, object detection using machine learning models trained by deep learning has been performed, and its use in AI image processing in surveillance cameras and edge computing has been considered. Examples of such machine learning models include YOLO (You Only Look Once) and SSD (Single Shot Multibox Detector) (Non-Patent Documents 1 and 2).

[0004] In these deep learning-based object detection models, there are restrictions on the size of the image input to the machine learning model. For example, in the standard model of YOLOv3 (Non-Patent Document 1), the input size is either 320 (width) × 320 (height) pixels, 416 × 416 pixels, or 608 × 608 pixels.

[0005] A method has been disclosed for enabling deep learning-based object detection for high-resolution images such as full HD and 4K under the constraints of the input size to a machine learning model as described above (Non-Patent Document 3). Specifically, this method involves performing overall processing in which an input image is reduced to an input size that satisfies the constraints and object detection is performed on the entire image, and segmentation processing in which the input image is divided and object detection is performed on each divided image, and then combining the results of the overall processing and the segmentation processing to obtain a final detection result.

[0006] Joseph Redmon et.al, "YOLOv3: An Incremental Improvement", arXiv:1804.02767v1 [cs.CV] 8 Apr 2018.Wei Liu et.al, "SSD: Single Shot MultiBox Detector", arXiv:1512.02325v5 [cs.CV] 29 Dec 2016.H.Uzawa et.al, "High-definition object detection technology based on AI inference scheme and its implementation", IEICE Electronics Express, 2021, Volume 18, Issue 22, Pages 20210323.

[0007] In the method of Non-Patent Document 3, when ultra-high definition video such as 4K (3840 × 2160 pixels) is input, the number of divided images increases, resulting in a problem of a huge amount of calculation required for the division process. For example, in the case of YOLOv3, when the input size is restricted to 608 × 608 pixels, the maximum number of divided images is 28, resulting in a huge amount of calculation required for the division process of detecting objects from each divided image.

[0008] One possible method for reducing the amount of calculation required for object detection is to narrow down (thin out) the number of faces on which object detection is performed for the same frame (faces corresponding to reduced images and faces corresponding to divided images). It is possible to determine the number of faces on which object detection is performed per frame depending on the frame rate of the object detection process, but because the frame rate of the object detection process can fluctuate, it is not uniquely determined, making it difficult to determine an appropriate number of faces. If the number of faces on which object detection is performed is too small, the accuracy of object detection decreases, and if the number of faces on which object detection is performed is too large, the amount of calculation increases, so it is desirable to determine an appropriate number of faces.

[0009] The disclosed technology has been developed in consideration of the above points, and aims to appropriately narrow down the number of surfaces on which object detection is performed when there are multiple target surfaces on which object detection is performed per frame.

[0010] A first aspect of the present disclosure is an object detection device that includes a detection unit that detects an object from each of a predetermined number of faces for each frame, among faces corresponding to each of divided images obtained by dividing an image of each frame included in a video into multiple parts and reduced images obtained by reducing the entire image; a synthesis unit that synthesizes the object detection results for each of the predetermined number of faces; and a determination unit that determines the predetermined number based on an index related to the difference between the input frame rate of the video and the frame rate of the object detection process in the detection unit.

[0011] A second aspect of the present disclosure is an object detection method executed by an object detection device including a detection unit, a synthesis unit, and a determination unit, in which the detection unit detects an object from each of a predetermined number of faces for each frame among faces corresponding to each of divided images obtained by dividing an image of each frame included in a video into multiple parts and reduced images obtained by reducing the entire image, the synthesis unit synthesizes the object detection results for each of the predetermined number of faces, and the determination unit determines the predetermined number based on an index related to the difference between the input frame rate of the video and the frame rate of the object detection process in the detection unit.

[0012] According to the disclosed technology, when there are multiple target surfaces for which object detection is to be performed per frame, it is possible to appropriately narrow down the number of surfaces for which object detection is to be performed.

[0013] FIG. 1 is a diagram for explaining object detection in this embodiment. FIG. 2 is a block diagram showing the hardware configuration of an object detection device. FIG. 3 is a block diagram showing an example of the functional configuration of an object detection device. FIG. 4 is a control diagram for determining the total number of faces. FIG. 5 is a diagram for explaining the determination of the number of narrowed-down faces for each frame in the first embodiment. FIG. 6 is a flowchart showing the flow of object detection processing. FIG. 7 is a flowchart showing the flow of determination processing. FIG. 8 is a diagram for explaining the determination of the number of narrowed-down faces for each frame in the second embodiment.

[0014] An example of an embodiment of the disclosed technology will be described below with reference to the drawings. Note that the same or equivalent components and parts in each drawing are given the same reference numerals. Also, the dimensional proportions in the drawings are exaggerated for the convenience of explanation and may differ from the actual proportions.

[0015] <Outline of the Present Disclosure> Before describing each embodiment in detail, an outline of the present disclosure will be described with reference to Fig. 1. An object detection device according to the present disclosure performs a segmentation process in which an input image of one frame of video is segmented into a plurality of images and object detection is performed for each segmented image, and an overall process in which object detection is performed for a reduced image obtained by reducing the entire input image.

[0016] In the overall processing, the object detection device performs image scaling processing to reduce an input image to an image size that can be input into an object detection model prepared in advance for the reduced image, generating a reduced image, and inputting the reduced image into the object detection model. The object detection device then performs object detection processing using the object detection model, detects an object contained in the reduced image, and obtains a detection result. The detection result includes information identifying an area indicating the detected object, information indicating the object's attributes, and the reliability of the detection result. In the example of FIG. 1 , the information identifying the area indicating the object is, for example, position information of the object's circumscribing rectangle (hereinafter referred to as a "detection frame"). The position information may be, for example, a predetermined point (e.g., a center point, a point in the upper left corner, etc.) and size (width and height) of the detection frame. The information indicating the object's attributes is information indicating the object's type, etc., such as "dog" or "bicycle" in the example of FIG. 1 . Because these are detection results for the reduced image, the object detection device scales the detection frame (adjusts metadata) to map it to the input image before reduction. From the group of detected objects after metadata adjustment, objects with reliability equal to or greater than a preset threshold are selected and output to subsequent synthesis processing.

[0017] In a segmentation process, an object detection device generates segmented images by dividing an input image into image sizes that can be input into an object detection model prepared in advance for the segmented images, and inputs each segmented image to the object detection model. The object detection device then performs an object detection process using the object detection model to detect objects contained in each segmented image and obtain detection results. Because the detection results obtained here are detection results for the segmented images, the object detection device adjusts metadata to map them to the input image before segmentation. In addition, in the segmentation process, the object detection device performs screening on the detection results for each segmented image, retaining only one detection result of the same object that has been detected multiple times and removing the rest. After the metadata adjustment and screening, the object detection device selects objects with a reliability above a threshold, as in the overall processing, and outputs the objects to the subsequent synthesis processing.

[0018] In the synthesis process, the object detection device outputs a final detection result obtained by synthesizing the detection result obtained by the division process and the detection result obtained by the overall process. For example, in the synthesis process, the object detection device interpolates objects not detected in the overall process with objects detected in the division process. In selecting objects to be interpolated, a match determination is made between the objects detected in the overall process and the objects detected in the division process, and objects detected in the division process that do not match the objects detected in the overall process are output as objects to be interpolated.

[0019] In this way, object detection is performed from a reduced image obtained by reducing the entire input image and each divided image obtained by dividing the input image, so both large and small objects contained in the input image can be detected at once, even for high-definition video that exceeds the input size constraints of the object detection model.

[0020] As described above, when ultra-high definition video such as 4K is input, the number of divided images increases, posing a problem of enormous computational complexity in the division process. One possible method for reducing the computational complexity of object detection is to narrow down (thin out) the surfaces on which object detection is performed for the same frame. Note that the term "surface" here refers to an entire surface corresponding to a reduced image obtained by reducing the entire input image, and a partial surface corresponding to each divided image obtained by dividing the input image. For example, in the case of an input image with 28 divided images as described above, the number of surfaces per frame is 29.

[0021] The number of planes on which object detection is performed for the same frame (hereinafter referred to as the "number of narrowed-down planes M") is, for example, (N×R ｏｂｊｅｃｔ ) / R ｉｎｐｕｔ where R ｉｎｐｕｔ is the input frame rate of the video, R ｏｂｊｅｃｔ is the frame rate of object detection processing for one screen by a computing core, and N is the number of computing cores. ｏｂｊｅｃｔ represents the maximum frame rate of the object detection process. ｏｂｊｅｃｔ ) / M is the frame rate of the object detection process for one frame.

[0022] However, R ｏｂｊｅｃｔ Since M may fluctuate, it cannot be uniquely determined, making it difficult to appropriately narrow down the number of faces on which object detection is performed. In other words, it is difficult to appropriately determine the value of M in advance.

[0023] Therefore, in the following embodiments, the input frame rate of the video (R ｉｎｐｕｔ ) and the frame rate of the object detection process ((N × R ｏｂｊｅｃｔ The value of the number of narrowing down surfaces M is dynamically adjusted in accordance with the speed difference between (N × R ｏｂｊｅｃｔ ) / M < R ｉｎｐｕｔ In the case of (N×R), the next frame will come in before the execution of object detection on the M planes is completed, and the number of frames waiting for execution of object detection will increase. In order to reduce the number of frames waiting for execution, the value of M is reduced. On the other hand, in the case of (N×R), the next frame will come in before the execution of object detection on the M planes is completed, and the number of frames waiting for execution of object detection will increase. ｏｂｊｅｃｔ ) / M≧R ｉｎｐｕｔ In this case, the value of M is maintained or increased. ｏｂｊｅｃｔEven if the value of M fluctuates, it is automatically adjusted to be optimal accordingly. Each embodiment will be described in detail below.

[0024] <First embodiment> Fig. 2 is a block diagram showing the hardware configuration of an object detection device 10. As shown in Fig. 2, the object detection device 10 includes a central processing unit (CPU) 11, a read-only memory (ROM) 12, a random access memory (RAM) 13, a storage 14, an input unit 15, a display unit 16, and a communication interface (I / F) 17. Each component is connected to each other via a bus 19 so as to be able to communicate with each other.

[0025] The CPU 11 is a central processing unit that executes various programs and controls each part. That is, the CPU 11 reads a program from the ROM 12 or the storage 14 and executes the program using the RAM 13 as a work area. The CPU 11 controls each of the above components and performs various arithmetic processing in accordance with the program stored in the ROM 12 or the storage 14. In this embodiment, the ROM 12 or the storage 14 stores an object detection program, which will be described later.

[0026] The ROM 12 stores various programs and various data. The RAM 13 temporarily stores programs or data as a working area. The storage 14 is configured by a storage device such as an HDD (Hard Disk Drive) or an SSD (Solid State Drive), and stores various programs including an operating system and various data.

[0027] The input unit 15 includes a pointing device such as a mouse and a keyboard, and is used to input various types of information. The display unit 16 is, for example, a liquid crystal display, and displays various types of information. The display unit 16 may also function as the input unit 15 by employing a touch panel system.

[0028] The communication I / F 17 is an interface for communicating with other devices, and the communication may use a wired communication standard such as Ethernet (registered trademark) or FDDI, or a wireless communication standard such as 4G, 5G, or Wi-Fi (registered trademark).

[0029] Next, the functional configuration of the object detection device 10 will be described. FIG. 3 is a block diagram showing an example of the functional configuration of the object detection device 10. As shown in FIG. 3, the object detection device 10 includes, as its functional components, a division unit 21, a reduction unit 22, a first buffer 23, a determination unit 24, a first selection unit 25, a detection unit 26, a second buffer 27, a result holding unit 28, and a second selection unit 29. The object detection device 10 also includes a motion estimation unit 30, a third buffer 31, a correction unit 32, and a synthesis unit 33. Each functional component is realized when the CPU 11 reads out an object detection program stored in the ROM 12 or the storage 14, expands it in the RAM 13, and executes it.

[0030] The division unit 21 acquires each input image corresponding to each frame included in a video input to the object detection device 10. The division unit 21 divides the acquired input image into a plurality of divided images having a size according to the input size of an object detection model used by the detection unit 26 (described later). The division unit 21 assigns consecutive numbers (hereinafter referred to as "partial surface numbers") to the plurality of partial surfaces corresponding to the plurality of divided images in order, for example, starting from the partial surface corresponding to the upper left of the frame, and stores the partial surfaces in the first buffer 23.

[0031] The reduction unit 22 acquires each input image corresponding to each frame included in the video input to the object detection device 10. The reduction unit 22 reduces the acquired input image to a reduced image of a size corresponding to the input size of an object detection model used by the detection unit 26 (described later). The reduction unit 22 accumulates the entire surface corresponding to the reduced image in the first buffer 23.

[0032] That is, frames awaiting execution of object detection are accumulated in the first buffer 23. As will be described in detail later, the first buffer 23, the second buffer 27, and the third buffer 31 are stored at an input frame rate (R ｉｎｐｕｔ ) and the frame rate ((N×R ｏｂｊｅｃｔ) / M) is a buffer for speed adjustment.

[0033] The determination unit 24 determines the number of planes for performing object detection for the same frame in the detection unit 26, i.e., the number of narrowed-down planes M, based on an index related to the difference between the input frame rate of the video and the frame rate of the object detection process in the detection unit 26.

[0034] Specifically, when the index indicates that the input frame rate is higher than the frame rate of the object detection process, the determination unit 24 determines M to be a number that is reduced from the currently set M. Furthermore, when the index indicates that the input frame rate is equal to or lower than the frame rate of the object detection process, the determination unit 24 determines to maintain the currently set M, or determines M to be a number that is increased from the currently set M.

[0035] For example, the determination unit 24 sets the index as the difference between the number of frames waiting for the detection unit 26 to execute the object detection process and a predetermined target number, and determines M so that the difference approaches 0. A specific example of how M is determined using the difference as the index will be described below.

[0036] The determination unit 24 executes the process of determining the number of narrowed-down frames M at a cycle of J frames (hereinafter referred to as a "determination cycle"), where J is a predetermined integer equal to or greater than 1. The determination unit 24 determines the number of frames B waiting to be executed stored in the first buffer 23. ａｃｃｕｍ is a predetermined target value B ｔａｒｇｅｔ To determine M so that

[0037] First, the determination unit 24 determines the total number of faces for which object detection will be performed in the next determination cycle (total number of faces). The total number of faces is the total number of narrowed-down faces M in each frame included in the determination cycle. The determination unit 24 determines the number of frames B waiting for execution. ａｃｃｕｍ is the target value B ｔａｒｇｅｔ The total number of frames is determined based on, for example, PD control so that the number of frames converges to . FIG. 4 shows a control diagram for determining the total number of frames. The determination unit 24 determines the number of frames waiting for execution B stored in the first buffer 23 at the end of the current determination cycle. ａｃｃｕｍ and obtain the target value Bｔａｒｇｅｔ The difference value e (e = B ｔａｒｇｅｔ -B ａｃｃｕｍ ). The determination unit 24 adds a proportional value P obtained by multiplying the calculated difference value e by a constant Kp to a differential value D obtained by multiplying the difference value between the difference value e in the previous determination cycle (e_prev) and the difference value e in the current determination cycle by a constant Kd. If the evaluation value SUM_INT obtained by converting the sum of the obtained proportional value P and the differential value D into an integer value is 0, the determination unit 24 maintains the currently set total number of faces. On the other hand, if the evaluation value is not 0, the determination unit 24 sets the total number of faces in the next determination cycle to the greater of the value obtained by adding the evaluation value SUM_INT to the currently set total number of faces or a predetermined lower limit value for the total number of faces.

[0038] Note that Kp is an example of a "first constant" in the present disclosure, and the proportional value P is an example of a "first value" in the present disclosure. Also, Kd is an example of a "second constant" in the present disclosure, and D is an example of a "second value" in the present disclosure. Also, the value obtained by adding the evaluation value SUM_INT to the currently set total number of faces is an example of a "third value" in the present disclosure.

[0039] Furthermore, the determination unit 24 determines the number of frames M to be narrowed down for each frame in the next determination cycle based on the determined total number of frames. ｉ (i=0, 1, 2, ..., J-1). Specifically, as shown in A of Fig. 5, the determination unit 24 evenly distributes the total number of faces (in the example of Fig. 5, the total number of faces = 10) to the frames in the next determination cycle (four frames in the example of Fig. 5). At this stage, two faces are distributed to each frame, and the remaining number is two faces. As shown in B of Fig. 5, the determination unit 24 sweeps the frame numbers and adds the remaining number one by one to the number of faces to be distributed to the corresponding frame. As shown in C of Fig. 5, the determination unit 24 calculates the number of faces M distributed to each frame when the remaining number becomes zero. ｉ is determined as the number M of narrowing-down planes to be executed for the i-th frame in the next determination cycle. In the example of FIG. 5, as shown by the dashed line, the number M of narrowing-down planes ｉ However, in the next determination cycle, it is determined that there will be three faces in frame #0, three faces in frame #1, two faces in frame #2, and two faces in frame #3.

[0040] The determination unit 24 also determines which partial surface object detection is to be performed at what timing, i.e., in which frame of the next determination cycle. ｉ One of the faces is assigned to the entire face, and the remaining M ｉ -1 plane is assigned to the partial plane. The determination unit 24 determines which partial plane of each frame is to be used to perform object detection based on, for example, the absolute value of the difference between the average value of the number of object detections in past determination cycles and the number of object detections in the current determination cycle. Specifically, the determination unit 24 calculates the absolute value of the difference in the number of object detections for each partial plane. The determination unit 24 allocates (total number of planes - number of frames in the determination cycle) to each partial plane so that the ratio is in accordance with the calculated absolute value of the difference for each partial plane. Then, the determination unit 24 narrows down the number of narrowed-down planes M so that the allocated number of partial planes are selected in the next determination cycle. ｉ By allocating the range of the pixel values ​​to each frame, the partial surface on which object detection is performed in the next determination cycle may be determined.

[0041] The method of determining the partial surface for executing object detection is not limited to the above example. For example, the determination unit 24 may randomly determine M ｉ Alternatively, for example, the determination unit 24 may select a partial surface of the −1 surface and determine it as a partial surface for executing object detection. ｉ A sub-plane of the −1 plane may be selected and determined as the sub-plane on which object detection is performed.

[0042] The decision unit 24 notifies the first selection unit 25 and the second selection unit 29 of decision information that associates the frame number of the i-th frame in the next decision cycle with the partial surface number of the partial surface on which object detection is performed in that frame.

[0043] Based on the determination information notified by the determination unit 24, the first selection unit 25 reads out from the first buffer 23 the whole surface and the partial surface on which object detection is to be performed, and outputs them to the detection unit 26. Furthermore, the first selection unit 25 deletes from the first buffer 23 the partial surface that was not selected as the partial surface on which object detection is to be performed in each frame, i.e., the partial surface that was thinned out in each frame.

[0044] The detection unit 26 detects an object from each of the overall surface and the partial surface output from the first selection unit 25 using an object detection model, and stores the detection result in the second buffer 27. The object detection model is, for example, a machine learning model such as YOLO.

[0045] Here, object detection by the detection unit 26 is performed in parallel by N processing cores. That is, object detection for N planes is performed in parallel. Therefore, the N planes are allowed to include a mixture of whole planes and partial planes of a plurality of frames. For example, when N=4, M ｉ = 2, and M ｉ＋１ If = 2, then M ｉ +M ｉ＋１ = 4, M ｉ and M to be executed for the i+1th frame. ｉ＋１ The above two operations are executed in parallel using N processing cores.

[0046] The second buffer 27 divides the accumulated detection results into detection results for each frame and outputs them to the synthesis unit 33 in order, and also notifies the result holding unit 28 of the detection results.

[0047] The result holding unit 28 holds the detection results for each partial surface of each frame in the previous determination cycle (hereinafter referred to as the "previous detection results"). When the result holding unit 28 is notified of the detection results for partial surfaces by the detection unit 26 from the second buffer 27, it updates the detection results it has held for the partial surface of the corresponding frame in the determination cycle with the notified detection results. Furthermore, when the result holding unit 28 is notified of detection results corrected based on motion vectors (details will be described later) for thinned-out partial surfaces from the correction unit 32 (described later), it updates the detection results it has held for the partial surface of the corresponding frame in the determination cycle with the notified detection results.

[0048] Based on the determination information notified by the determination unit 24 , the second selection unit 29 reads out the detection results of the previous partial surface that has been thinned out in each frame from the result storage unit 28 and outputs them to the correction unit 32 .

[0049] The motion estimation unit 30 estimates a motion vector for each frame from the difference between frames of the input image. Since a conventional method may be applied as a motion vector estimation method, detailed description thereof will be omitted here. The motion estimation unit 30 stores the estimated motion vector for each frame in a third buffer 31.

[0050] The third buffer 31 outputs the accumulated motion vectors to the correction unit 32 frame by frame.

[0051] The correction unit 32 corrects the position coordinates of the detection frame in the previous detection result output from the second selection unit 29 based on the motion vector output from the third buffer 31, and substitutes the corrected position coordinates as the detection result for the thinned-out partial surface. In other words, for the thinned-out partial surface, instead of object detection using the object detection model by the detection unit 26, the correction unit 32 corrects the detection result stored in the result storage unit 28 based on the motion vector and uses the result as the detection result for the corresponding partial surface of the corresponding frame in the current determination cycle. The correction unit 32 outputs the detection result for the thinned-out partial surface to the synthesis unit 33, and notifies the result storage unit 28 of the detection result.

[0052] The synthesis unit 33 synthesizes the detection results based on the object detection model for the entire surface and the selected partial surface output from the second buffer 27 and the detection results corrected with the motion vector for the thinned partial surface output from the correction unit 32, to obtain a final detection result. The synthesis unit 33 outputs the final detection result.

[0053] Next, a description will be given of the operation of the object detection device 10 according to the first embodiment. Fig. 6 is a flowchart showing the flow of the object detection process performed by the object detection device 10. The object detection process is performed by the CPU 11 reading an object detection program from the ROM 12 or the storage 14, expanding it into the RAM 13, and executing it.

[0054] In step S10, the CPU 11, functioning as the determination unit 24, determines whether it is time to execute a process for determining the number of narrowed-down surfaces M for the next determination cycle. If it is time to execute the process, the process proceeds to step S20, where the determination process is executed. On the other hand, if it is not time to execute the process, the process proceeds to step S30.

[0055] The determination process will now be described with reference to FIG.

[0056] In step S21, the CPU 11, as the determination unit 24, determines the total number of faces of the narrowed-down face number M for each frame included in the next determination cycle. Next, in step S22, the CPU 11, as the determination unit 24, determines the narrowed-down face number M for each frame in the next determination cycle based on the determined total number of faces. ｉ (i=0, 1, 2, ..., J-1) is determined.

[0057] Next, in step S23, the CPU 11, as the determination unit 24, determines M ｉ One of the faces is assigned to the entire face, and the remaining M ｉ Next, in step S24, the CPU 11, functioning as the determination unit 24, notifies the first selection unit 25 and the second selection unit 29 of determination information that associates the frame number of the i-th frame in the next determination cycle with the partial surface number of the partial surface on which object detection is to be performed in that frame, and then returns to the object detection process ( FIG. 6 ).

[0058] Next, in step S30, the CPU 11, functioning as the dividing unit 21, divides the input image into a plurality of divided images, assigns partial face numbers to a plurality of partial faces corresponding to the plurality of divided images, and stores the partial faces in the first buffer 23. The CPU 11, functioning as the reducing unit 22, reduces the input image into reduced images, and stores the whole faces corresponding to the reduced images in the first buffer 23.

[0059] Next, in step S40, the CPU 11, functioning as the first selection unit 25, reads out from the first buffer 23 the full surface and partial surface on which object detection is to be performed, based on the determination information notified from the determination unit 24, and outputs them to the detection unit 26. Then, the CPU 11, functioning as the detection unit 26, detects objects from each of the full surface and partial surface output from the first selection unit 25 using an object detection model, and stores the detection results in the second buffer 27.

[0060] Next, in step S50, the CPU 11, functioning as the first selection unit 25, deletes, in each frame, partial faces that were not selected as partial faces for performing object detection, i.e., thinned partial faces, from the first buffer 23. Next, in step S60, the CPU 11, functioning as the second buffer 27, divides the accumulated detection results into detection results for each frame and outputs them in order to the synthesis unit 33, and notifies the result holding unit 28 of the detection results. The CPU 11, functioning as the result holding unit 28, updates the detection result of the relevant partial face of the relevant frame, among the previous detection results for each partial face that it holds, with the notified detection result.

[0061] Next, in step S70, the CPU 11, functioning as the second selection unit 29, reads out the previous detection results of the thinned partial surfaces in each frame from the result storage unit 28 and outputs them to the correction unit 32. Also, the CPU 11, functioning as the motion estimation unit 30, estimates motion vectors for each frame and stores them in the third buffer 31. Also, the CPU 11, functioning as the third buffer 31, outputs the stored motion vectors to the correction unit 32 frame by frame. Then, the CPU 11, functioning as the correction unit 32, corrects the position coordinates of the detection frame in the previous detection results output from the second selection unit 29 based on the motion vectors output from the third buffer 31, thereby obtaining detection results of the thinned partial surfaces. Also, the CPU 11, functioning as the correction unit 32, outputs the detection results of the thinned partial surfaces to the synthesis unit 33 and notifies the result storage unit 28 of the detection results. Also, the CPU 11, functioning as the result storage unit 28, updates the detection results of the relevant partial surfaces of the relevant frame, among the previous detection results for each partial surface stored therein, with the notified detection results.

[0062] Next, in step S80, the CPU 11 functions as the synthesis unit 33 to synthesize the detection results output in steps S60 and S70 to obtain and output a final detection result. Next, in step S90, the CPU 11 determines whether or not to end the object detection process. For example, the CPU 11 may determine to end the object detection process when the final frame of the video has been reached, when a command indicating the end of object detection has been input, or the like. If the determination is negative, the process returns to step S10, and if the determination is positive, the object detection process ends.

[0063] As described above, in the object detection device according to the first embodiment, the detection unit detects an object from each of a predetermined number of surfaces for each frame, among surfaces corresponding to each of the divided images obtained by dividing an image of each frame included in a video into multiple parts and each of the reduced images obtained by reducing the entire image. The synthesis unit also synthesizes the object detection results for each of the predetermined number of surfaces. The determination unit then determines the predetermined number based on an index related to the difference between the input frame rate of the video and the frame rate of the object detection process performed by the detection unit. This makes it possible to appropriately narrow down the number of surfaces on which object detection is performed when there are multiple target surfaces for object detection per frame.

[0064] Second Embodiment Next, a second embodiment will be described.

[0065] The hardware configuration of the object detection device 210 according to the second embodiment is similar to the hardware configuration of the object detection device 10 according to the first embodiment shown in FIG.

[0066] Next, a description will be given of the functional configuration of the object detection device 210. Note that the same functional configurations as those of the object detection device 10 according to the first embodiment will be assigned the same reference numerals and detailed description thereof will be omitted.

[0067] As shown in FIG. 3, the object detection device 210 includes, as a functional configuration, a determination unit 224 instead of the determination unit 24 of the object detection device 10 according to the first embodiment.

[0068] The determination unit 224 performs the same processing as the determination unit 24 of the first embodiment in determining the total number of faces and the partial faces for executing object detection. However, the number of narrowed-down faces M ｉ In the process of determining (i=0, 1, 2, . . . , J−1), the number of narrowing down planes M is fixed and the frame rate of the object detection process is adjusted, which is different from the determination unit 24 in the first embodiment.

[0069] Specifically, the determination unit 224 determines the number of narrowed-down faces M for each frame. ｉ It is allowed that there will be frames in which the number of frames M ｉFor frames where M' is not 0, the number of narrowed-down planes for each frame is determined to be a predetermined fixed value M'. As a result, frames in which object detection using the object detection model is skipped occur, but in frames in which object detection using the object detection model is performed, object detection can be performed on a sufficient number of planes, thereby improving detection performance.

[0070] For example, the determination unit 224 allocates a fixed value M' of face numbers to frames starting from the first frame of the next determination cycle based on the total number of faces. Here, the number of frames to which the number of faces M' can be allocated is obtained by Floor (total number of faces / M'). As shown in A of FIG. 8, the determination unit 224 allocates M' of face numbers to frames up to Floor (total number of faces / M') counting from the start of the next determination cycle. ｉ Furthermore, as shown in B of FIG. 8, the determination unit 224 allocates all the remaining numbers to the next frame, and sets M of the subsequent frames within the determination period. ｉ In the example of FIG. 8, as shown by the dashed line, the number of narrowed-down surfaces M ｉ However, in the next determination cycle, it is determined that there will be three faces in frames #0 to #2, and one face in frame #3.

[0071] M ｉ For frames for which .times. ...

[0072] Regarding the operation of the object detection device 210 according to the second embodiment, in step S22 of the determination process in FIG. 7, the determination unit 224 determines M using a fixed value M′. ｉ Since this is the same as the first embodiment except for the determination of

[0073] As described above, in the object detection device of the second embodiment, as in the object detection device of the first embodiment, when there are multiple target surfaces for which object detection is to be performed per frame, the number of surfaces for which object detection is to be performed can be appropriately narrowed down.

[0074] In the above embodiments, when determining the total number of narrowed-down faces in the next determination cycle, B ａｃｃｕｍ In consideration of processing time, the case where B is acquired a few frames before the end is described. ａｃｃｕｍ may be acquired to determine the total number of faces.

[0075] In addition, in each of the above embodiments, M ｉ One of the faces is assigned to the entire face, and the remaining M ｉ Although the case where the −1 surface is assigned to a partial surface has been described, this is not limiting. ｉ Alternatively, the number of faces may be selected. In this case, the result storage unit stores not only the previous detection result for each partial face, but also the previous detection result for the entire face. Then, as with the partial faces, if the entire face is thinned out from the targets of the object detection process using the object detection model, the previous detection result for the entire face is substituted with a detection result corrected by a motion vector.

[0076] In addition, in the above-described embodiments, the number of narrowed-down planes M is determined based on the difference between the number of frames waiting to be executed and the target value, but this is not limiting. For example, the input frame rate and the object detection frame rate in the current determination cycle may be measured, and the number of narrowed-down planes M may be determined based on the difference between the measured values.

[0077] In addition, the object detection process executed by the CPU in each of the above embodiments by reading the software (program) may be executed by various processors other than the CPU. Examples of processors in this case include PLDs (Programmable Logic Devices) whose circuit configuration can be changed after manufacture, such as FPGAs (Field-Programmable Gate Arrays), and dedicated electrical circuits, which are processors having a circuit configuration designed specifically for executing specific processes, such as ASICs (Application Specific Integrated Circuits). The object detection process may be executed by one of these various processors, or by a combination of two or more processors of the same or different types (e.g., multiple FPGAs, a combination of a CPU and an FPGA, etc.). The hardware structure of these various processors is, more specifically, an electrical circuit that combines circuit elements such as semiconductor elements.

[0078] In addition, in each of the above embodiments, the object detection program is pre-stored (installed) in the storage 14, but this is not limiting. The program may be provided in a form stored on a non-transitory storage medium such as a CD-ROM (Compact Disk Read Only Memory), a DVD-ROM (Digital Versatile Disk Read Only Memory), or a USB (Universal Serial Bus) memory. The program may also be downloaded from an external device via a network.

[0079] The following additional notes are provided regarding the above-described embodiments.

[0080] (Supplementary Item 1) An object detection device including: a detection unit that detects an object from each of a predetermined number of faces for each frame among faces corresponding to divided images obtained by dividing an image of each frame included in a video into a plurality of parts and reduced images obtained by reducing the entire image; a synthesis unit that synthesizes the object detection results for each of the predetermined number of faces; and a determination unit that determines the predetermined number based on an index related to the difference between the input frame rate of the video and the frame rate of the object detection process in the detection unit.

[0081] (Supplementary Item 2) The object detection device described in Supplementary Item 1, wherein the determination unit, when the indicator indicates that the input frame rate is higher than the frame rate of the object detection process, determines the predetermined number to be a number that is reduced from the currently set predetermined number, and when the indicator indicates that the input frame rate is equal to or lower than the frame rate of the object detection process, determines to maintain the currently set predetermined number, or determines to increase the predetermined number from the currently set predetermined number.

[0082] (Supplementary Item 3) The object detection device according to Supplementary Item 1 or Supplementary Item 2, wherein the determination unit sets the index as a difference value between the number of frames waiting for the detection unit to execute the object detection process and a predetermined target number, and determines the predetermined number so that the difference value approaches 0.

[0083] (Supplementary Item 4) The object detection device according to Supplementary Item 3, wherein the determination unit determines the total number of the predetermined numbers in frames included in a cycle of a predetermined number of frames for which the determination of the predetermined number is performed by maintaining the currently set total number of the predetermined numbers when an evaluation value obtained by converting an added value obtained by multiplying the difference value by a predetermined first constant and a second value obtained by multiplying the difference between the difference value in the previous cycle and the difference value in the current cycle by a predetermined second constant is zero; and when the evaluation value is not zero, determines the total number of the predetermined numbers as the larger of either a third value obtained by adding the evaluation value to the currently set integer of the predetermined number or a predetermined lower limit value of the total number of the predetermined numbers.

[0084] (Supplementary Item 5) The object detection device according to Supplementary Item 4, wherein the determination unit determines the predetermined number for each frame by evenly distributing the total number of the predetermined number among the frames included in the period and distributing the remainder one by one starting from the first frame included in the period.

[0085] (Supplementary Item 6) The object detection device according to Supplementary Item 4, wherein the determination unit assigns a predetermined fixed number as the predetermined number to the total number of the predetermined number, starting from the first frame included in the period, and assigns the remainder as the predetermined number for the frame following the frame to which the fixed number was last assigned, and sets the predetermined number for the remaining frames to 0.

[0086] (Supplementary Item 7) The object detection device according to any one of Supplementary Items 1 to 6, wherein the detection result includes information identifying an area representing the object, information indicating attributes of the object, and a reliability of the detection result.

[0087] (Supplementary Item 8) An object detection device according to any one of Supplementary Items 1 to 7, including: a motion estimation unit that estimates a motion vector for each frame based on a difference in images between frames; and a correction unit that obtains an object detection result for a current frame by correcting, for faces not included in the predetermined number of faces in each frame, an object detection result for a previous face based on the motion vector estimated by the motion estimation unit, wherein the synthesis unit synthesizes the object detection result for each of the predetermined number of faces with the object detection result obtained by the correction unit.

[0088] (Supplementary Item 9) An object detection program that causes a computer to function as: a detection unit that detects an object from each of a predetermined number of faces for each frame among faces corresponding to each of divided images obtained by dividing an image of each frame included in a video into multiple parts and reduced images obtained by reducing the entire image; a synthesis unit that synthesizes the object detection results for each of the predetermined number of faces; and a determination unit that determines the predetermined number based on an index related to the difference between the input frame rate of the video and the frame rate of the object detection process in the detection unit.

[0089] (Supplementary Item 10) An object detection device including: a memory; and at least one processor connected to the memory, wherein the processor is configured to: detect an object from each of a predetermined number of faces for each frame among faces corresponding to divided images obtained by dividing an image of each frame included in a video into multiple parts and reduced images obtained by reducing the entire image; combine the object detection results for each of the predetermined number of faces; and determine the predetermined number based on an index related to the difference between the input frame rate of the video and the frame rate of the object detection process.

[0090] (Supplementary Item 11) A non-transitory storage medium storing a program executable by a computer to perform object detection processing, wherein the object detection processing detects an object from each of a predetermined number of faces for each frame among faces corresponding to divided images obtained by dividing an image of each frame included in a video into multiple parts and reduced images obtained by reducing the entire image, combines the object detection results for each of the predetermined number of faces, and determines the predetermined number based on an index related to the difference between the input frame rate of the video and the frame rate of the object detection processing.

[0091] 10, 210 Object detection device 11 CPU 12 ROM 13 RAM 14 Storage 15 Input unit 16 Display unit 17 Communication I / F 19 Bus 21 Dividing unit 22 Reducing unit 23 First buffer 24, 224 Determining unit 25 First selecting unit 26 Detection unit 27 Second buffer 28 Result holding unit 29 Second selecting unit 30 Motion estimating unit 31 Third buffer 32 Correcting unit 33 Combining unit

Claims

1. An object detection device comprising: a detection unit that detects an object from each of a predetermined number of faces for each frame among faces corresponding to each of divided images obtained by dividing an image of each frame included in a video into multiple parts and reduced images obtained by reducing the entire image; a synthesis unit that synthesizes the object detection results for each of the predetermined number of faces; and a determination unit that determines the predetermined number based on an index related to the difference between the input frame rate of the video and the frame rate of the object detection process in the detection unit.

2. The object detection device of claim 1, wherein the determination unit determines the predetermined number to be a number that is reduced from the currently set predetermined number when the indicator indicates that the input frame rate is greater than the frame rate of the object detection process, and determines the predetermined number to be maintained at the currently set predetermined number or determines the predetermined number to be a number that is increased from the currently set predetermined number when the indicator indicates that the input frame rate is less than or equal to the frame rate of the object detection process.

3. An object detection device as described in claim 1 or claim 2, wherein the determination unit sets the index as the difference between the number of frames waiting for the detection unit to execute the object detection process and a predetermined target number, and determines the predetermined number so that the difference value approaches 0.

4. An object detection method executed by an object detection device including a detection unit, a synthesis unit, and a determination unit, wherein the detection unit detects an object from each of a predetermined number of faces for each frame among faces corresponding to each of divided images obtained by dividing an image of each frame included in a video into multiple parts and reduced images obtained by reducing the entire image, the synthesis unit synthesizes the object detection results for each of the predetermined number of faces, and the determination unit determines the predetermined number based on an index related to the difference between the input frame rate of the video and the frame rate of the object detection process in the detection unit.

Images