Object detection device, object detection method, and object detection program

Abstract

Provided is an object detection device that outputs metadata from an image in each frame of a video image, the metadata including at least the center coordinates of an object included in the image and a bounding box around the object. This object detection device decimates a frame on which to perform object detection and determines, according to a predetermined method, the appropriateness of reusing the metadata detected in the past as a detection result for the decimated frame. If reuse is determined to be appropriate, the object detection device translates the center coordinates of the metadata on the basis of a motion vector obtained from the difference with respect to the image of the preceding frame, and treats the result as the detection result for the decimated frame. If reuse is determined to be inappropriate, the object detection device excludes the metadata from reuse.

Description

Object detection device, object detection method, and object detection program

[0001] The disclosed technology relates to an object detection device, an object detection method, and an object detection program.

[0002] An object detection device is a device that detects metadata from an input image, consisting of the position of an object contained in the image (a square frame surrounding the object), attributes (type of object, such as a person or a car), and reliability.

[0003] In recent years, YOLO (You Only Look Once) and SSD (Single Shot multibox Detector) have been disclosed, which detect metadata based on deep learning results (Non-Patent Documents 1 and 2), and their use in AI (Artificial Intelligence) image processing in surveillance cameras and edge computing has been studied.

[0004] These deep learning-based object detection methods have limitations on the input image size. For example, in YOLOv3 (Non-Patent Document 1), the input image size is limited to 320 (width) × 320 (height) pixels, 416 × 416 pixels, or 608 × 608 pixels.

[0005] In order to enable deep learning-based object detection for high-definition images such as full HD (High Definition) and 4K under the above-mentioned image size constraints, a method has been disclosed in which an input image is reduced to the above-mentioned constrained image size, and an overall process is performed to perform object detection on the entire image, and a segmentation process is performed to divide the input image and perform object detection for each divided image, and the overall process result and the segmentation process result are combined to obtain a final object detection result (Non-Patent Document 1).

[0006] Figure 1 shows the processing flow of this method. In the overall processing, an image scaling process is performed to reduce the input image to an image size that can be input to an object detection process (hereinafter referred to as the detection process) based on the deep learning results, and then the detection process is performed to detect objects contained in the reduced image. Of the detected objects, the rectangular frame (hereinafter referred to as BB) indicating the object's position includes at least the center coordinates (X, Y), frame height (H), and frame width (W) of the frame. Since these are for the reduced image, scaling of the BB (metadata scaling) is performed to map it to the original image before reduction. From the group of detected objects after this scaling, objects with a reliability equal to or greater than a preset threshold (hereinafter referred to as the reliability threshold) are selected and output to the subsequent synthesis process.

[0007] In the segmentation process, the original image is segmented, and then the detection process is performed for each segmented image (partial surface) to detect the object contained in each partial surface. Since the BB of the detected object is for the partial surface, metadata adjustment is performed to map it to the original image before segmentation. After this adjustment, as with the overall processing, only objects whose reliability is equal to or greater than the reliability threshold are selected and output to the subsequent synthesis process.

[0008] In the synthesis process, objects not detected in the overall process are interpolated with objects detected in the segmentation process. In selecting the objects to be interpolated, a match determination is made between the objects detected in the overall process and the objects detected in the segmentation process, and objects that do not match are output as objects to be interpolated. In the synthesis process, the output of the metadata selection process for each segmented image, the output of the overall processing unit, and the output of the interpolation process are combined and output as the final object detection result.

[0009] The above method makes it possible to simultaneously detect both large and small objects, even in high-definition video that exceeds the input constraint size of deep learning-based object detection.

[0010] Joseph Redmon et.al, “YOLOv3: An Incremental Improvement”, https: / / arxiv.org / abs / 1804.02767 Wei Liu et.al, “SSD: Single Shot MultiBox Detector”, https: / / arxiv.org / pdf / 1512.02325.pdf H. Uzawa et. al, “High-definition object detection technology based on AI inference scheme and its implementation”, IEICE Electronics Express, 2021, Volume 18, Issue 22, Pages 20210323. Takayuki Ujiie et. al, “Reducing the load of CNN object detection using motion vectors in video codecs”, Vol. 2018-CVIM-210 No. 4 IPSJ Research Report

[0011] In the case of ultra-high definition video such as 4K (3840 x 2160), the number of image divisions (number of partial faces) in the division process is large, which poses a problem of enormous computational complexity. For example, in the case of YOLOv3, when the input constraint size is 608 x 608, the maximum number of divided images is 28, which results in an enormous computational complexity.

[0012] One method for reducing the amount of calculation is to narrow down (thin out) the partial surfaces on which object detection is performed for the same video frame (Non-Patent Document 4). This is a method for interpolating the detection results for the thinned out frames by translating and reusing the center coordinate position of the previous detection result (detection frame) based on the motion vector.

[0013] However, as shown in Figure 2, if the center coordinates overlap with another object during the thinned-out period, the detection frame will undergo the same parallel movement as the object with the larger movement, resulting in a problem of the detection frame being translated to a position where no object is present.

[0014] The disclosed technology has been made in consideration of the above points, and aims to provide an object detection device, an object detection method, and an object detection program that can reduce the amount of calculation and perform object detection with high accuracy.

[0015] A first aspect of the present disclosure is an object detection device that outputs metadata from an image in each frame of a moving image, the metadata including at least the center coordinates of an object included in the image and a frame surrounding the object, the device thins out frames on which object detection is performed, and determines using a predetermined method whether it is appropriate to reuse the metadata detected in the past as the detection result for the thinned frame, and if it is determined to be appropriate, it translates the center coordinates of the metadata based on a motion vector obtained from the difference with the image of the previous frame to use it as the detection result for the thinned frame, and if it is determined to be inappropriate, it excludes the metadata from being reused.

[0016] A second aspect of the present disclosure is an object detection method in an object detection device that outputs metadata including at least the center coordinates of an object included in an image and a frame surrounding the object from an image in each frame of a moving image, thinning out frames on which object detection is performed, and determining, using a predetermined method, the appropriateness of reusing the metadata detected in the past as the detection result for the thinned out frame, and if determined to be appropriate, translating the center coordinates of the metadata based on a motion vector obtained from the difference with the image of the previous frame, and using this as the detection result for the thinned out frame, and if determined to be inappropriate, excluding the metadata from being reused.

[0017] A third aspect of the present disclosure is an object detection program for causing a computer to function as the object detection device of the first aspect.

[0018] According to the disclosed technology, it is possible to reduce the amount of calculation and perform object detection with high accuracy.

[0019] FIG. 1 is a schematic diagram for explaining an overview of the first embodiment. FIG. 1 is a schematic diagram for explaining an overview of the first embodiment. FIG. 2 is a block diagram showing input / output of the object detection device of the first embodiment. FIG. 3 is a block diagram showing an example of the hardware configuration of the object detection devices of the first and second embodiments. FIG. 4 is a block diagram showing the configuration of the object detection device of the first and second embodiments. FIG. 5 is a block diagram showing the configuration of a segmentation processing unit of the object detection device of the first embodiment. FIG. 6 is a block diagram showing the configuration of an interpolation processing unit of the object detection device of the first embodiment. FIG. 7 is a block diagram showing the configuration of an overall processing unit of the object detection device of the first embodiment. FIG. 8 is a flowchart showing the flow of object detection processing of the object detection device of the first embodiment. FIG. 9 is a flowchart showing the flow of segmentation processing of the object detection device of the first embodiment. FIG. 10 is a flowchart showing the flow of interpolation processing of the object detection device of the first embodiment. FIG. 11 is a flowchart showing the flow of interpolation determination processing of the object detection device of the first embodiment. FIG. 12 is a flowchart showing the flow of interpolation processing of the object detection device of the second embodiment. FIG. 13 is a flowchart showing the flow of interpolation determination processing of the object detection device of the second embodiment. FIG. 14 is a flowchart showing the flow of allocation control processing of the object detection device of the second embodiment.

[0020] An example of an embodiment of the disclosed technology will be described below with reference to the drawings. Note that the same or equivalent components and parts in each drawing are given the same reference numerals. Also, the dimensional proportions in the drawings are exaggerated for the convenience of explanation and may differ from the actual proportions.

[0021] [Summary of the Disclosure] With the technology disclosed herein, when translating the center coordinates of a past detection result (detection frame) based on a motion vector, the validity of the translation is determined, and if it is determined to be invalid, the detection frame is deemed unusable and removed from the interpolation targets. This makes it possible to prevent the translation of a detection frame from appearing in a position where no object is present.

[0022] Specifically, the absolute difference between the motion vector corresponding to the center coordinate and the motion vector corresponding to the coordinate within the same detection frame is calculated, and if the calculated value is below a predetermined threshold (hereinafter referred to as the judgment threshold), it is judged to be valid; otherwise, it is judged to be invalid. Here, the coordinates within the detection frame are, for example, four points above, below, left, and right of the center coordinate shown in FIG. 3 , and are the midpoints between the center coordinate and the edge of the detection frame. Because these coordinates are close to the center coordinate, they usually exhibit motion vectors similar to those corresponding to the center coordinate. However, if a part of the detection frame overlaps with another object, the absolute difference between the motion vector corresponding to any of the four points that overlaps with the other object becomes large. Therefore, validity can be judged based on whether the absolute difference between the motion vectors of the midpoint and the center coordinate exceeds the judgment threshold. FIG. 3 illustrates an example in which, in the detection frame [k], the motion vector of point cc[k][3] above the center coordinate is different from the motion vector of center coordinate c[k].

[0023] The object detection device of the present disclosure will be described in detail below.

[0024] [First Embodiment] In the first embodiment, as shown in Fig. 4, an object detection device 10 receives a moving image as input and outputs metadata including at least attributes, reliability, center coordinates, and a frame surrounding the object of an object included in the image from an image in each frame of the moving image. Fig. 5 is a block diagram showing the hardware configuration of the object detection device 10 of this embodiment.

[0025] 5, the object detection device 10 includes a CPU (Central Processing Unit) 11, a ROM (Read Only Memory) 12, a RAM 13, a storage 14, an input unit 15, a display unit 16, and a communication interface (I / F) 17. Each component is connected to each other via a bus 19 so as to be able to communicate with each other.

[0026] The CPU 11 is a central processing unit that executes various programs and controls each component. That is, the CPU 11 reads programs from the ROM 12 or the storage 14 and executes the programs using the RAM 13 as a work area. The CPU 11 controls the above components and performs various arithmetic processing in accordance with the programs stored in the ROM 12 or the storage 14. In this embodiment, an object detection program is stored in the ROM 12 or the storage 14. The object detection program may be a single program or a group of programs composed of multiple programs or modules.

[0027] The ROM 12 stores various programs and various data. The RAM 13 temporarily stores programs or data as a working area. The storage 14 is configured with an HDD (Hard Disk Drive) or an SSD (Solid State Drive) and stores various programs including the operating system and various data.

[0028] The input unit 15 includes a pointing device such as a mouse and a keyboard, and is used to input various types of information.

[0029] The display unit 16 is, for example, a liquid crystal display, and displays various information including the processing results. The display unit 16 may be a touch panel type and function as the input unit 15.

[0030] The communication interface 17 is an interface for communicating with other devices, such as a camera that captures input images and a database server that stores the input images captured by the camera, etc. Standards such as Ethernet (registered trademark), FDDI, and Wi-Fi (registered trademark) are used for this communication.

[0031] Next, a description will be given of the functional configuration of the object detection device 10. Fig. 6 is a block diagram showing an example of the functional configuration of the object detection device 10.

[0032] The object detection device 10 thins out frames on which object detection is performed, and determines, for each piece of metadata within a frame, using a predetermined method, whether it is appropriate to reuse previously detected metadata as the detection result for the thinned frame. If the object detection device 10 determines that the metadata is appropriate, it translates the center coordinates of the metadata based on a motion vector obtained from the difference with the image of the previous frame, and uses the resulting metadata as the detection result for the thinned frame. If the object detection device 10 determines that the metadata is inappropriate, it excludes the metadata determined to be inappropriate from being reused.

[0033] Specifically, as shown in FIG. 6, the object detection device 10 includes a motion estimation unit 100, a division processing unit 110, an interpolation processing unit 120, an allocation control unit 126, an overall processing unit 130, and a synthesis processing unit 140.

[0034] The motion estimation unit 100 calculates a motion vector across the entire image for each frame of the input moving image.

[0035] The division processing unit 110 divides the image of each frame of the video image and performs object detection.

[0036] The interpolation processing unit 120 performs interpolation processing on the partial surfaces determined by the allocation control unit 126 where object detection is not performed by translating the center coordinates of the metadata detected in the past.

[0037] The allocation control unit 126 determines, for each frame image, partial surfaces that are partial images obtained by dividing the image, on which object detection is to be performed.

[0038] The overall processing unit 130 reduces the entire image and performs object detection.

[0039] The synthesis processing unit 140 synthesizes the object detection result of the division processing unit 110, the interpolation processing result of the interpolation processing unit 120, and the object detection result of the overall processing unit 130 to obtain a final object detection result.

[0040] As shown in FIG. 7, the division processing unit 110 includes a division unit 111 , a first selection unit 112 , an object detection unit 113 , and a holding unit 114 .

[0041] The dividing unit 111 divides the image of each frame of the moving image and outputs it as partial planes, which are a plurality of partial images.

[0042] The first selection unit 112 selects a partial surface for object detection for each image of a frame in accordance with the determination by the allocation control unit 126 .

[0043] The object detection unit 113 uses an object detection model, which is a convolutional neural network that takes an image as input and outputs object position information, to perform object detection to determine object position information for each of the selected partial surfaces, and outputs a detection frame representing the object.

[0044] Specifically, the object detection model is a trained model that has been trained in advance to output a detection frame surrounding each object included in an input image.

[0045] The object detection model may be stored in a storage medium such as the storage 14, or may be stored in an external cloud server. As a technique for performing object detection based on such images, known methods can be appropriately applied.

[0046] The storage unit 114 stores the object detection results for each partial surface by the object detection unit 113 for the image of each frame.

[0047] As shown in FIG. 8, the interpolation processing unit 120 includes a second selection unit 121 and an interpolation determination unit 122 .

[0048] The second selection unit 121 selects, for each image of a frame, a partial surface on which object detection is not performed, in accordance with the determination by the allocation control unit 126 .

[0049] The interpolation determination unit 122 determines the appropriateness of reusing previously detected metadata as a detection result for a thinned frame for a partial surface where object detection is not performed, for each piece of metadata within that partial surface of the frame, and if it determines that the metadata is appropriate, it translates the center coordinates of the metadata based on the motion vector to use it as a detection result for the thinned frame.If it determines that the metadata is inappropriate, the interpolation determination unit 122 excludes the metadata determined to be inappropriate from the objects to be reused.

[0050] Specifically, the interpolation determination unit 122 determines that the value obtained by calculating the absolute difference between the motion vector corresponding to the center coordinate of the metadata and the motion vector corresponding to the coordinate within the metadata frame is valid if it is below a predetermined threshold, and determines that the value is invalid otherwise.

[0051] As shown in FIG. 9, the overall processing unit 130 includes a reduction unit 131 and an object detection unit 132 .

[0052] The reduction unit 131 reduces the entire image for each frame of the moving image.

[0053] The object detection unit 132 detects an object from the reduced image of each frame of the video.

[0054] <Operation of the Object Detection Device According to the Present Embodiment> Next, the operation of the object detection device 10 according to the present embodiment will be described.

[0055] 10 is a flowchart showing the flow of object detection by object detection device 10. Object detection is performed by CPU 11 reading an object detection program from ROM 12 or storage 14, expanding it into RAM 13, and executing it. It is also assumed that a moving image is input to object detection device 10. Object detection is an example of an object detection method.

[0056] In step S100, the CPU 11 acquires an image of a frame of the input moving image.

[0057] In step S102, the CPU 11 functions as the motion estimation unit 100 to calculate a motion vector for the entire image of the acquired frame.

[0058] In step S104, the CPU 11 functions as the division processing unit 110 to divide the acquired frame image and perform object detection.

[0059] In step S106, the CPU 11 functions as the interpolation processing unit 120 to perform interpolation processing on the partial surfaces determined by the allocation control unit 126 where object detection is not performed, by translating the center coordinates of the metadata detected in the past.

[0060] In step S108, the CPU 11 functions as the overall processing unit 130 to reduce the entire image of the acquired frame and perform object detection.

[0061] In step S110, the CPU 11 functions as the synthesis processing unit 140 to synthesize the object detection result of the division processing unit 110 and the object detection result of the overall processing unit 130 to obtain a final object detection result.

[0062] The step S104 is realized by a division processing routine shown in FIG.

[0063] In step S120, the CPU 11 functions as the division unit 111 to divide the image of each frame of the moving image and output it as partial planes, which are a plurality of partial images.

[0064] In step S122, the CPU 11 functions as the first selection unit 112 to select a partial surface on which object detection is to be performed for each image of the frame in accordance with the determination by the allocation control unit 126.

[0065] In step S124, the CPU 11 uses an object detection model, which is a convolutional neural network that receives an image as input and outputs object position information, as the object detection unit 113 to perform object detection to determine object position information for each of the selected partial surfaces, and outputs a detection frame representing the object.

[0066] In step S126, the CPU 11 causes the storage unit 114 to store the object detection results for each partial surface by the object detection unit 113 for the image of each frame.

[0067] Step S106 is implemented by an interpolation processing routine shown in FIG.

[0068] In step S130, the CPU 11, as the second selection unit 116, selects a partial surface for which object detection will not be performed for the image of the frame in accordance with the decision made by the allocation control unit 126, and obtains the result of object detection for the image of the previous frame for that partial surface.

[0069] In step S132, the CPU 11, as the interpolation determination unit 122, determines the appropriateness of reusing previously detected metadata as the detection result for the thinned frames for the partial surfaces where object detection is not performed, and performs interpolation processing by parallel movement of the center coordinates of the previously detected metadata.

[0070] Step S132 is implemented by an interpolation determination processing routine shown in Fig. 13. This processing is repeatedly executed for each partial surface in the frame in which object detection is not performed.

[0071] First, in step S140, a variable k for identifying an object detected in the previous frame is set to 0 for a partial surface for which object detection is not performed.

[0072] In step S142, it is determined whether the value of k is less than Nk, where Nk is the number of objects detected in the partial surface in the previous frame. If the value of k reaches Nk, the interpolation determination processing routine is terminated. On the other hand, if the value of k is less than Nk, the process proceeds to step S144.

[0073] In step S144, a variable p for identifying the coordinates within the detection frame used for the judgment is set to 0, the maximum value of the absolute difference with the x component mvx_diff_max is set to 0, and similarly, the maximum value of the absolute difference with the y component mvy_diff_max is set to 0.

[0074] In step S146, it is determined whether the value of p is less than Np. Here, Np is the number of coordinates within the detection frame used in this determination; in the example of FIG. 3, there are four coordinates, above, below, left, and right of the center coordinate, so Np=4. Furthermore, mvx is the x-component of the motion vector of the corresponding coordinate, and mvy is the y-component. If the value of p reaches Np, the process proceeds to step S162. On the other hand, if the value of p is less than Np, the process proceeds to step S148.

[0075] In step S148, the coordinates cc[k][p] within the detection frame to be used for this determination are calculated for the detection frame [k] previously detected on the target partial surface.

[0076] In step S150, the absolute difference mvx_diff between the x component (mvx[c[k]]) of the motion vector at the center coordinate c[k] and the x component (mvx[c[k]]) of the motion vector corresponding to cc[k][p] is calculated, and similarly, the absolute difference mvy_diff for the y component is calculated.

[0077] In step S152, the absolute difference value of the obtained mvx_diff obtained within the same detection window is compared with the maximum value mvx_diff_max to determine whether it exceeds the maximum value. If it exceeds the maximum value, the process proceeds to step S154. On the other hand, if it does not reach the maximum value, the process proceeds to step S156. In step S154, mvx_diff is set as a new maximum value.

[0078] In step S156, the absolute difference value of the obtained mvy_diff obtained within the same detection window is compared with the maximum value mvy_diff_max to determine whether it exceeds the maximum value. If it exceeds the maximum value, the process proceeds to step S158. On the other hand, if it does not reach the maximum value, the process proceeds to step S160. In step S158, mvy_diff is set as a new maximum value.

[0079] In step S160, the value of p is incremented, and the process returns to step S146. As a result, the processes of steps S148 to S158 are executed for all determination target coordinates (cc[k][0] to cc[k][Np-1]) for detection frame [k].

[0080] In step S162, it is determined whether at least one of the obtained maximum values ​​mvx_diff_max, mvy_diff_max is equal to or greater than a determination threshold. If at least one of the obtained maximum values ​​mvx_diff_max, mvy_diff_max is equal to or greater than the determination threshold, it is determined that translation is inappropriate, and in step S164, detection frame [k] is removed from the interpolation targets. On the other hand, if both of the obtained maximum values ​​mvx_diff_max, mvy_diff_max are less than the determination threshold, in step S166, the center coordinates of detection frame [k] are translated based on the motion vector and interpolated.

[0081] In step S168, the value of k is incremented, and the process returns to step S142.

[0082] 10 is executed for the image of each frame, the CPU 11, as the allocation control unit 126, determines a partial surface on which object detection will be performed from among the partial surfaces that are the partial images obtained by dividing, and changes the allocation of partial surfaces on which object detection will be performed in the next cycle. For example, a cycle spanning multiple frames is determined, and based on the fluctuation value of the number of detections obtained in the current cycle, partial surfaces with larger fluctuations are determined so that object detection will be performed preferentially in the next frame.

[0083] As described above, the object detection device according to the first embodiment thins out frames on which object detection is performed, and determines, using a predetermined method, whether it is appropriate to reuse previously detected metadata as the detection result for the thinned-out frames. If it determines that the metadata is appropriate, it translates the center coordinates of the metadata based on the motion vector obtained from the difference with the image of the previous frame to use the result as the detection result for the thinned-out frames. If it determines that the metadata is inappropriate, it excludes the metadata that is determined to be inappropriate from being reused. This reduces the amount of calculation and enables highly accurate object detection.

[0084] Furthermore, the image of each frame of a video is divided and object detection is performed, the entire frame image is reduced and object detection is performed, and interpolation processing is performed on the partial surface where object detection is not performed by translating the center coordinates of the metadata detected in the past, and the result of object detection after division, the result of the interpolation processing, and the result of object detection after reducing the entire image are combined to obtain a final object detection result. This makes it possible to achieve CNN object detection with a small amount of calculation and high accuracy even for high-resolution images that exceed the constraint size.

[0085] Furthermore, the number of partial planes on which object detection is performed in the segmentation processing unit is made smaller than the total number of partial planes obtained by segmenting the input image, thereby reducing the amount of calculation, and the allocation control unit determines which partial planes to perform and when. Compared to conventional methods, the interpolation processing unit performs the validity determination, and outputs only those detection frames determined to be valid by translating the center coordinates of the detection frames based on the motion vector, while eliminating other detection frames. This reduces the amount of calculation and enables object detection with high accuracy.

[0086] Second Embodiment Next, an object detection device according to a second embodiment will be described. Note that parts having the same configuration as those in the first embodiment will be given the same reference numerals and descriptions thereof will be omitted.

[0087] The second embodiment differs from the first embodiment in that when a detection frame to be removed occurs, allocation control is performed so that object detection is performed preferentially on a partial surface including the detection frame in the next frame.

[0088] The allocation control unit 126 of the object detection device 10 of the second embodiment decides to perform object detection preferentially in the next frame for partial surfaces on which object detection is not performed and for which it is determined that it is not appropriate to reuse metadata.

[0089] Step S106 is implemented by an interpolation processing routine shown in FIG.

[0090] In step S170, a variable j is set to 0 for identifying the position of the target partial surface in the y direction (height direction) of the image of the frame.

[0091] In step S172, it is determined whether the variable j is less than Ny. Ny is the number of divisions in the y direction (height direction) of the frame image. If the variable j is less than Ny, the process proceeds to step S174. On the other hand, if the variable j has reached Ny, the process proceeds to step S186.

[0092] In step S174, a variable i for identifying the position of the target partial surface in the x direction (horizontal direction) of the image of the frame is set to 0.

[0093] In step S176, it is determined whether the variable i is less than Nx, where Nx is the number of divisions in the x direction (horizontal direction) of the frame image. If the variable i is less than Nx, the process proceeds to step S180. On the other hand, if the variable i has reached Nx, the process proceeds to step S178, where the value of j is incremented.

[0094] In step S180, it is determined whether the partial surface [i][j] is a partial surface to be thinned out without performing object detection. If the partial surface [i][j] is a partial surface to be thinned out, in step S182, interpolation determination is performed. On the other hand, if the partial surface [i][j] is not a partial surface to be thinned out, in step S184, irr_req[i][j] is set to 0 and mv_diff_max_hold[i][j] is set to 0.

[0095] In step S185, the value of i is incremented, and the process returns to step S176.

[0096] In step S186, it is determined whether irr_req[i][j] for all partial surfaces is 0. If irr_req[i][j] for at least one partial surface is 1, the interpolation processing routine ends. If irr_req[i][j] for all partial surfaces is 0, in step S188, irr_req_total, which is a flag indicating that removal has occurred for at least one partial surface, is set to 0.

[0097] In this way, for partial surfaces that were not subject to thinning (partial surfaces for which object detection is performed), irr_req[i][j] and mv_diff_max_hold[i][j] are both set to 0. When the above process is completed for all partial surfaces, it is checked whether irr_req for all partial surfaces is 0, and if all are 0, irr_total_req is set to 0.

[0098] Step S182 is implemented by the interpolation determination process routine shown in Fig. 15. This process is repeatedly executed for each of the partial surfaces [i][j] in the frame where object detection is not performed.

[0099] First, in step S200, a variable k for identifying an object detected in the previous frame is set to 0 for a partial surface for which object detection is not performed.

[0100] In step S202, it is determined whether the value of k is less than Nk, where Nk is the number of objects detected in the partial surface in the previous frame. If the value of k reaches Nk, the interpolation determination processing routine is terminated. On the other hand, if the value of k is less than Nk, the process proceeds to step S204.

[0101] In step S204, a variable p for identifying the coordinates within the detection frame used for the judgment is set to 0, the maximum value of the absolute difference with the x component mvx_diff_max is set to 0, and similarly, the maximum value of the absolute difference with the y component mvy_diff_max is set to 0.

[0102] In step S206, it is determined whether the value of p is less than Np. Here, Np is the number of coordinates within the detection frame used in this determination. Furthermore, mvx is the x-component of the motion vector of the corresponding coordinate, and mvy is the y-component. If the value of p reaches Np, the process proceeds to step S222. On the other hand, if the value of p is less than Np, the process proceeds to step S208.

[0103] In step S208, the coordinates cc[k][p] within the detection frame to be used for this determination are calculated for the detection frame [k] previously detected on the target partial surface.

[0104] In step S210, the absolute difference mvx_diff between the x component of the motion vector at the center coordinate c[k] (mvx[c[k]]) and the x component of the motion vector corresponding to cc[k][p] (mvx[cc[k][p]]) is calculated, and similarly, the absolute difference mvy_diff for the y component is calculated.

[0105] In step S212, the absolute difference value of the obtained mvx_diff is compared with the maximum value mvx_diff_max obtained within the same detection window up to that point to determine whether it exceeds the maximum value. If it exceeds the maximum value, the process proceeds to step S214. On the other hand, if it does not reach the maximum value, the process proceeds to step S216. In step S214, mvx_diff is set as a new maximum value.

[0106] In step S216, the absolute difference value of the obtained mvy_diff is compared with the maximum value mvy_diff_max obtained within the same detection window up to that point to determine whether it exceeds the maximum value. If it exceeds the maximum value, the process proceeds to step S218. On the other hand, if it does not reach the maximum value, the process proceeds to step S220. In step S218, mvy_diff is set as a new maximum value.

[0107] In step S220, the value of p is incremented, and the process returns to step S146.

[0108] This is performed for all determination target coordinates (cc[k][0] to cc[k][Np-1]) for the detection frame [k].

[0109] In step S222, it is determined whether at least one of the obtained maximum values ​​mvx_diff_max, mvy_diff_max is greater than or equal to the determination threshold. If at least one of the obtained maximum values ​​mvx_diff_max, mvy_diff_max is greater than or equal to the determination threshold, it is determined that translation is not appropriate, and in step S224, detection frame[k] is removed from the interpolation targets. In step S226, a flag irr_req[i][j] indicating the occurrence of removal is set to 1, and a flag irr_req_total indicating that the removal has occurred in at least one partial surface is also set to 1. On the other hand, if both of the obtained maximum values ​​mvx_diff_max, mvy_diff_max are less than the determination threshold, in step S228, the center coordinates of detection frame[k] are translated based on the motion vector to perform interpolation.

[0110] In step S230, the maximum value of the absolute difference values ​​occurring within the thinning period is set to mv_diff_max_hold[i][j].

[0111] In step S232, the value of k is incremented, and the process returns to step S202.

[0112] Furthermore, every time the object detection shown in FIG. 10 is executed for an image of each frame, the CPU 11, as the allocation control unit 126, executes an allocation control processing routine shown in FIG.

[0113] First, in step S300, it is determined whether irr_total_req is 1. If irr_total_req is 1, in step S302, partial planes on which object detection will be performed in the next frame are determined in descending order of mv_diff_max_hold so that the number of partial planes on which object detection can be performed falls within the number of partial planes on which object detection can be performed. On the other hand, if irr_total_req is 0, in step S304, normal allocation control is performed. For example, a period spanning multiple video frames is determined, and based on the fluctuation value of the number of detections obtained from the current period, partial planes with the largest fluctuations are determined so that object detection will be performed in the next frame preferentially.

[0114] In this way, among the partial surfaces that are the partial images obtained by division, the partial surfaces on which object detection is performed are determined, and the allocation of the partial surfaces on which object detection is performed in the next cycle is changed.

[0115] The other configurations and operations of the object detection device 10 according to the second embodiment are the same as those of the first embodiment, and therefore will not be described again.

[0116] As described above, the object detection device according to the second embodiment determines, by the allocation control unit, to perform object detection preferentially in the next frame for partial surfaces for which object detection is not performed and for which it is determined that reusing metadata is inappropriate. This reduces the amount of calculation and enables accurate object detection.

[0117] In the above embodiments, the various processes executed by the CPU after reading the software (programs) may be executed by various processors other than the CPU. Examples of such processors include a PLD (Programmable Logic Device) whose circuit configuration can be changed after manufacture, such as an FPGA (Field-Programmable Gate Array), and a dedicated electrical circuit, such as an ASIC (Application Specific Integrated Circuit), which is a processor having a circuit configuration designed specifically to execute specific processes. Furthermore, the various processes may be executed by one of these various processors, or by a combination of two or more processors of the same or different types (e.g., multiple FPGAs, or a combination of a CPU and an FPGA). Furthermore, the hardware structure of these various processors is, more specifically, an electrical circuit that combines circuit elements such as semiconductor elements.

[0118] In addition, in each of the above embodiments, the object detection program is pre-stored (installed) in the storage 14, but this is not limiting. The program may be provided in a form stored on a non-transitory storage medium such as a CD-ROM (Compact Disk Read Only Memory), a DVD-ROM (Digital Versatile Disk Read Only Memory), or a USB (Universal Serial Bus) memory. The program may also be downloaded from an external device via a network.

[0119] The technology of the present disclosure can also be appropriately combined with the above-described exemplary embodiments and modified examples. The above-described description and illustrations are detailed descriptions of the parts related to the technology of the present disclosure and are merely examples of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, unnecessary parts may be deleted, new elements may be added, or new elements may be replaced with other elements from the above-described description and illustrations, within the scope of the gist of the technology of the present disclosure.

[0120] The following additional notes are provided regarding the above-described embodiments.

[0121] (Supplementary Item 1) An object detection device that outputs metadata including at least the center coordinates of an object included in an image and a frame surrounding the object from an image in each frame of a moving image, the object detection device comprising: a memory; and at least one processor connected to the memory, wherein the processor is configured to: thin out frames on which object detection is performed, and determine, using a predetermined method, the appropriateness of reusing the metadata detected in the past as the detection result for the thinned out frame; if it is determined to be appropriate, translate the center coordinates of the metadata based on a motion vector obtained from the difference with the image of the previous frame, and use this as the detection result for the thinned out frame; and if it is determined to be inappropriate, exclude the metadata from being reused.

[0122] (Supplementary Item 2) A non-transitory storage medium storing a program executable by a computer to execute an object detection process that outputs metadata including at least the center coordinates of an object included in an image and a frame surrounding the object from an image in each frame of a moving image, wherein the object detection process thins out frames on which object detection is executed, and determines using a predetermined method whether it is appropriate to reuse the metadata detected in the past as the detection result for the thinned out frame, and if it is determined to be appropriate, translates the center coordinates of the metadata based on a motion vector obtained from the difference with the image of the previous frame to use it as the detection result for the thinned out frame, and if it is determined to be inappropriate, excludes the metadata from being reused.

[0123] REFERENCE SIGNS LIST 10 Object detection device 11 CPU 14 Storage 15 Input unit 16 Display unit 100 Motion estimation unit 110 Segmentation processing unit 111 Segmentation unit 112 Selection unit 113 Object detection unit 114 Storage unit 116 First selection unit 120 Interpolation processing unit 121 Second selection unit 122 Interpolation determination unit 126 Allocation control unit 130 Overall processing unit 131 Reduction unit 132 Object detection unit 140 Synthesis processing unit

Claims

1. An object detection device that outputs metadata including at least the central coordinates of an object contained in an image and a frame surrounding the object from an image in each frame of a video, thins out frames in which object detection is performed, and determines using a predetermined method the appropriateness of reusing the metadata detected in the past as the detection result for the thinned out frames, and if determined to be appropriate, translates the central coordinates of the metadata based on a motion vector obtained from the difference with the image of the previous frame to use as the detection result for the thinned out frame, and if determined to be inappropriate, excludes the metadata from being reused.

2. An object detection device as described in claim 1, which determines that the absolute difference between a motion vector corresponding to the center coordinates of the metadata and a motion vector corresponding to coordinates within the frame of the metadata is valid if the value obtained is below a predetermined threshold, and determines that the difference is invalid otherwise.

3. An object detection device as described in claim 1, comprising: a segmentation processing unit that segments an image of each frame of the video and performs object detection; an allocation control unit that determines, for each frame image, partial surfaces that are partial images obtained by segmenting the image, which partial surfaces will perform object detection; an overall processing unit that reduces the entire image of the frame and performs object detection; an interpolation processing unit that performs interpolation processing by parallel translation of center coordinates of the metadata previously detected, for the partial surfaces in which object detection is not performed; and a combination processing unit that obtains a final object detection result by combining the object detection result of the segmentation processing unit, the interpolation processing result of the interpolation processing unit, and the object detection result of the overall processing unit, wherein the interpolation processing unit determines the appropriateness of reusing the metadata previously detected as a detection result for a thinned-out frame, for the partial surfaces in which object detection is not performed, and if determined to be appropriate, translates the center coordinates of the metadata based on the motion vector to use it as the detection result for the thinned-out frame, and if determined to be inappropriate, excludes the metadata from the targets for reuse.

4. The object detection device of claim 3, wherein the allocation control unit determines that object detection is to be performed preferentially in the next frame for a partial surface in which object detection is not performed and for which it is determined that it is inappropriate to reuse the metadata.

5. An object detection method in an object detection device that outputs metadata including at least the central coordinates of an object contained in an image and a frame surrounding the object from an image in each frame of a video, the method comprising: thinning out frames in which object detection is performed; determining, using a predetermined method, the appropriateness of reusing the metadata detected in the past as the detection result for the thinned-out frames; if determined to be appropriate, translating the central coordinates of the metadata based on a motion vector obtained from the difference with the image of the previous frame, and using the metadata as the detection result for the thinned-out frames; if determined to be inappropriate, excluding the metadata from the scope of possible reuse.

6. An object detection program for causing a computer to function as the object detection device according to any one of claims 1 to 4.