Object detection device, and object detection method

Abstract

This object detection device comprises: a calculation unit that sets, as a target frame, each frame of a moving image including a plurality of frames and calculates a motion vector based on a reference frame over the target frame; a determination unit that determines an update region in a cumulative image on the basis of a comparison between the target frame and the cumulative image cumulatively generated on the basis of a past frame and motion vector before the target frame; an update unit that updates the cumulative image by performing motion compensation using the motion vector and overwrites the update region using the target frame; a detection unit that detects an object using an object detection model in a region in the target frame corresponding to the update region; and a correction unit that corrects, using the motion vector, the detection result of the object detected using the object detection model from a past frame before the target frame, in a region in the target frame corresponding to a region other than the update region.

Description

Object detection device and object detection method

[0001] The disclosed technology relates to an object detection device and an object detection method.

[0002] In recent years, technologies for performing object detection and the like through inference processing using neural networks have become widespread. For example, techniques such as YOLO (You Only Look Once) v3 (Non-Patent Document 1) are known. In such technologies, objects contained in an input image are detected by inputting the input image into a convolutional neural network (hereinafter referred to as "CNN"), which is an object detection model. When an input image contains multiple objects, each object is detected separately. In addition, attributes of the detected objects, reliability of the detection results, rectangular information representing the position and size of an area containing the detected object in the image, and the like are generally output as the detection results of the object detection. The object attributes refer to the type of object, such as a car, a person, or a dog.

[0003] When applying the above-mentioned method to video (images), it is necessary to repeat the object detection process for all frames included in the video, which poses a problem of increasing the amount of processing. In response to this, there is a method that aims to achieve a similar effect while reducing the amount of processing compared to performing object detection by CNN on all frames by dividing the frames into those in which object detection by CNN is performed and those in which objects are tracked based on the results.

[0004] In addition, a method has been proposed in which object detection is performed using a CNN in key frames, and in other frames, feature maps in the middle layers of the CNN are predicted from the key frames using motion compensation, and normal CNN processing is performed in subsequent layers (Non-Patent Document 2).

[0005] Also, a method has been proposed in which a moving image is divided into multiple regions and object detection processing is performed for each region (Patent Document 1).Furthermore, a method has been proposed in which, based on the motion vector of a detected object, it is determined whether object detection processing can be skipped for each region, and if skipped, past detection results of the object are corrected using the motion vector (Patent Document 2).

[0006] International Publication No. 2021 / 161513 International Publication No. 2022 / 172341

[0007] J. Redomon, et.al., "YOLOv3: An Incremental Improvement", https: / / arxiv.org / abs / 1804.02767M.Buckler, et.al., "EVA2: Exploiting Temporal Redundancy In Live Computer Vision", ISCA2018

[0008] However, this method of dividing the frame into one in which object detection is performed using CNN and one in which the object is tracked based on the results of that detection has the problem that if a new object appears in a frame in which only tracking is performed, the new object cannot be detected.

[0009] The method described in Non-Patent Document 2 has the problem that the processing load in the latter layers of the CNN is large. Also, for video images with many small changes in motion, motion compensation of the feature map does not function effectively, resulting in a decrease in the accuracy and efficiency of object detection.

[0010] The techniques described in Patent Documents 1 and 2 have had problems in that when an object crosses regions between frames, it takes a long time to detect the object or the object cannot be detected properly.

[0011] In recent years, there has been a demand for a method for more efficiently detecting objects in each frame of a video without reducing accuracy, such as when processing object detection in higher-resolution video in real time.

[0012] The disclosed technology has been made in consideration of the above points, and aims to provide an object detection device and an object detection method that improve efficiency in object detection in moving images without reducing accuracy.

[0013] A first aspect of the present disclosure is an object detection device comprising: a calculation unit that sets each frame of a moving image including a plurality of frames as a target frame, and calculates a motion vector for the entire target frame relative to a reference frame; a determination unit that determines an update area in the cumulative image based on a comparison between the target frame and a frame earlier than the target frame and a cumulative image that is cumulatively generated based on the motion vector; an update unit that updates the cumulative image by performing motion compensation using the motion vector and overwrites the update area using the target frame; a detection unit that detects an object using an object detection model in an area in the target frame that corresponds to the update area; and a correction unit that corrects, using the motion vector, a detection result of an object detected using the object detection model from a frame earlier than the target frame in an area in the target frame that corresponds to an area that is not the update area.

[0014] A second aspect of the present disclosure is an object detection method, in which a computer executes the following processes: each frame of a moving image including a plurality of frames is set as a target frame; motion vectors are calculated for the entire target frame relative to a reference frame; an update region in the cumulative image is determined based on a comparison of the target frame with frames earlier than the target frame and a cumulative image that is cumulatively generated based on the motion vector; the cumulative image is updated by performing motion compensation using the motion vector and the update region is overwritten using the target frame; an object is detected using an object detection model in a region in the target frame that corresponds to the update region; and a detection result of the object detected using the object detection model from a frame earlier than the target frame is corrected using the motion vector in a region in the target frame that corresponds to a region that is not the update region.

[0015] According to the disclosed technology, it is possible to improve efficiency in object detection in moving images without reducing accuracy.

[0016] FIG. 1 is a block diagram showing the hardware configuration of an object detection device. FIG. 2 is a block diagram showing an example of the functional configuration of an object detection device according to a first embodiment. FIG. 3 is a diagram for explaining a method of detecting an object. FIG. 4 is a flowchart showing the flow of object detection in the first embodiment. FIG. 5 is a block diagram showing an example of the functional configuration of an object detection device according to a second embodiment. FIG. 6 is a diagram for explaining generation of a partial image and a reduced image. FIG. 7 is a block diagram showing an example of the functional configuration of an object detection device according to a third embodiment. FIG. 8 is a diagram for explaining the prior art. FIG. 9 is a diagram for explaining the prior art.

[0017] An example of an embodiment of the disclosed technology will be described below with reference to the drawings. Note that the same reference numerals are used to designate identical or equivalent components and parts in each drawing. Also, the dimensional proportions in the drawings are exaggerated for the sake of explanation and may differ from the actual proportions.

[0018] The disclosed technology relates to a method for dividing a video into multiple regions and detecting an object for each region using an object detection model. Specifically, as shown in Fig. 8 , for each frame (n-2, n-1, n, n+1, ...) of the video, processing is performed only for some regions where object detection is required, and past object detection results are corrected and used instead for other regions.

[0019] Areas where object detection is desirable are those with greater movement compared to previous frames in chronological order. Conventionally, a method has been used to determine the target area for object detection based on an index (hereinafter referred to as the "cumulative index") that indicates the cumulative value of differences in pixel values ​​between frames. In this method, when the cumulative index for a certain area becomes higher than a predetermined threshold, object detection processing is performed for that area. Once object detection processing is completed, the cumulative index for that area is initialized, and past object detection results are corrected until the cumulative index becomes high again.

[0020] However, with this method, when an object crosses over regions, the object may slip through the judgment and the detection result of the object may not be updated appropriately. For example, as shown in Figure 9, when object X moves from region A, where the cumulative index is high and object detection processing will be performed soon, to region B, where the cumulative index is low and object detection processing will not be performed for a while, it may take a long time to detect object X or object X may not be detected at all.

[0021] Therefore, the object detection device according to the technology of the present disclosure uses cumulatively updated images (hereinafter referred to as "cumulative images") when correcting the object detection results (i.e., motion compensation using motion vectors) to more accurately grasp the deviation status of the object detection results. This allows the target area for object detection processing to be determined with high accuracy, and realizes efficient processing while avoiding a decrease in detection accuracy due to object movement, deformation, appearance, etc.

[0022] 1 is a block diagram showing the hardware configuration of an object detection device 10 according to a first embodiment. As shown in Fig. 1, the object detection device 10 includes a CPU (Central Processing Unit) 11, a ROM (Read Only Memory) 12, a RAM (Random Access Memory) 13, a storage 14, an input unit 15, a display unit 16, and a communication I / F (Interface) 17. Each component is connected to each other via a bus 19 so as to be able to communicate with each other.

[0023] The CPU 11 is a central processing unit that executes various programs and controls each component. That is, the CPU 11 reads a program from the ROM 12 or the storage 14 and executes the program using the RAM 13 as a work area. The CPU 11 controls the above components and performs various arithmetic processing in accordance with the program stored in the ROM 12 or the storage 14. In this embodiment, the ROM 12 or the storage 14 stores an object detection program for executing the object detection processing described below.

[0024] The ROM 12 stores various programs and various data. The RAM 13 temporarily stores programs or data as a working area. The storage 14 is configured by a storage device such as an HDD (Hard Disk Drive) or an SSD (Solid State Drive), and stores various programs including an operating system and various data.

[0025] The input unit 15 includes a pointing device such as a mouse and a keyboard, and is used to input various types of information. The display unit 16 is, for example, a liquid crystal display, and displays various types of information. The display unit 16 may also function as the input unit 15 by employing a touch panel system.

[0026] The communication I / F 17 is an interface for communicating with other devices, and the communication may use a wired communication standard such as Ethernet (registered trademark) or FDDI (Fiber Distributed Data Interface), or a wireless communication standard such as 4G, 5G, or Wi-Fi (registered trademark).

[0027] Next, the functional configuration of the object detection device 10 will be described. Fig. 2 is a block diagram showing an example of the functional configuration of the object detection device 10. As shown in Fig. 2, the object detection device 10 has, as its functional configuration, an input unit 21, a calculation unit 22, an input image buffer 23, a motion vector buffer 24, an update unit 25, an accumulated image buffer 26, and a determination unit 27. The object detection device 10 further has, as its functional configuration, an image formation unit 28, a detection unit 29, a correction unit 30, a detection result buffer 31, and an output unit 32. Each functional configuration is realized when the CPU 11 reads out an object detection program stored in the ROM 12 or the storage 14, expands it in the RAM 13, and executes it.

[0028] The input unit 21 accepts a moving image including a plurality of frames input to the object detection device 10, and sequentially outputs each frame constituting the moving image as an input image to the calculation unit 22 and the image formation unit 28 in chronological order of the moving image. Here, it is assumed that each frame included in the moving image is assigned a frame number n=0, 1, 2, ... in order from the beginning. Hereinafter, the frame with frame number n will be referred to as "frame n." The input unit 21 sets frame n as a target frame.

[0029] The calculation unit 22 receives the target frame output from the input unit 21. The calculation unit 22 also acquires a reference frame stored in the input image buffer 23. The reference frame is a frame that serves as a reference when calculating a motion vector for the target frame. The input image buffer 23 stores frames that are older than the target frame (i.e., frames that are earlier in chronological order). The calculation unit 22 selects a reference frame from the frames stored in the input image buffer 23.

[0030] The calculation unit 22 then calculates motion vectors for the entire target frame with respect to the reference frame. The motion vectors are obtained by performing motion search for each block obtained by dividing the entire target frame. That is, the motion vectors are expressed as vectors for each block. The block here refers to the smallest unit area for which a motion vector is calculated, and may be, for example, a block with sx pixels in the horizontal direction and sy pixels in the vertical direction. As an example, the motion vector may be calculated with integer precision for each block of 16 pixels in the horizontal direction and 16 pixels in the vertical direction. Conventionally known methods such as a block matching method and a gradient method may be used as the motion search method, and the method is not limited in this embodiment.

[0031] The calculation unit 22 also stores the target frame (frame n) in the input image buffer 23 to use it as a reference frame when frame n+1 becomes the target frame. The calculation unit 22 also stores the calculated motion vector in the motion vector buffer 24.

[0032] The update unit 25 updates the cumulative image by performing motion compensation using the motion vector. The cumulative image is an image cumulatively generated based on frames past the target frame. The cumulative image is used as a basis for determining whether each region constituting the target frame is a target region for object detection processing.

[0033] Furthermore, the update unit 25 updates the cumulative image by overwriting the update area in the cumulative image with the target frame. That is, the update unit 25 updates the previous cumulative image while changing the update method depending on whether each area in the cumulative image is an update area.

[0034] The update unit 25 also stores the cumulative image updated by motion compensation and overwriting in the cumulative image buffer 26 .

[0035] The determination unit 27 determines an update area in the cumulative image based on a comparison between the target frame and the cumulative image, and outputs the determination result of the update area to the update unit 25, the image forming unit 28, the detection unit 29, and the correction unit 30.

[0036] Specific examples of the functions of the calculation unit 22, the update unit 25, and the determination unit 27 will be described with reference to FIG. 3. Below, a case will be described in which the reference frame is the frame immediately preceding the target frame, i.e., the reference frame for frame n is frame n-1. In the example of FIG. 3, frame n-1, frame n, cumulative image C(n-1), cumulative image C(n)p, and cumulative image C(n) are each divided into six regions A to F. From frame n-1 to frame n, object X moves from region A to region B, and object Y newly appears across regions E to F.

[0037] First, if the target frame is the first frame n (i.e., n=0), the update unit 25 sets cumulative image C(0) using frame 0. Specifically, the update unit 25 prepares a cumulative image that is the same size as the input image and in which all pixels are initialized to predetermined initial values. Next, the determination unit 27 determines an area in frame 0, the target frame, where object detection is to be performed based on a predetermined rule, and instructs the detection unit 29 to perform object detection in that area. Then, the update unit 25 overwrites the initial cumulative image using frame 0 for the area determined by the determination unit 27 to perform object detection, and sets this overwritten cumulative image as cumulative image C(0).

[0038] 3, for example, if a rule is defined to perform object detection for all regions A to F in frame 0, the update unit 25 obtains cumulative image C(0) by overwriting the cumulative image for all regions A to F using frame 0. As a result, in the first frame n (i.e., n=0), cumulative image C(0) and frame 0 are the same.

[0039] Next, a case will be described where the target frame is frame n where n > 0. The calculation unit 22 performs a motion search between the target frame (frame n) and the reference frame (frame n-1) and calculates a motion vector V(n).

[0040] The update unit 25 performs motion compensation for the object X based on the immediately preceding accumulated image C(n-1) using the motion vector V(n) calculated by the calculation unit 22. As a result, an accumulated image C(n)p including the object X that has moved from region A to region B is obtained.

[0041] The determination unit 27 determines an update region in the cumulative image C(n)p based on a comparison between the target frame n and the cumulative image C(n)p. Specifically, the determination unit 27 calculates a difference value for each pixel between the target frame n and the cumulative image C(n)p, and calculates an aggregate value of the results of comparing the difference value with a predetermined threshold for each of the regions A to F. The determination unit 27 then determines that the region among the regions A to F whose aggregate value for each region is greater than the predetermined threshold is an update region.

[0042] For example, suppose frame n includes multiple regions m, and each region m includes multiple pixels x. n, m, and x are all integers. For each pixel x, the determination unit 27 calculates the difference D1(n, x) between the absolute value of the difference between the pixel value of the target frame n and the pixel value of the cumulative image C(n)p and a predetermined first threshold. The determination unit 27 also calculates a sum S(n, m) of the differences D1(n, x) for each region m as an aggregate value, and determines a region for which the sum S(n, m) is greater than a predetermined second threshold to be an update region.

[0043] The comparison method with the threshold may be a conventionally known method, and the method is not limited to this embodiment. For example, the threshold used in the threshold processing of the difference value for each pixel (the first threshold in the above example) may be a fixed value set in advance. Alternatively, for example, the threshold may be set according to the absolute value of the pixel value gradient between the target pixel and an adjacent pixel, and the difference value of the target pixel may be binarized as 0 if it is less than the threshold and 1 if it is greater than or equal to the threshold, and isolated points may then be removed by referring to neighboring pixels. In this case, the effects of image noise and the accuracy of motion vector calculation can be eliminated, thereby more emphasizing changes in the image pattern (i.e., object movement) and improving the accuracy of determining the update area.

[0044] Alternatively, for example, the threshold value used in the threshold processing for the aggregate value for each region (the second threshold value in the above example) may be set to a fixed value in advance, or may be set by performing feedback control based on past determination results for the update regions so that a desired number of update regions are selected.

[0045] Furthermore, for example, the determination of the update region may be based on other indices other than the difference value for each pixel. For example, an index representing the variation in motion vectors may be used, as in the technology described in Patent Document 2.

[0046] Furthermore, when the determining unit 27 determines that an area is an update area, the updating unit 25 overwrites the cumulative image C(n)p with the target frame n for that update area. In the example of FIG. 3, areas E to F are determined to be update areas, so areas E to F in the cumulative image C(n)p are overwritten with frame n, which is the target frame. As a result, a cumulative image C(n) is obtained in which motion compensation is performed on areas A to D using the motion vector V(n) based on the cumulative image C(n-1), and areas E to F are overwritten with frame n.

[0047] The image forming unit 28 receives the target frame output from the input unit 21. Furthermore, based on the update region determination result by the determination unit 27, the image forming unit 28 cuts out a region in the target frame that corresponds to the update region, and then outputs the region to the detection unit 29. Furthermore, for example, the image forming unit 28 may resize the region cut out from the target frame to a size that can be input to the object detection model of the detection unit 29, and then output the region to the detection unit 29. However, the size of the update region may be set in advance to a size that can be input to the object detection model of the detection unit 29. In this case, the resizing is not necessary, and therefore efficiency can be further improved.

[0048] The detection unit 29 receives the image output from the image formation unit 28 (i.e., an image showing the area in the target frame corresponding to the update area) and inputs the image into an object detection model to detect an object included in the image. That is, the detection unit 29 detects an object using the object detection model in the area in the target frame corresponding to the update area. The object detection model may be, for example, a model that uses a CNN such as YOLO.

[0049] If there are multiple objects, the detection unit 29 detects each of the multiple objects. Then, the detection unit 29 outputs, to the output unit 32, the object detection results, such as object attributes, the reliability of the detection result, and rectangular information indicating the position and size of an area including the detected object within the target frame.

[0050] The correction unit 30 corrects the detection results of an object detected using an object detection model from a frame prior to the target frame (hereinafter referred to as "past detection results of the object") in an area within the target frame corresponding to an area that is not an update area, using a motion vector.

[0051] Specifically, the correction unit 30 acquires past object detection results stored in the detection result buffer 31. The correction unit 30 also acquires, from the motion vector buffer 24, a motion vector for the coordinates (position information) of the target frame that correspond to the center coordinates of the object rectangle information included in the past object detection results. The correction unit 30 then corrects the center coordinates of the object rectangle information included in the past object detection results by the amount of the acquired motion vector.

[0052] The correction unit 30 then generates rectangle information representing the position and size of a rectangle whose center coordinates are the corrected coordinates and whose size is the same as the rectangle indicated by the object rectangle information included in the past object detection results. The correction unit 30 also acquires the object attributes included in the past object detection results, the reliability of the detection results, etc. The correction unit 30 compiles this information and outputs it to the output unit 32 as the detection result of the object detection in the target frame.

[0053] The output unit 32 receives and outputs the detection results output from the detection unit 29 and / or the correction unit 30. Note that, as a method for integrating the detection results for each region and outputting the integrated detection result, the techniques described in Patent Documents 1 and 2, for example, can be appropriately applied. Furthermore, the output unit 32 stores the detection result of frame n in the detection result buffer 31 so that the result can be referenced when frame n+1 becomes the target frame.

[0054] Next, the operation of the object detection device 10 will be described. Fig. 4 is a flowchart showing the flow of the object detection process performed by the object detection device 10. The object detection process is performed by the CPU 11 reading an object detection program from the ROM 12 or the storage 14, expanding it into the RAM 13, and executing it. Note that the object detection process is an example of an object detection method disclosed herein.

[0055] First, as an initial state, the CPU 11 sets a variable n, which indicates the frame number of each frame constituting a moving image, to 0. In step S10, the CPU 11, acting as the input unit 21, sets frame n of the moving image as the target frame. In step S12, the CPU 11, acting as the input unit 21, determines whether n is greater than 0. If n>0, the process proceeds to step S18, and if n=0, the process proceeds to step S14.

[0056] In step S14, the CPU 11, functioning as the determination unit 27, instructs the detection unit 29 to perform object detection on a predetermined region (e.g., the entire region) of frame n (here, n = 0). In response to this instruction, the CPU 11, functioning as the detection unit 29, detects an object using an object detection model in a predetermined region (e.g., the entire region) in frame n. In step S16, the CPU 11, functioning as the update unit 25, sets a cumulative image C(n) (here, n = 0) using frame n (here, n = 0) for the region in which object detection was performed in step S14. When step S16 is completed, the process proceeds to step S36.

[0057] In step S18, the CPU 11, functioning as the input unit 21, sets frame n-1 as the reference frame. In step S20, the CPU 11, functioning as the calculation unit 22, performs motion search between frames n and n-1, and calculates the motion vector V(n) of frame n.

[0058] In step S22, the CPU 11, functioning as the update unit 25, performs motion compensation using the motion vector V(n) calculated in step S20 based on the immediately preceding cumulative image C(n-1), and designates the image to which this motion compensation has been applied as cumulative image C(n)p. In step S24, the CPU 11, functioning as the determination unit 27, calculates a difference value for each pixel between frame n (target frame) and cumulative image C(n)p, compares the difference value with a predetermined threshold, and calculates a total value of the results for each region m, where m is an integer of 2 or greater.

[0059] In step S26, the CPU 11, functioning as the determination unit 27, determines whether the aggregate value in region m exceeds a predetermined threshold. If step S26 is positive, the region m is determined to be an update region, and the process proceeds to step S28. In step S28, the CPU 11, functioning as the determination unit 27, instructs the detection unit 29 to perform object detection for region m in frame n. In response to this instruction, the CPU 11, functioning as the detection unit 29, detects an object in region m in frame n using an object detection model. In step S30, the CPU 11, functioning as the update unit 25, overwrites region m in the cumulative image C(n)p with region m in frame n detected in step S28.

[0060] On the other hand, if the result in step S26 is negative, it is determined that the region m is not an update region, and the process proceeds to step S32. In step S32, the CPU 11, functioning as the correction unit 30, corrects the detection result of the object detected in frame n-1 in region m within frame n using the motion vector V(n) calculated in step S20, and sets the corrected detection result as the detection result of the object in frame n.

[0061] When step S30 or step S32 is completed, the process proceeds to step S34. In step S34, the CPU 11, functioning as the update unit 25, determines whether updating has been completed for all regions m in the cumulative image C(n)p. If the result of step S34 is negative, steps S26 to S34 are repeated until updating has been completed for all regions m in the cumulative image C(n)p.

[0062] On the other hand, if the result of step S34 is affirmative, the process proceeds to step S36. In step S36, the CPU 11, functioning as the output unit 32, receives the object detection results in steps S14, S28, and / or S32 and outputs them as the object detection result for frame n. In step S38, the CPU 11, functioning as the input unit 21, increments n and returns to step S12.

[0063] As described above, the object detection device 10 according to the first embodiment sets each frame of a video including multiple frames as a target frame and calculates motion vectors for the entire target frame relative to a reference frame. The object detection device 10 also determines an update region in the cumulative image by comparing the target frame with a cumulative image generated cumulatively based on frames and motion vectors from before the target frame. The object detection device 10 also updates the cumulative image by performing motion compensation using the motion vectors and overwrites the update region with the target frame. The object detection device 10 also detects an object using an object detection model in a region in the target frame that corresponds to the update region. The object detection device 10 also corrects, using the motion vector, the detection result of an object detected using the object detection model from a frame from before the target frame in a region in the target frame that corresponds to a region that is not an update region.

[0064] According to the object detection device 10 of the first embodiment, by using a cumulative image obtained by cumulatively updating an image using an input image, it is possible to grasp the difference between an image obtained by repeated motion compensation using a motion vector and the actual input image. In particular, the object detection device 10 of the first embodiment overwrites the cumulative image using the target frame for the updated region, so that the cumulative image is close to the actual state and the aggregate value of the differences between frames can be accurately grasped. Therefore, it is possible to accurately determine whether or not object detection processing can be omitted for each region, thereby improving efficiency without reducing the accuracy of object detection.

[0065] Furthermore, the motion vector search, motion compensation, difference determination, and the like performed by the object detection device 10 can be implemented efficiently because they can utilize hardware circuits used in general-purpose processing such as video encoding.

[0066] Second Embodiment Next, a second embodiment will be described. An object detection device 210 according to the second embodiment integrates the object detection results of each partial image obtained by dividing a frame into multiple parts with the object detection results of a reduced image obtained by reducing the entire frame, that is, performs object detection using frame division and synthesis. In the object detection process for each partial image, the object detection device 210 according to the second embodiment determines whether to skip the object detection process using an object detection model based on a comparison between the target frame and the cumulative image, as in the first embodiment.

[0067] In the object detection device 210 according to the second embodiment, the same components as those in the object detection device 10 according to the first embodiment are denoted by the same reference numerals and detailed description thereof will be omitted. Furthermore, the hardware configuration of the object detection device 210 according to the second embodiment is similar to the hardware configuration of the object detection device 10 according to the first embodiment shown in FIG. 1, and therefore description thereof will be omitted.

[0068] First, the functional configuration of the object detection device 210 according to the second embodiment will be described. FIG. 5 is a block diagram showing an example of the functional configuration of the object detection device 210 according to the second embodiment. As shown in FIG. 5, the object detection device 210 has, as its functional configuration, an input unit 221, a calculation unit 222, an input image buffer 23, a motion vector buffer 24, an update unit 225, an accumulated image buffer 26, and a determination unit 227. Furthermore, the object detection device 210 has, as its functional configuration, a partial image generation unit 228A, a reduced image generation unit 228B, a detection unit 229, a correction unit 230, a detection result buffer 31, and an output unit 232. Furthermore, the detection unit 229 has a partial image detection unit 229-m (here, m = 1, 2, 3, 4) and a reduced image detection unit 229B. Each functional configuration is realized when the CPU 11 reads out an object detection program stored in the ROM 12 or storage 14, expands it in the RAM 13, and executes it.

[0069] In addition, detailed explanations of the common contents between the functional configuration of the object detection device 210 according to the second embodiment and the functional configuration of the object detection device 10 according to the first embodiment, where the last two digits of the reference numerals are the same, will be omitted.

[0070] The input unit 221 outputs frame n of the moving image as a target frame to each of the calculation unit 222, the partial image generation unit 228A, and the reduced image generation unit 228B.

[0071] The partial image generation unit 228A generates a plurality of partial images by dividing the target frame into a plurality of portions. For example, as shown in the upper part of Fig. 6, the partial image generation unit 228A extracts four overlapping portions located at the upper left, upper right, lower left, and lower right of the target frame. The partial image generation unit 228A then generates each of the extracted portions as partial image 1 (long dashed line), partial image 2 (short dashed line), partial image 3 (dotted line), and partial image 4 (dash-dotted line). The m in partial image m (here, m = 1, 2, 3, 4) is the number of the partial image.

[0072] Here, the size of each partial image is a size that can be input to the object detection model. Furthermore, each partial image and each region (see the first embodiment) used when determining the update region of the cumulative image may be the same or different. In the example of FIG. 6 , as shown by the solid lines, the target frame is divided into four regions A to D that do not overlap with each other, and regions A to D and partial images 1 to 4 are each in a different division. As described above, partial images 1 to 4 partially overlap with each other, so regions A to D are each included in multiple partial images.

[0073] As in the example of Figure 6, if the partial images are set to overlap, the accuracy of object detection in boundary areas can be improved. Furthermore, if the regions used to determine the update region are set not to overlap, each region can be made smaller, thereby improving efficiency. Note that the number of partial images generated by the partial image generation unit 228A and the number of regions into which the target frame is divided are not limited to the example of Figure 6. Furthermore, the number of partial images and the number of regions into which the target frame is divided may be different.

[0074] The partial image generation unit 228A outputs each generated partial image m to the corresponding partial image detection unit 229-m. Here, partial image 1 is output to partial image detection unit 229-1, partial image 2 is output to partial image detection unit 229-2, partial image 3 is output to partial image detection unit 229-3, and partial image 4 is output to partial image detection unit 229-4.

[0075] 6, the reduced image generation unit 228B generates a reduced image (indicated by a two-dot chain line) by reducing the entire target frame, and outputs the reduced image to the reduced image detection unit 229B. Here, the size of the reduced image is a size that can be input to the object detection model.

[0076] The calculation unit 222 calculates a motion vector V(n) for a target frame (frame n) using a reference frame as a reference, in the same manner as the calculation unit 22 in the first embodiment.

[0077] The update unit 225 performs motion compensation using the motion vector V(n) calculated by the calculation unit 222. Here, the update unit 225 according to this embodiment refers to whether each of a plurality of regions included in the cumulative image was an update region in the previous cumulative image, and changes the target for motion compensation depending on the result. Note that in the present disclosure, "previous" refers to the frame immediately before.

[0078] Specifically, when the reference destination of the motion vector V(n) is a region that was not an update region in the previous cumulative image C(n-1), the updating unit 225 updates the cumulative image C(n) by performing motion compensation based on the previous cumulative image C(n-1) using the motion vector V(n), as in the first embodiment. On the other hand, when the reference destination of the motion vector V(n) is a region that was an update region in the previous cumulative image C(n-1), the updating unit 225 updates the cumulative image C(n) by performing motion compensation based on the previous target frame (frame n-1) using the motion vector V(n).

[0079] That is, unlike the first embodiment, the update unit 225 according to this embodiment does not overwrite the update area using the target frame. For example, for an update area determined based on frame n-1, the cumulative image C(n-1) is not overwritten with that frame n-1. Instead, when updating (generating) the cumulative image C(n) corresponding to the next frame n, motion compensation is performed based on the previous target frame (frame n-1) using the motion vector V(n). This causes the contents of frame n-1 to be reflected in the cumulative image C(n).

[0080] The determination unit 227 determines an updated region in the cumulative image based on a comparison between the target frame and the cumulative image, and outputs the determination result regarding the updated region to the partial image detection unit 229-m and the correction unit 230.

[0081] Each detection unit included in the detection unit 229 uses an object detection model to detect an object from a region corresponding to an updated region in the partial image m or the reduced image, similar to the detection unit 29 in the first embodiment. Each detection unit outputs a detection result including the attribute, reliability, and rectangle information of the object.

[0082] That is, the detection result for partial image m is output from partial image detection unit 229-m, which is responsible for partial image m that is the target for executing object detection processing and includes an area corresponding to the update area determined by determination unit 227. Furthermore, reduced image detection unit 229B outputs a detection result for a reduced image for each frame. Note that with regard to the rectangle information, in both the case of partial image m and the case of a reduced image, information converted into intra-image coordinates of the target frame is output.

[0083] Similar to the correction unit 30 in the first embodiment, the correction unit 230 corrects past detection results of an object using a motion vector in an area in the target frame that corresponds to an area that is not an update area.

[0084] The output unit 232 integrates the detection result of the detection unit 229 and the detection result of the correction unit 230 , outputs the detection result for the target frame, and stores it in the detection result buffer 31 .

[0085] Specifically, output unit 232 obtains object detection results based on partial image m (m = 1, 2, 3, 4) from partial image detection unit 229-m and correction unit 230. For example, assume that determination unit 227 determines that partial image 1 and partial image 2 contain areas corresponding to the update areas, but that partial image 3 and partial image 4 do not contain areas corresponding to the update areas. In this case, output unit 232 obtains the object detection results for partial image 1 from partial image detection unit 229-1 and the object detection results for partial image 2 from partial image detection unit 229-2. Furthermore, output unit 232 obtains the object detection results for partial image 3 and the object detection results for partial image 4 from correction unit 230. As a result, for partial image 1 and partial image 2, which contain areas corresponding to the update areas, object detection results using the object detection model are obtained. Furthermore, for partial image 3 and partial image 4, which do not contain areas corresponding to the update areas, results of correcting past object detection results using motion vectors are obtained.

[0086] Furthermore, the output unit 232 acquires the detection results of the reduced images output from the reduced image detection unit 229 B. Each detection result may include detection results for multiple objects detected from each image.

[0087] As a method for integrating the detection result of the reduced image and the detection result of the partial image and outputting the integrated detection result, it is possible to appropriately apply, for example, the technology described in Patent Document 2. For example, when it is determined that the detection result of the reduced image and the detection result of the partial image are detection results for the same object, the output unit 232 may delete the detection result of the partial image.

[0088] As described above, the object detection device 210 according to the second embodiment, like the first embodiment, determines an update area in the cumulative image based on a comparison between the target frame and the cumulative image, and detects an object using an object detection model in an area in the target frame corresponding to the update area. Meanwhile, when updating the cumulative image, the update unit 225, instead of overwriting the update area with the target frame, refers to whether each of multiple areas included in the cumulative image was an update area in the previous cumulative image. Then, for areas that were update areas in the previous cumulative image, motion compensation is performed based on the previous target frame. Meanwhile, for areas that were not update areas in the previous cumulative image, motion compensation is performed based on the previous cumulative image.

[0089] That is, according to the object detection device 210 of this embodiment, if the area referenced by the motion vector is included in the update area in the previous cumulative image C(n-1), motion compensation is performed based on frame n-1 stored in the input image buffer 23. According to this embodiment, the cumulative image can be updated collectively regardless of whether each area included in the cumulative image is an update area. In other words, there is no need to independently overwrite (update) the cumulative image with frame n immediately after determining the update area, as was done in the first embodiment. Therefore, the amount of data transfer and processing can be further reduced, enabling efficient implementation.

[0090] Furthermore, the object detection device 210 according to this embodiment integrates the detection results of the reduced image and the partial image, making it possible to detect both relatively small objects relative to the target frame, which are difficult to extract when reduced, and relatively large objects that span multiple partial images. Therefore, even when targeting high-resolution moving images, efficient object detection can be performed without reducing the accuracy of object detection.

[0091] In the second embodiment, the detection unit 229 includes the partial image detection unit 229-m and the reduced image detection unit 229B. However, the detection unit 229 may include only the partial image detection unit 229-m.

[0092] Furthermore, in the second embodiment, a case where a partial image detection unit 229-m is provided for each partial image to be generated has been described, but this is not limiting. At least one partial image detection unit may be provided, and a detection instruction for a partial image for which the determination unit 227 has determined that object detection should be performed using an object detection model may be output to the partial image generation unit 228A. In this case, only the partial image that is the target of the detection instruction may be output from the partial image generation unit 228A to the partial image detection unit.

[0093] Third Embodiment Next, a third embodiment will be described. An object detection device 310 according to the third embodiment calculates a motion vector using a reduced image obtained by reducing the entire frame. Then, similar to the first embodiment, the object detection device 310 according to the third embodiment determines whether to skip object detection processing using an object detection model based on a comparison between the reduced image of the target frame and the cumulative image.

[0094] In the object detection device 310 according to the third embodiment, the same components as those in the object detection device 10 according to the first embodiment and the object detection device 210 according to the second embodiment are denoted by the same reference numerals and detailed descriptions thereof will be omitted. Furthermore, the hardware configuration of the object detection device 310 according to the third embodiment is similar to the hardware configuration of the object detection device 10 according to the first embodiment shown in FIG. 1 , and therefore a description thereof will be omitted.

[0095] First, the functional configuration of the object detection device 310 according to the third embodiment will be described. FIG. 7 is a block diagram showing an example of the functional configuration of the object detection device 310 according to the third embodiment. As shown in FIG. 7, the object detection device 310 has, as its functional configuration, an input unit 321, a first calculation unit 322A, an input image buffer 23, a motion vector buffer 24, a reduced image generation unit 328B, a second calculation unit 322B, and a reduced image buffer 23B. Furthermore, the object detection device 210 has, as its functional configuration, an update unit 325, an accumulated image buffer 26, a determination unit 327, a partial image generation unit 328A, a detection unit 329, a correction unit 330, a detection result buffer 31, and an output unit 332. Furthermore, the detection unit 329 has a partial image detection unit 329-m (here, m = 1, 2, 3, 4) and a reduced image detection unit 329B. Each functional configuration is realized by the CPU 11 reading out an object detection program stored in the ROM 12 or the storage 14, expanding it in the RAM 13, and executing it.

[0096] In addition, detailed explanations of the common contents between the functional configuration of the object detection device 310 according to the third embodiment, the functional configuration of the object detection device 10 according to the first embodiment, and the functional configuration of the object detection device 210 according to the second embodiment, in which the last two digits of the numbers in the reference numerals are the same, will be omitted.

[0097] The input unit 321 outputs frame n of the moving image as a target frame to each of the first calculation unit 322A, the partial image generation unit 328A, and the reduced image generation unit 328B.

[0098] The partial image generation unit 328A generates a plurality of partial images m by dividing the target frame into a plurality of parts, similar to the partial image generation unit 228A in the second embodiment. The partial image generation unit 228A also outputs each of the generated partial images m to the corresponding partial image detection unit 329-m.

[0099] Similar to the reduced image generation unit 228B in the second embodiment, the reduced image generation unit 328B generates a reduced image by reducing the entire target frame and outputs the reduced image to the reduced image detection unit 329B. The reduced image generation unit 328B also outputs the generated reduced image to the second calculation unit 322B. Note that the reduced image output to the reduced image detection unit 329B and the reduced image output to the second calculation unit 322B may be the same, or may be images reduced at different ratios.

[0100] The second calculation unit 322B performs motion search between the reduced image of the target frame (frame n) and the reduced image of the reference frame (frame n-1) to calculate a motion vector U(n). The second calculation unit 322B outputs the calculated motion vector U(n) to the first calculation unit 322A and the update unit 325. The second calculation unit 322B also stores the reduced image of frame n in the reduced image buffer 23B to use it as a reduced image of the reference frame when frame n+1 becomes the target frame.

[0101] Like the motion vector V(n), the motion vector U(n) is expressed as a motion vector for each block obtained by dividing the entire reduced image of the target frame. As an example, the motion vector U(n) may be calculated with integer precision for each block of 8 pixels horizontally and 8 pixels vertically. Furthermore, the motion estimation method may be a conventionally known method such as a block matching method or a gradient method, and the method is not limited to this embodiment.

[0102] The first calculation unit 322A calculates a motion vector V(n) based on the reference frame, based on the motion vector U(n) and the target frame (frame n). The accuracy of the motion vector V(n) is higher than that of the motion vector U(n). The first calculation unit 322A stores the calculated motion vector V(n) in the motion vector buffer 24.

[0103] When the target frame is the first frame n (i.e., n=0), the update unit 325 prepares a cumulative image that is the same size as the reduced image generated by the reduced image generation unit 328B and in which all pixels are initialized to predetermined initial values. The determination unit 327 determines an area in frame 0, the target frame, where object detection is to be performed based on predetermined rules, and instructs the detection unit 329 to perform object detection in that area. The update unit 325 then overwrites the initial cumulative image using frame 0 for the area determined by the determination unit 327 to perform object detection, and designates this overwritten cumulative image as cumulative image C(0).

[0104] For example, if a rule is defined to perform object detection for the entire region in frame 0, the update unit 325 obtains the cumulative image C(0) by overwriting the cumulative image for the entire region with the cumulative image of frame 0. As a result, in the first frame n (i.e., n=0), the cumulative image C(0) and frame 0 are the same.

[0105] The update unit 325 also performs motion compensation using the motion vector U(n) calculated by the second calculation unit 322 B. Here, similar to the update unit 225 of the second embodiment, the update unit 325 of this embodiment refers to whether each of a plurality of regions included in the cumulative image was an update region in the previous cumulative image, and changes the target of motion compensation depending on the result.

[0106] Specifically, when the reference destination of the motion vector U(n) is a region that was not an update region in the previous cumulative image C(n-1), the updating unit 325 updates the cumulative image C(n) by performing motion compensation based on the previous cumulative image C(n-1) using the motion vector U(n), as in the first embodiment. On the other hand, when the reference destination of the motion vector U(n) is a region that was an update region in the previous cumulative image C(n-1), the updating unit 325 updates the cumulative image C(n) by performing motion compensation based on the reduced image of the previous target frame (frame n-1) using the motion vector U(n).

[0107] The determination unit 327 determines an updated region in the cumulative image based on a comparison between the reduced image of the target frame and the cumulative image. The determination unit 227 also notifies the partial image detection unit 329-m and the correction unit 330 of the determination result regarding the updated region.

[0108] Each detection unit included in the detection unit 329 uses an object detection model to detect an object from a region corresponding to the updated region in the partial image m or the reduced image, similar to the detection unit 29 in the first embodiment. Each detection unit outputs a detection result including the attribute, reliability, and rectangle information of the object.

[0109] That is, the detection result for partial image m is output from partial image detection unit 329-m, which is responsible for partial image m that is the target for executing object detection processing and includes an area corresponding to the update area determined by determination unit 327. Furthermore, reduced image detection unit 329B outputs a detection result for a reduced image for each frame. Note that with regard to the rectangle information, in both the case of partial image m and the case of a reduced image, information converted into intra-image coordinates of the target frame is output.

[0110] Similar to the correction unit 30 in the first embodiment, the correction unit 330 corrects the past detection results of the object using the motion vector V(n) in the area in the target frame that corresponds to the area that is not the update area.

[0111] The output unit 332 integrates the detection result of the detection unit 329 and the detection result of the correction unit 330 , outputs the detection result for the target frame, and stores it in the detection result buffer 31 .

[0112] As described above, the object detection device 310 according to the third embodiment determines an update area in the cumulative image based on a comparison between a reduced image of a target frame and a cumulative image of the same size as the reduced image. That is, compared to the first and second embodiments, the object detection device 310 according to the third embodiment can grasp the difference between an image that has undergone repeated motion compensation and an actual input image while reducing the amount of data transfer and processing by the size of the image. Therefore, more efficient implementation is possible without reducing the accuracy of object detection.

[0113] In the above embodiments, the reference frame is the frame immediately preceding the target frame, but this is not limiting and any frame earlier than the target frame can be used.

[0114] In addition, the object detection process executed by the CPU in each of the above embodiments by reading the software (program) may be executed by various processors other than the CPU. Examples of processors in this case include PLDs (Programmable Logic Devices) whose circuit configuration can be changed after manufacture, such as FPGAs (Field-Programmable Gate Arrays), and dedicated electrical circuits, which are processors having a circuit configuration designed specifically for executing specific processes, such as ASICs (Application Specific Integrated Circuits). The object detection process may be executed by one of these various processors, or by a combination of two or more processors of the same or different types (e.g., multiple FPGAs, a combination of a CPU and an FPGA, etc.). The hardware structure of these various processors is, more specifically, an electrical circuit that combines circuit elements such as semiconductor elements.

[0115] In the above embodiment, the object detection processing program is pre-stored (installed) in the ROM 12 or the storage 14, but this is not limiting. The program may be provided in a form stored on a non-transitory storage medium such as a CD-ROM (Compact Disk Read Only Memory), a DVD-ROM (Digital Versatile Disk Read Only Memory), or a USB (Universal Serial Bus) memory. The program may also be downloaded from an external device via a network.

[0116] The following additional notes are provided regarding the above-described embodiments.

[0117] (Supplementary Item 1) An object detection device including: a memory; and at least one processor connected to the memory, wherein the processor is configured to: determine an update area in the cumulative image based on a comparison of each frame of a moving image including a plurality of frames as a target frame, calculate a motion vector based on a reference frame for the entire target frame, determine an update area in the cumulative image based on a comparison of the target frame with a frame earlier than the target frame and a cumulative image generated cumulatively based on the motion vector, update the cumulative image by performing motion compensation using the motion vector, and overwrite the update area with the target frame, detect an object using an object detection model in an area in the target frame corresponding to the update area, and correct, using the motion vector, a detection result of the object detected using the object detection model from a frame earlier than the target frame.

[0118] (Supplementary Item 2) A non-transitory recording medium storing a program executable by a computer to perform object detection processing, the object detection processing including: setting each frame of a moving image including a plurality of frames as a target frame, and calculating a motion vector with a reference frame as a reference for the entire target frame; determining an update area in the cumulative image based on a comparison between the target frame and a frame earlier than the target frame and a cumulative image that is cumulatively generated based on the motion vector; updating the cumulative image by performing motion compensation using the motion vector, and overwriting the update area with the target frame; detecting an object using an object detection model in an area in the target frame that corresponds to the update area; and correcting, using the motion vector, a detection result of the object detected using the object detection model from a frame earlier than the target frame.

[0119] 10, 210, 310 Object detection device 11 CPU 12 ROM 13 RAM 14 Storage 15 Input unit 16 Display unit 17 Communication I / F 19 Bus 21, 221, 321 Input unit 22, 222 Calculation unit 23 Input image buffer 23B Reduced image buffer 24 Motion vector buffer 25, 225, 325 Update unit 26 Accumulated image buffer 27, 227, 327 Determination unit 28 Image formation unit 29, 229, 329 Detection unit 30, 230, 330 Correction unit 31 Detection result buffer 32, 232, 332 Output unit 228A, 328A Partial image generation unit 228B, 328B Reduced image generation unit 229-1 to 229-4 Partial image detection unit 229B Reduced image detection unit 322A First calculation unit 322B Second calculation unit C(n-1), C(n), C(n)p Accumulated image X, Y Object

Claims

1. An object detection device comprising: a calculation unit that sets each frame of a moving image including a plurality of frames as a target frame and calculates a motion vector for the entire target frame relative to a reference frame; a determination unit that determines an update area in the cumulative image based on a comparison between the target frame and a cumulative image that is generated cumulatively based on frames earlier than the target frame and the motion vector; an update unit that updates the cumulative image by performing motion compensation using the motion vector and overwrites the update area with the target frame; a detection unit that detects an object using an object detection model in an area in the target frame that corresponds to the update area; and a correction unit that corrects, using the motion vector, a detection result of an object detected using the object detection model from a frame earlier than the target frame in an area in the target frame that corresponds to an area that is not the update area.

2. The object detection device of claim 1, wherein, instead of overwriting the update area using the target frame, when updating the cumulative image, the update unit refers to whether or not each of a plurality of areas included in the cumulative image was the update area in the previous cumulative image, updates the cumulative image by performing the motion compensation based on the previous target frame for areas that were the update area in the previous cumulative image, and updates the cumulative image by performing the motion compensation based on the previous cumulative image for areas that were not the update area in the previous cumulative image.

3. The object detection device according to claim 1, further comprising a reduced image generation unit that generates a reduced image by reducing the entire target frame, wherein the calculation unit calculates the motion vector using the reduced image as the target frame.

4. An object detection method in which a computer performs the following processes: each frame of a moving image including multiple frames is treated as a target frame; motion vectors are calculated for the entire target frame relative to a reference frame; an update area in the cumulative image is determined based on a comparison of the target frame with frames earlier than the target frame and a cumulative image generated cumulatively based on the motion vectors; updating the cumulative image by performing motion compensation using the motion vectors and overwriting the update area with the target frame; detecting an object using an object detection model in an area in the target frame corresponding to the update area; and correcting, using the motion vector, the detection result of the object detected using the object detection model from a frame earlier than the target frame in an area in the target frame corresponding to an area that is not the update area.

Images