Image processing device, image processing method, and program

Abstract

An image processing device (10) according to the present disclosure comprises: a feature point detection tracking unit (12) which detects a feature point in a first frame that is a frame at a first time, tracks the feature point from the first time to a second time after the first time, and calculates a two-dimensional vector that indicates a movement of the feature point; a feature point movement amount calculation unit (13) which calculates a movement amount of the feature point on the basis of the calculated two-dimensional vector; and a determination unit (14) which determines whether the calculated movement amount is equal to or greater than a prescribed threshold value, and when the movement amount is determined as being equal to or greater than the threshold value, outputs, as a cutout frame, a second frame that is a frame at the second time.

Description

[Technical field]

[0001] The present disclosure relates to an image processing device, an image processing method, and a program. [Background technology]

[0002] Recently, in order to improve the efficiency of inspections of large infrastructure structures such as road bridges and road tunnels, an unmanned aerial vehicle (UAV) is flown around the infrastructure structure, photographing the infrastructure structure with a camera mounted on the UAV, and the photographed video is visually checked by an inspector. After checking the video, the inspector usually creates a report of the inspection results, including still images extracted from the video. In this case, it is desirable to extract a frame as a still image from the video every time any part of the infrastructure moves a certain distance in the video so that parts of the infrastructure do not overlap, making it easier to understand the state of the infrastructure structure.

[0003] Various methods have been proposed for outputting a specific frame from a video as a still image. For example, Non-Patent Documents 1 and 2 describe a method for outputting a frame as a still image at the timing when it is detected that an object captured within the angle of view of a video captured with the viewpoint of a camera fixed has moved a certain amount. In addition, Non-Patent Documents 3 and 4 describe a method for detecting a frame as a still image at the timing when it is detected that a specific object, such as an airplane, ship, or bus, is captured within the angle of view, or when it is detected that a specific motion, such as the action of eating, is detected. [Prior art documents] [Non-patent literature]

[0004] [Non-Patent Document 1] Idehara, Shigemichi, Sugita, Fumito, Development of a System for Automatically Extracting and Streaming Keyframes from Videos, Tama University Bulletin of Management and Information Studies, pp. 195-198, 2015. [Non-Patent Document 2] Shinya Takahashi, Sakashi Maeda, Koji Hashimoto, Naoyuki Tsuruta, Hiroyuki Ai, Automatic detection of honeybee waggle dance using frame-by-frame difference video, Bulletin of Engineering, Fukuoka University, pp. 75-80, 2018. [Non-Patent Document 3] Hayato Kobayashi and Keiji Yanai, Automatic Detection of Specific Action Scenes from Television Footage, DEIM Forum, E5-6, 2016. [Non-Patent Document 4] Kazuya Hizume and Keiji Yanai, Analysis of Video Recognition Using Multi-Frame Recognition, Research Report Computer Vision and Image Media, pp. 1-8, 2011. Summary of the Invention [Problem to be solved by the invention]

[0005] The methods described in Non-Patent Documents 1 and 2 are intended for video images captured from a fixed viewpoint. Therefore, when capturing images while moving the viewpoint, all objects captured within the angle of view move, making it difficult to apply the methods described in Non-Patent Documents 1 and 2. In addition, in video images captured of infrastructure structures, the structures are uniform and do not move, making it difficult to apply methods for detecting specific objects or motion, such as the methods described in Non-Patent Documents 3 and 4.

[0006] In consideration of the above-mentioned problems, the objective of the present disclosure is to provide an image processing device, an image processing method, and a program that can output a frame at a timing when a specified part of an object is displaced a specified amount in a moving image of the object captured while moving the viewpoint. [Means for solving the problem]

[0007] In order to solve the above problem, an image processing device according to the present disclosure is an image processing device that outputs, as an extracted frame, a frame at a timing when a predetermined part of an object is displaced by a predetermined amount in a moving image consisting of a plurality of frames, in which an object is photographed while moving a viewpoint, and that includes a feature point detection and tracking unit that detects a feature point in a first frame that is a frame at a first time, tracks the feature point from the first time to a second time that is later than the first time, and calculates a two-dimensional vector that indicates movement of the feature point, a feature point movement amount calculation unit that calculates an amount of movement of the feature point based on the calculated two-dimensional vector, and a determination unit that determines whether the calculated amount of movement is equal to or greater than a predetermined threshold, and if it is determined that the amount of movement is equal to or greater than the threshold, outputs the second frame that is a frame at the second time as the extracted frame.

[0008] In addition, in order to solve the above-mentioned problems, an image processing method according to the present disclosure is an image processing method for outputting, as an extracted frame, a frame at a timing when a predetermined part of an object is displaced by a predetermined amount in a moving image consisting of a plurality of frames, in which the object is photographed while moving a viewpoint, and includes the steps of: detecting a feature point in a first frame, which is a frame at a first time, tracking the feature point from the first time to a second time that is later than the first time, and calculating a two-dimensional vector indicating movement of the feature point; calculating an amount of movement of the feature point based on the calculated two-dimensional vector; determining whether the calculated amount of movement is equal to or greater than a predetermined threshold, and if it is determined that the amount of movement is equal to or greater than the threshold, outputting a second frame, which is a frame at the second time, as the extracted frame.

[0009] In order to solve the above problem, a program according to the present disclosure causes a computer to operate as the image processing device described above. Effect of the Invention

[0010] According to the image processing device, image processing method, and program of the present disclosure, in a moving image of an object captured while moving the viewpoint, a frame can be output at a timing when a specified part of the object has been displaced by a specified amount. [Brief description of the drawings]

[0011] [Figure 1] 1 is a diagram illustrating an example of the configuration of an image processing device according to a first embodiment of the present disclosure. [Diagram 2] 2 is a flowchart showing an example of the operation of the image processing device shown in FIG. [Diagram 3] 2 is a diagram illustrating an example of a hardware configuration of the image processing device illustrated in FIG. 1. [Figure 4] FIG. 11 is a diagram illustrating an example of the configuration of an image processing device according to a second embodiment of the present disclosure. [Diagram 5] 5 is a diagram showing an example of setting of a feature point detection region in a frame by the feature point detection region setting section shown in FIG. 4. [Figure 6] FIG. 13 is a diagram illustrating an example of the configuration of an image processing device according to a third embodiment of the present disclosure. [Figure 7] 7 is a diagram for explaining removal of off-axis characteristic points by the off-axis movement amount removal unit shown in FIG. 6. FIG. [Figure 8] FIG. 11 is a diagram illustrating an example of the configuration of an image processing device according to a fourth embodiment of the present disclosure. [Figure 9] 9 is a diagram for explaining the operation of a movement amount gradient calculation unit shown in FIG. 8. FIG. [Figure 10] 9 is a diagram for explaining frame division and vector correction by the movement amount gradient calculation unit shown in FIG. 8. FIG. [Figure 11A] FIG. 13 is a diagram illustrating an example of the configuration of an image processing device according to a fifth embodiment of the present disclosure. [Figure 11B] FIG. 13 is a diagram illustrating another exemplary configuration of the image processing device according to the fifth embodiment of the present disclosure. [Figure 11C] FIG. 13 is a diagram illustrating yet another exemplary configuration of the image processing device according to the fifth embodiment of the present disclosure. [Figure 11D]FIG. 13 is a diagram illustrating yet another exemplary configuration of the image processing device according to the fifth embodiment of the present disclosure. [Figure 11E] FIG. 13 is a diagram illustrating yet another exemplary configuration of the image processing device according to the fifth embodiment of the present disclosure. [Figure 11F] FIG. 13 is a diagram illustrating yet another exemplary configuration of the image processing device according to the fifth embodiment of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] Hereinafter, embodiments of the present disclosure will be described with reference to the drawings.

[0013] (First embodiment) 1 is a diagram showing a configuration example of an image processing device 10 according to a first embodiment of the present disclosure. The image processing device 10 according to this embodiment cuts out a frame at a timing when a predetermined part of an object such as an infrastructure structure is displaced by a predetermined amount from a moving image captured by an imaging device mounted on a UAV or the like while moving the viewpoint, and outputs the frame.

[0014] As shown in FIG. 1, an image processing device 10 according to this embodiment includes a moving image input unit 11, a feature point detection and tracking unit 12, a feature point movement amount calculation unit 13, a determination unit 14, and a storage unit 15.

[0015] Video input unit 11 receives video captured by a photographing device such as a digital video camera. The video is, for example, an object such as an infrastructure structure photographed by a photographing device mounted on a moving object such as a UAV while moving the viewpoint, and is composed of a plurality of frames (still images) arranged in time series. The resolution and frame rate of the video input to video input unit 11 are arbitrary. Video input unit 11 outputs the input video to feature point detection and tracking unit 12.

[0016] The feature point detection and tracking unit 12 detects feature points of a frame (first frame) at an arbitrary time t1 (first time) of a moving image output from the moving image input unit 11. A feature point is, for example, a pixel having brightness or color information that satisfies a certain condition in a frame (still image). Also, a feature point is, for example, a pixel whose feature amount calculated from brightness or color gradient information around a pixel satisfies a certain condition. The feature point detection and tracking unit 12 detects at least one or more feature points.

[0017] The feature point detection and tracking unit 12 tracks the detected feature points from time t1 to time t2 (second time) after time t1, and calculates a two-dimensional vector indicating the movement of the feature points from time t1 to time t2. When the feature point detection and tracking unit 12 detects a plurality of feature points, it calculates a two-dimensional vector for each of the detected plurality of feature points. The feature points can be tracked by a tracking process based on optical flow, such as the Lucas-Kanade method. When there are a plurality of frames between time t1 and time t2, the feature point detection and tracking unit 12 calculates a two-dimensional vector of the feature points between two frames for each one or a plurality of frames. It is possible to arbitrarily set the number of frames for which the two-dimensional vector is calculated. Note that at least one feature point is detected, and parameters of the tracking process based on optical flow are adjusted so that a two-dimensional vector is calculated for the detected feature points between time t1 and time t2.

[0018] The feature point detection and tracking unit 12 outputs the calculation result of the two-dimensional vector to the feature point movement amount calculation unit 13 .

[0019] The feature point movement amount calculation unit 13 calculates the movement amount of the feature point based on the two-dimensional vector calculated by the feature point detection and tracking unit 12. Specifically, the feature point movement amount calculation unit 13 calculates the sum of the magnitudes of the two-dimensional vectors calculated between time t1 and time t2. When a two-dimensional vector has been calculated for one feature point, the feature point movement amount calculation unit 13 calculates the sum of the magnitudes of the two-dimensional vectors as the movement amount.

[0020] Furthermore, when a two-dimensional vector has been calculated for each of a plurality of feature points, the feature point movement amount calculation unit 13 calculates the sum of the magnitudes of the two-dimensional vectors calculated for all of the feature points between time t1 and time t2. Then, the feature point movement amount calculation unit 13 calculates the sum of the magnitudes of the calculated two-dimensional vectors divided by the number of feature points detected at time t1 as the movement amount of the feature point. In other words, when a plurality of feature points are detected in the frame at time t1, the feature point movement amount calculation unit 13 calculates the average value of the magnitudes of the two-dimensional vectors calculated for each of the plurality of feature points as the movement amount of the feature point.

[0021] For example, if 10 feature points are detected between time t1 and time t2, and a two-dimensional vector is calculated for each of the 10 feature points, the feature point movement amount calculation unit 13 calculates the average value obtained by dividing the sum of the magnitudes of the two-dimensional vectors calculated for each of the 10 feature points by 10, as the movement amount of the feature point.

[0022] The feature point movement amount calculation unit 13 outputs the calculation result of the movement amount of the feature point to the determination unit 14.

[0023] The determination unit 14 determines whether the amount of movement calculated by the feature point movement amount calculation unit 13 is equal to or greater than a predetermined threshold. If the determination unit 14 determines that the calculated amount of movement is equal to or greater than the predetermined threshold, the determination unit 14 outputs the frame at time t2 (the second frame) as an extracted frame to the storage unit 15 to store it. On the other hand, if the determination unit 14 determines that the calculated amount of movement is not equal to or greater than the predetermined threshold, the determination unit 14 does not output the frame at time t2 to the storage unit 15 (does not store it).

[0024] When the amount of movement of the feature point between time t1 and time t2 is equal to or greater than a predetermined threshold, it is considered that a predetermined part of the object within the angle of view has moved by a certain amount between time t1 and time t2. Therefore, when the amount of movement of the feature point between time t1 and time t2 is equal to or greater than a predetermined threshold, the image processing device 10 according to this embodiment outputs and stores the frame at time t2 as an extracted frame in the storage unit 15. In this way, it is possible to output a frame at a timing when a predetermined part of the object has been displaced by a predetermined amount in a moving image of the object captured while moving the viewpoint.

[0025] When the determination unit 14 causes the storage unit 15 to store the frame at time t2 as an extracted frame, it notifies the feature point detection and tracking unit 12 that the frame at time t2 has been stored. Upon being notified that the frame at time t2 has been stored, the feature point detection and tracking unit 12 detects feature points and tracks the detected feature points, treating the frame next to the frame at time t2 as a new frame at time t1.

[0026] Furthermore, if the determination unit 14 does not cause the storage unit 15 to store the frame at time t2 as an extracted frame, it notifies the feature point detection and tracking unit 12 that the frame at time t2 was not stored. When notified that the frame at time t2 was not stored, the feature point detection and tracking unit 12 sets a time t2+Δt, which is a predetermined time Δt after time t2, as a new time t2, and tracks the feature points detected in the frame at time t1 from time t1 to the new time t2. The predetermined time Δt can be set to any time.

[0027] The above-mentioned process is repeated until the new time t1 or the new time t2 exceeds the final time of the input video.

[0028] Next, the operation of the image processing device 10 according to this embodiment will be described.

[0029] FIG. 2 is a flowchart showing an example of the operation of the image processing device 10 according to the present embodiment, and is a diagram for explaining an image processing method by the image processing device 10.

[0030] The video input unit 11 receives input of a video image of an object captured while moving the viewpoint (step S11).

[0031] The feature point detection and tracking unit 12 detects feature points of a frame at an arbitrary time t1 (step S12). Then, the feature point detection and tracking unit 12 tracks the detected feature points from time t1 to time t2, which is x seconds after time t1, and calculates two-dimensional vectors indicating the movement of the feature points.

[0032] The feature point movement amount calculation unit 13 calculates the amount of movement of the feature point based on the two-dimensional vector calculated by the feature point detection and tracking unit 12 (step S13).

[0033] The determination unit 14 determines whether or not the amount of movement calculated by the feature point movement amount calculation unit 13 is equal to or greater than a predetermined threshold value (step S14).

[0034] When it is determined that the calculated movement amount is equal to or greater than a predetermined threshold (step S14: Yes), the determination unit 14 outputs the frame at time t2 as an extracted frame to the storage unit 15 to store it. After the frame at time t2 is stored in the storage unit 15 as the extracted frame, the time t1 is updated, and the process is repeated from step S12.

[0035] If it is determined that the calculated movement amount is not equal to or greater than the predetermined threshold value (step S14: No), the judgment unit 14 judges whether or not a predetermined threshold value T or more has elapsed since time t1, i.e., whether or not the elapsed time from time t1 to time t2 is equal to or greater than T (step S16).

[0036] If it is determined that a predetermined threshold value T or more has elapsed since time t1 (step S16: Yes), the determination unit 14 outputs the frame at time t2 as an extracted frame to the storage unit 15 and stores it (step S15). This makes it possible to prevent the process from terminating without storing any extracted frame even if an error occurs in image recognition.

[0037] If it is determined that a predetermined threshold value T or more has not elapsed since time t1 (step S16: No), the determination unit 14 notifies the feature point detection and tracking unit 12 that the frame at time t2 will not be saved as an extracted frame. This notification causes the feature point detection and tracking unit 12 to calculate a two-dimensional vector up to the new time t2, and the feature point movement amount calculation unit 13 to calculate the amount of movement.

[0038] Next, the hardware configuration of the image processing device 10 according to this embodiment will be described.

[0039] Fig. 3 is a diagram showing an example of a hardware configuration of the image processing device 10 according to the present embodiment. Fig. 3 shows an example of a hardware configuration of the image processing device 10 when the image processing device 10 is configured by a computer capable of executing program instructions. Here, the computer may be a general-purpose computer, a dedicated computer, a workstation, a PC (Personal computer), an electronic notepad, etc. The program instructions may be program code, code segments, etc. for performing necessary tasks.

[0040] 3, the image processing device 10 has a processor 21, a ROM (Read Only Memory) 22, a RAM (Random Access Memory) 23, a storage 24, an input unit 25, a display unit 26, and a communication interface (I / F) 27. Each component is connected to be able to communicate with each other via a bus 29. The processor 21 is specifically a CPU (Central Processing Unit), an MPU (Micro Processing Unit), a GPU (Graphics Processing Unit), a DSP (Digital Signal Processor), a SoC (System on a Chip), or the like, and may be composed of multiple processors of the same type or different types.

[0041] The processor 21 is a control unit that controls each component and executes various arithmetic processing. That is, the processor 21 reads a program from the ROM 22 or the storage 24, and executes the program using the RAM 23 as a working area. The processor 21 controls each component and executes various arithmetic processing according to the program stored in the ROM 22 or the storage 24. In this embodiment, the ROM 22 or the storage 24 stores a program for operating a computer as the image processing device 10 according to the present disclosure. The program is read and executed by the processor 21, thereby realizing each component of the image processing device 10, that is, the feature point detection and tracking unit 12, the feature point movement amount calculation unit 13, and the determination unit 14.

[0042] The program may be provided in a form stored in a non-transitory storage medium such as a CD-ROM (Compact Disk Read Only Memory), a DVD-ROM (Digital Versatile Disk Read Only Memory), a USB (Universal Serial Bus) memory, etc. The program may also be provided in a form downloaded from an external device via a network.

[0043] The ROM 22 stores various programs and various data. The RAM 23 temporarily stores programs or data as a working area. The storage 24 is configured with an HDD (Hard Disk Drive) or an SSD (Solid State Drive) and stores various programs including an operating system and various data. The ROM 22 or the storage 24 stores, for example, frames output as cut-out frames.

[0044] The input unit 25 includes a pointing device such as a mouse, and a keyboard, and is used to perform various input operations.

[0045] The display unit 26 is, for example, a liquid crystal display, and displays various information. The display unit 26 may function as the input unit 25 by adopting a touch panel system.

[0046] The communication interface 27 is an interface for communicating with other devices (for example, a photographing device that photographs a moving image), and is, for example, an interface for a LAN.

[0047] A computer can be suitably used to function as each part of the image processing device 10 described above. Such a computer can be realized by storing a program describing the processing contents for realizing the functions of each part of the image processing device 10 in a storage unit of the computer, and reading and executing the program by a processor of the computer. In other words, the program can cause the computer to function as the image processing device 10 described above. The program can also be recorded on a non-transitory recording medium. The program can also be provided via a network.

[0048] Thus, the image processing device 10 according to this embodiment includes a feature point detection and tracking unit 12, a feature point movement amount calculation unit 13, and a determination unit 14. The feature point detection and tracking unit 12 detects feature points in a frame (first frame) at time t1 (first time), tracks the feature points from time t1 to time t2 (second time) that is later than time t1, and calculates a two-dimensional vector indicating the movement of the feature point. The feature point movement amount calculation unit 13 calculates the movement amount of the feature point based on the calculated two-dimensional vector. The determination unit 14 determines whether the calculated movement amount is equal to or greater than a predetermined threshold, and when it determines that the movement amount is equal to or greater than the threshold, outputs the frame at time t2 (second frame) as an extracted frame.

[0049] If the amount of movement of the feature point between time t1 and time t2 is equal to or greater than a predetermined threshold, it is considered that a predetermined part of the object within the angle of view has moved by a certain amount between time t1 and time t2. Therefore, the image processing device 10 according to this embodiment outputs a frame at time t2 as an extracted frame when the amount of movement of the feature point is equal to or greater than a predetermined threshold. In this way, it is possible to output a frame at a timing when a predetermined part of the object has been displaced by a predetermined amount in a moving image of the object captured while moving the viewpoint.

[0050] Second Embodiment FIG. 4 is a diagram showing an example of the configuration of an image processing device 10A according to the second embodiment of the present disclosure.

[0051] An image processing device 10A shown in FIG. 4 differs from the image processing device 10 shown in FIG. 1 in that a feature point detection region setting section 16 is added.

[0052] 5, the feature point detection area setting unit 16 sets a feature point detection area in which feature points are detected in the frames of the input video image. The feature point detection area setting unit 16 sets the feature point detection area in response to, for example, an input from a user via the input unit 25. The feature point detection and tracking unit 12 detects and tracks feature points in the feature point detection area set by the feature point detection area setting unit 16 in the frames from time t1 to time t2.

[0053] Although FIG. 5 shows an example in which the feature point detection region is rectangular, the feature point detection region is not limited to this, and may be set in any shape.

[0054] As described above, in this embodiment, the image processing device 10A further includes a feature point detection area setting unit 16 that sets a feature point detection area in which feature points are detected in a frame. Then, the feature point detection and tracking unit 12 detects and tracks feature points in the set feature point detection area.

[0055] Setting a feature point detection area can prevent a feature point detected in a frame at time t1 from moving out of the frame until time t2 due to movement of the viewpoint of the imaging device. For example, if a feature point is detected in a pixel near the edge of a frame at time t1, there is a high possibility that the feature point will not be captured in a frame at a later time due to movement of the viewpoint of the imaging device. For this reason, it is effective to set the center of the frame, or, if the imaging device is moving in a fixed direction while capturing an image, the area where the target object is present in the next frame, as the feature point detection area.

[0056] (Third embodiment) FIG. 6 is a diagram showing an example of the configuration of an image processing device 10B according to the third embodiment of the present disclosure.

[0057] The image processing device 10B shown in Fig. 6 differs from the image processing device 10A shown in Fig. 4 in that the feature point movement amount calculation unit 13 is changed to a feature point movement amount calculation unit 13B. The feature point movement amount calculation unit 13B includes an off-axis movement amount removal unit 131.

[0058] As described above, when the feature point detection and tracking unit 12 detects a plurality of feature points in a frame at time t1, it calculates a two-dimensional vector for each of the detected feature points. The deviation movement amount removal unit 131 divides the sum of the magnitudes of the two-dimensional vectors calculated for each of the plurality of feature points by the number of feature points detected at time t1. That is, the deviation movement amount removal unit 131 calculates the average value of the magnitudes of the two-dimensional vectors calculated for each of the plurality of feature points. Then, the deviation movement amount removal unit 131 determines whether or not there is a feature point (hereinafter referred to as an "outlier feature point") whose magnitude of the two-dimensional vector exceeds a predetermined range (for example, within the average value ± several percent) with respect to the calculated average value among the plurality of feature points. If there is an outlier feature point, the deviation movement amount removal unit 131 calculates the average value of the magnitudes of the two-dimensional vectors calculated for feature points excluding the outlier feature point among the plurality of detected feature points as the movement amount of the feature point.

[0059] As described above, in this embodiment, the feature point movement amount calculation unit 13B (outlier movement amount removal unit 131) calculates the average value of the magnitudes of the two-dimensional vectors calculated for each of the multiple feature points. Then, when there is an outlier feature point among the multiple feature points whose calculated two-dimensional vector magnitude exceeds a predetermined range with respect to the average value, the feature point movement amount calculation unit 13B calculates, as the movement amount of the feature point, the average value of the magnitudes of the two-dimensional vectors calculated for the multiple feature points excluding the outlier feature points.

[0060] For example, as shown in Fig. 7, if a flying object such as a bird crosses the frame at a speed faster than the viewpoint of the photographing device, when optical flow processing is performed on the flying object, the two-dimensional vector of the feature point corresponding to the flying object becomes large, and erroneous processing may occur as if the viewpoint of the photographing device moved quickly. As in this embodiment, by excluding outlying feature points whose magnitude of the two-dimensional vector exceeds a predetermined range with respect to the average value of the magnitudes of the two-dimensional vectors calculated for a plurality of feature points, it is possible to exclude the feature points caused by the flying object as described above from the calculation of the movement amount.

[0061] In the present embodiment, the feature point movement amount calculation unit 13 of the image processing device 10A according to the second embodiment is replaced with the feature point movement amount calculation unit 13B, but the present invention is not limited to this. The feature point movement amount calculation unit 13 of the image processing device 10 according to the first embodiment may be replaced with the feature point movement amount calculation unit 13B.

[0062] (Fourth embodiment) FIG. 8 is a diagram showing an example of the configuration of an image processing device 10C according to the fourth embodiment of the present disclosure.

[0063] The image processing device 10C shown in Fig. 8 differs from the image processing device 10B shown in Fig. 6 in that the feature point movement amount calculation unit 13B is changed to a feature point movement amount calculation unit 13C. The feature point movement amount calculation unit 13C differs from the feature point movement amount calculation unit 13B in that a movement amount gradient calculation unit 132 is added.

[0064] When calculating the sum of the magnitudes of the two-dimensional vectors of the feature points in the feature point movement amount calculation unit 13C, the movement amount gradient calculation unit 132 sets the magnitudes of vector components other than the vector components in a specific direction to 0. For example, as shown in Fig. 9, when the viewpoint during shooting moves downward from the top to the bottom of the frame, the movement amount gradient calculation unit 132 sets the magnitudes of vector components other than the vector components in the same direction or the opposite direction to the movement direction of the viewpoint to 0.

[0065] That is, in this embodiment, the feature point movement amount calculation unit 13C calculates the movement amount using only vector components in a specific direction among the two-dimensional vector components of the feature point. In this way, even if the viewpoint of the card device moves unintentionally due to factors such as wind or unstable equipment installation during outdoor shooting, the influence of the unintentional movement of the viewpoint can be reduced and the movement amount of the feature point in only the direction in which the viewpoint moves can be more accurately calculated.

[0066] In addition, the movement gradient calculation unit 132 may divide the frame into a plurality of regions and adjust the magnitude of the two-dimensional vector of the feature point according to the region that the two-dimensional vector is included in. For example, as shown in Fig. 10, the movement gradient calculation unit 132 may perform a process of dividing the frame into two, upper and lower, leaving the magnitude of the two-dimensional vector included in the upper half region unchanged and multiplying the magnitude of the two-dimensional vector included in the lower half region by a predetermined coefficient α.

[0067] For example, when an object is photographed at a constant elevation angle, the object actually captured in the frame may be farther away at the bottom of the frame than at the top of the frame. In this case, the magnitude of the two-dimensional vector at the bottom of the frame may be smaller than that at the top of the frame. Therefore, by dividing the frame into multiple regions and adjusting the magnitude of the two-dimensional vector according to the region in which it is included, the influence of photographing at a constant elevation angle can be reduced and the amount of movement can be calculated more accurately.

[0068] In FIG. 10, an example is described in which the frame is divided into upper and lower halves, but the method of dividing the frame is not limited to this, and the frame may be divided into any number of parts and in any direction according to the conditions at the time of shooting.

[0069] In the present embodiment, the feature point movement amount calculation unit 13B of the image processing device 10B according to the third embodiment is replaced by the feature point movement amount calculation unit 13C, but the present invention is not limited to this. The feature point movement amount calculation unit 13 of the image processing device 10 according to the first embodiment or the image processing device 10A according to the second embodiment may be replaced by the feature point movement amount calculation unit 13C.

[0070] Fifth embodiment Fig. 11A is a diagram showing an example of the configuration of an image processing device 10D according to a fifth embodiment of the present disclosure, In Fig. 11A, the same components as those in Fig. 1 are denoted by the same reference numerals, and description thereof will be omitted.

[0071] 11A, an image processing device 10D according to this embodiment includes a moving image input unit 11, a feature point detection and tracking unit 12, a feature point movement amount calculation unit 13D, a determination unit 14, a storage unit 15, and a size calculation unit 17. The image processing device 10D shown in FIG. 11A differs from the image processing device 10 shown in FIG. 1 in that a size calculation unit 17 is added and that the feature point movement amount calculation unit 13 is changed to a feature point movement amount calculation unit 13D.

[0072] The size calculation unit 17 receives a moving image from the moving image input unit 11. The size calculation unit 17 calculates the horizontal size and vertical size (first direction and second direction) of frames (still images) constituting the input moving image. The size calculation unit 17 outputs the calculation result to the feature point movement amount calculation unit 13D.

[0073] Similar to feature point movement amount calculation unit 13, feature point movement amount calculation unit 13D calculates the amount of movement of a feature point based on the two-dimensional vector calculated by feature point detection and tracking unit 12. Here, feature point movement amount calculation unit 13D normalizes the amount of movement of a feature point in accordance with the ratio between the horizontal size and the vertical size of the frame calculated by size calculation unit 17.

[0074] Specifically, the feature point movement amount calculation unit 13D calculates a composite vector of the two-dimensional vectors calculated between time t1 and time t2. When a two-dimensional vector is calculated for one feature point, the feature point movement amount calculation unit 13D sets the two-dimensional vector as a composite vector.

[0075] Furthermore, when a two-dimensional vector has been calculated for each of a plurality of feature points, the feature point movement amount calculation unit 13D synthesizes the two-dimensional vectors calculated for all feature points between time t1 and time t2, and then divides the magnitude of the vector by the number of feature points detected at time t1 to obtain a synthesized vector.

[0076] For example, if 10 feature points are detected between time t1 and time t2, and two-dimensional vectors are calculated for each of the 10 feature points, the feature point movement amount calculation unit 13D combines the two-dimensional vectors calculated for each of the 10 feature points, and then divides the magnitude of the combined vector by 10 to calculate a combined vector.

[0077] The feature point movement amount calculation unit 13D calculates the movement amount of the feature point from the calculated resultant vector according to the following formula (1).

[0078]

number

[0079] In formula (1), a is the magnitude of the horizontal component (horizontal direction) of the composite vector, b is the magnitude of the vertical component (vertical direction) of the composite vector, w is the horizontal size of the frame (still image), and h is the vertical size of the frame (still image). By dividing the magnitude of the horizontal component of the composite vector by the horizontal size of the frame and dividing the vertical component of the composite vector by the vertical size of the frame, the amount of movement can be standardized according to the ratio of the horizontal size and the vertical size of the frame. By standardizing the amount of movement according to the ratio of the horizontal size and the vertical size of the frame, the timing at which a predetermined part of the object in the frame is displaced by a predetermined amount can be detected without being affected by the horizontal size and the vertical size of the frame. The feature point movement amount calculation unit 13D outputs the calculation result of the amount of movement of the feature point to the determination unit 14.

[0080] In this embodiment, the image processing device 10D has been described using an example in which a size calculation unit 17 is added to the image processing device 10 according to the first embodiment and the feature point movement amount calculation unit 13 is changed to a feature point movement amount calculation unit 13D, but the present invention is not limited to this.

[0081] For example, as shown in FIG. 11B, an image processing device 10D may be configured by adding a size calculation unit 17 to the image processing device 10A according to the second embodiment and changing the feature point movement amount calculation unit 13 to a feature point movement amount calculation unit 13D.

[0082] 11C, an image processing device 10D may be configured by adding a size calculation unit 17 to the image processing device 10B according to the third embodiment and changing the feature point movement amount calculation unit 13B to a feature point movement amount calculation unit 13D. In this case, the feature point movement amount calculation unit 13D includes a deviation movement amount removal unit 131D as shown in FIG.

[0083] The deviant movement amount removal unit 131D determines whether or not there is a feature point (deviant feature point) among the multiple feature points whose magnitude of a two-dimensional vector exceeds a predetermined range (for example, within the average value ± a few percent) of the average value of the magnitudes of the two-dimensional vectors calculated for each of the multiple feature points. If it is determined that there is an deviant feature point, the feature point movement amount calculation unit 13D calculates a composite vector from the two-dimensional vectors calculated for the feature points excluding the deviant feature points among the multiple detected feature points. Then, the feature point movement amount calculation unit 13D calculates the movement amount of the feature point from the calculated composite vector based on the above-mentioned formula (1).

[0084] 11D, an image processing device 10D may be configured by adding a size calculation unit 17 to the image processing device 10C according to the fourth embodiment and changing the feature point movement amount calculation unit 13C to a feature point movement amount calculation unit 13D. In this case, the feature point movement amount calculation unit 13D includes a deviation movement amount removal unit 131D and a movement amount gradient calculation unit 132D, as shown in FIG.

[0085] When the feature point movement amount calculation unit 13D calculates a composite vector of two-dimensional vectors of the feature point, the movement amount gradient calculation unit 132D sets vector components other than the vector components in a specific direction to 0. For example, as shown in FIG. 9, when the viewpoint during shooting moves downward from the top to the bottom of the frame, the movement amount gradient calculation unit 132D sets vector components other than the vector components in the same direction or the opposite direction to the viewpoint movement direction to 0. The feature point movement amount calculation unit 13D calculates a composite vector from the two-dimensional vectors in which vector components other than the vector components in the specific direction are set to 0, and calculates the movement amount of the feature point from the calculated composite vector based on the above-mentioned formula (1). That is, in this embodiment, the feature point movement amount calculation unit 13D calculates a composite vector using only the vector components in a specific direction among the two-dimensional vector components of the feature point.

[0086] The movement amount gradient calculation unit 132D may divide the frame into a plurality of regions and adjust the magnitude of the two-dimensional vector of the feature point according to the region that the two-dimensional vector is included in. For example, as shown in Fig. 10, the movement amount gradient calculation unit 132D may perform a process of dividing the frame into two, upper and lower, leaving the magnitude of the two-dimensional vector included in the upper half region as it is, and multiplying the magnitude of the two-dimensional vector included in the lower half region by a predetermined coefficient α.

[0087] Moreover, as shown in FIG. 11E, an image processing device 10D according to this embodiment may be configured such that, compared to the image processing device 10D shown in FIG. 11B, the feature point detection and tracking unit 12 is changed to a feature point detection and tracking unit 12D.

[0088] The feature point detection and tracking unit 12D calculates two-dimensional vectors by optical flow for all pixels in an area specified as a feature point detection area by the feature point detection area setting unit 16 between time t1 and time t2. The feature point movement amount calculation unit 13D calculates the movement amount of the feature point based on the two-dimensional vector calculated by the feature point detection and tracking unit 12D. Specifically, the feature point movement amount calculation unit 13D calculates a composite vector of the two-dimensional vectors of all pixels calculated between time t1 and time t2, and calculates the movement amount of the feature point from the calculated composite vector based on the above-mentioned formula (1). By calculating the two-dimensional vectors for all pixels, it becomes possible to deal with sudden changes in the movement amount due to the loss of a tracked feature point and erroneous recognition of a feature point in a method of tracking a specific feature point.

[0089] Also, as shown in FIG. 11F, an image processing device 10D according to this embodiment may have a configuration in which the feature point movement amount calculation section 13D in the image processing device 10D shown in FIG. 11E includes a movement gradient calculation section 132D.

[0090] In addition, the image processing device 10A according to the second embodiment, the image processing device 10B according to the third embodiment, the image processing device 10C according to the fourth embodiment, and the image processing device 10D according to the fifth embodiment can also be configured by a computer having the hardware configuration described with reference to FIG. 3.

[0091] The following supplementary notes are further disclosed regarding the above embodiment.

[0092] [Additional note 1] An image processing device that cuts out a frame at a timing when a predetermined part of an object is displaced by a predetermined amount from a moving image composed of a plurality of frames, the moving image being obtained by photographing an object while moving a viewpoint, and outputs the cut-out frame, Memory, A control unit connected to the memory; Equipped with The control unit is detecting a feature point in a first frame, which is a frame at a first time, tracking the feature point from the first time to a second time later than the first time, and calculating a two-dimensional vector indicating a movement of the feature point; Calculating the amount of movement of the feature point based on the calculated two-dimensional vector; an image processing device that determines whether the calculated amount of movement is greater than or equal to a predetermined threshold, and if it determines that the amount of movement is greater than or equal to the threshold, outputs a second frame, which is a frame at the second time, as the extracted frame.

[0093] [Additional note 2] In the image processing device according to claim 1, The control unit is A feature point detection area is set in the frame, the feature point being an area for detecting the feature point; An image processing device that detects and tracks the feature points in the set feature point detection area.

[0094] [Additional note 3] In the image processing device according to claim 1 or 2, The control unit is When a plurality of feature points are detected in the first frame, the two-dimensional vector is calculated for each of the detected plurality of feature points; The image processing device calculates, as the amount of movement, an average value of magnitudes of two-dimensional vectors calculated for each of the plurality of feature points.

[0095] [Additional note 4] In the image processing device according to claim 1 or 2, The control unit is When a plurality of feature points are detected in the first frame, the two-dimensional vector is calculated for each of the detected plurality of feature points; an image processing device that calculates an average value of the magnitudes of the two-dimensional vectors calculated for each of the plurality of feature points, and if there are outlying feature points among the plurality of feature points, which are feature points whose magnitudes of the calculated two-dimensional vectors exceed a predetermined range with respect to the average value, calculates as the amount of movement the average value of the magnitudes of the two-dimensional vectors calculated for feature points among the plurality of feature points excluding the outlying feature points.

[0096] [Additional note 5] In the image processing device according to any one of claims 1 to 4, The control unit calculates the amount of movement by using only vector components of the two-dimensional vector in a specific direction.

[0097] [Additional note 6] In the image processing device according to any one of claims 1 to 5, The control unit divides the frame into a plurality of regions, and adjusts a size of the two-dimensional vector according to the region in which the two-dimensional vector is included.

[0098] [Additional note 7] In the image processing device according to claim 1, A size calculation unit is further provided to calculate a size of the frame in a first direction and a size of the frame in a second direction perpendicular to the first direction, The feature point movement amount calculation unit normalizes the movement amount in accordance with a ratio between the size of the frame in the first direction and the size of the frame in the second direction detected by the size calculation unit.

[0099] [Additional note 8] 1. An image processing method for cutting out a frame at a timing when a predetermined part of an object is displaced by a predetermined amount from a moving image composed of a plurality of frames, the moving image being obtained by photographing an object while moving a viewpoint, and outputting the cutout frame, the method comprising: detecting a feature point in a first frame, which is a frame at a first time, tracking the feature point from the first time to a second time later than the first time, and calculating a two-dimensional vector indicating a movement of the feature point; Calculating the amount of movement of the feature point based on the calculated two-dimensional vector; An image processing method comprising: determining whether the calculated amount of movement is greater than or equal to a predetermined threshold; and, if it is determined that the amount of movement is greater than or equal to the threshold, outputting a second frame, which is a frame at the second time, as the extracted frame.

[0100] [Additional note 9] A non-transitory storage medium storing a program executable by a computer, the non-transitory storage medium storing the program causing the computer to operate as the image processing device described in any one of appended claims 1 to 7.

[0101] Although the above-mentioned embodiment has been described as a representative example, it is obvious to those skilled in the art that many modifications and substitutions can be made within the spirit and scope of the present disclosure. Therefore, the present invention should not be interpreted as being limited by the above-mentioned embodiment, and various modifications or changes can be made without departing from the scope of the claims. For example, it is possible to combine multiple building blocks shown in the configuration diagram of the embodiment into one, or to divide one building block. [Explanation of symbols]

[0102] 10, 10A, 10B, 10C Image processing device 11 Video input section 12 Feature point detection and tracking section 13, 13B, 13C, 13D Feature point movement amount calculation unit 14 Judgment section 15 Preservation Department 16 Feature point detection area setting section 17 Size Calculation Section 131,131D Deviated movement amount removal part 132,132D Movement amount slope calculation part 21 Processors 22 ROM 23 RAM 24. Storage 25 Input section 26 Display section 27 Communication I / F 29 Bus

Claims

1. An image processing device that cuts out a frame at a timing when a predetermined part of an object is displaced by a predetermined amount from a moving image composed of a plurality of frames, the moving image being obtained by photographing an object while moving a viewpoint, and outputs the cut-out frame, a feature point detection and tracking unit that detects feature points in a first frame that is a frame at a first time, tracks the feature points from the first time to a second time that is later than the first time, and calculates two-dimensional vectors that indicate movements of the feature points; a feature point movement amount calculation unit that calculates a movement amount of the feature point based on the calculated two-dimensional vector; a determination unit that determines whether the calculated movement amount is equal to or greater than a predetermined threshold, and when it is determined that the movement amount is equal to or greater than the threshold, outputs a second frame that is a frame at the second time as the extracted frame, When the determination unit determines that the amount of movement is not greater than or equal to the threshold, it determines whether or not a predetermined threshold T or more has elapsed since the first time, and when it determines that the predetermined threshold T or more has elapsed since the first time, it outputs the second frame as the cut-out frame.

2. 2. The image processing device according to claim 1, a feature point detection area setting unit that sets a feature point detection area in the frame, the feature point being an area where the feature point is to be detected, The feature point detection and tracking unit detects and tracks the feature points in the set feature point detection area.

3. 2. The image processing device according to claim 1, when a plurality of feature points are detected in the first frame, the feature point detection and tracking unit calculates the two-dimensional vector for each of the detected feature points; The feature point movement amount calculation unit calculates, as the movement amount, an average value of magnitudes of two-dimensional vectors calculated for each of the plurality of feature points.

4. 2. The image processing device according to claim 1, when a plurality of feature points are detected in the first frame, the feature point detection and tracking unit calculates the two-dimensional vector for each of the detected feature points; the feature point movement amount calculation unit calculates an average value of the magnitudes of the two-dimensional vectors calculated for each of the plurality of feature points, and if there are outlying feature points among the plurality of feature points, which are feature points whose magnitudes of the calculated two-dimensional vectors exceed a predetermined range with respect to the average value, calculates as the movement amount the average value of the magnitudes of the two-dimensional vectors calculated for feature points among the plurality of feature points excluding the outlying feature points.

5. 2. The image processing device according to claim 1, The feature point movement amount calculation unit calculates the movement amount by using only vector components of the two-dimensional vector in a specific direction.

6. 2. The image processing device according to claim 1, The feature point movement amount calculation unit divides the frame into a plurality of regions, and adjusts a size of the two-dimensional vector according to the region to which the two-dimensional vector belongs.

7. 2. The image processing device according to claim 1, a size calculation unit that calculates a size of the frame in a first direction and a size of the frame in a second direction perpendicular to the first direction; The feature point movement amount calculation unit normalizes the movement amount in accordance with a ratio between the size of the frame in the first direction and the size of the frame in the second direction detected by the size calculation unit.

8. An image processing method executed by an image processing device, which cuts out a frame at a timing when a predetermined part of an object is displaced by a predetermined amount from a moving image composed of a plurality of frames, the moving image being obtained by photographing an object while moving a viewpoint, and outputs the cut frame, the image processing method comprising: detecting a feature point in a first frame, which is a frame at a first time, tracking the feature point from the first time to a second time later than the first time, and calculating a two-dimensional vector indicating a movement of the feature point; calculating a movement amount of the feature point based on the calculated two-dimensional vector; determining whether the calculated movement amount is equal to or greater than a predetermined threshold, and when it is determined that the movement amount is equal to or greater than the threshold, outputting a second frame that is a frame at the second time as the extracted frame; When it is determined that the amount of movement is not equal to or greater than the threshold, it is determined whether or not a predetermined threshold T or more has elapsed since the first time, and when it is determined that the predetermined threshold T or more has elapsed since the first time, the image processing method outputs the second frame as the cut-out frame.

9. A program for causing a computer to operate as the image processing device according to claim 1.

Images