Image processing device, image processing method, and program

Abstract

An image processing device (10) according to the present disclosure comprises: a feature detecting unit (12) which detects features in each of a first frame which is a frame at a first time serving as a starting point and a second frame which is a frame at a second time at which a prescribed amount of time has elapsed since the first time; a superposing unit (13) which superposes the first frame and the second frame such that the distance between associated features in the first frame and the second frame is shortest; and an overlap ratio determining unit (14) which determines whether the ratio, with respect to the area of the first frame, of the area of an overlapping region in which the first frame and the second frame overlap is at most a prescribed threshold, and if it is determined that the ratio is at most the prescribed threshold, outputs the second frame as an extracted frame.

Description

[Technical field]

[0001] The present disclosure relates to an image processing device, an image processing method, and a program. [Background technology]

[0002] Recently, in order to improve the efficiency of inspections of large infrastructure structures such as road bridges and road tunnels, an unmanned aerial vehicle (UAV) is flown around the infrastructure structure, photographing the infrastructure structure with a camera mounted on the UAV, and the photographed video is visually checked by an inspector. After checking the video, the inspector usually creates a report of the inspection results, including still images extracted from the video. In this case, it is desirable to extract a frame as a still image from the video every time any part of the infrastructure moves a certain distance in the video so that parts of the infrastructure do not overlap, making it easier to understand the state of the infrastructure structure.

[0003] Various methods have been proposed for outputting a specific frame from a video as a still image. For example, Non-Patent Documents 1 and 2 describe a method for outputting a frame as a still image at the timing when it is detected that an object captured within the angle of view of a video captured with the viewpoint of a camera fixed has moved a certain amount. In addition, Non-Patent Documents 3 and 4 describe a method for detecting a frame as a still image at the timing when it is detected that a specific object, such as an airplane, ship, or bus, is captured within the angle of view, or when it is detected that a specific motion, such as the action of eating, is detected. [Prior art documents] [Non-patent literature]

[0004] [Non-Patent Document 1] Idehara, Shigemichi, Sugita, Fumito, Development of a System for Automatically Extracting and Streaming Keyframes from Videos, Tama University Bulletin of Management and Information Studies, pp. 195-198, 2015. [Non-Patent Document 2] Shinya Takahashi, Sakashi Maeda, Koji Hashimoto, Naoyuki Tsuruta, Hiroyuki Ai, Automatic detection of honeybee waggle dance using frame-by-frame difference video, Bulletin of Engineering, Fukuoka University, pp. 75-80, 2018. [Non-Patent Document 3] Hayato Kobayashi and Keiji Yanai, Automatic Detection of Specific Action Scenes from Television Footage, DEIM Forum, E5-6, 2016. [Non-Patent Document 4] Kazuya Hizume and Keiji Yanai, Analysis of Video Recognition Using Multi-Frame Recognition, Research Report Computer Vision and Image Media, pp. 1-8, 2011. Summary of the Invention [Problem to be solved by the invention]

[0005] The methods described in Non-Patent Documents 1 and 2 are intended for video images captured from a fixed viewpoint. Therefore, when capturing images while moving the viewpoint, all objects captured within the angle of view move, making it difficult to apply the methods described in Non-Patent Documents 1 and 2. In addition, in video images captured of infrastructure structures, the structures are uniform and do not move, making it difficult to apply methods for detecting specific objects or motion, such as the methods described in Non-Patent Documents 3 and 4.

[0006] In consideration of the above-mentioned problems, the objective of the present disclosure is to provide an image processing device, an image processing method, and a program that can output a frame at a timing when a specified part of an object is displaced a specified amount in a moving image of the object captured while moving the viewpoint. [Means for solving the problem]

[0007] In order to solve the above problem, an image processing device according to the present disclosure is an image processing device that outputs, as an excised frame, a frame at a timing when a predetermined part of an object is displaced by a predetermined amount in a moving image composed of a plurality of frames and captured while moving a viewpoint, and that is equipped with a feature point detection unit that detects feature points in each of a first frame which is a frame at a first time that serves as a base point and a second frame which is a frame at a second time a predetermined time after the first time, an overlay unit that overlays the first frame and the second frame so that a distance between corresponding feature points in the first frame and the second frame is minimized, and an overlap rate determination unit that determines whether a ratio of an area of ​​an overlap region where the first frame and the second frame overlap to an area of ​​the first frame is equal to or less than a predetermined threshold, and outputs the second frame as the excised frame if it is determined that the ratio is equal to or less than the predetermined threshold.

[0008] In addition, in order to solve the above-mentioned problems, an image processing method according to the present disclosure is an image processing method for outputting, as an excised frame, a frame at a timing when a predetermined part of an object is displaced by a predetermined amount in a moving image consisting of a plurality of frames, in which an object is photographed while moving a viewpoint, the image processing method including the steps of: detecting feature points in a first frame which is a frame at a first time and a second frame which is a frame at a second time a predetermined time after the first time; superimposing the first frame and the second frame so that a distance between corresponding feature points in the first frame and the second frame is minimized; determining whether a ratio of an area of ​​an overlapping region where the first frame and the second frame overlap to an area of ​​the first frame is equal to or less than a predetermined threshold, and outputting the second frame as the excised frame if it is determined that the ratio is equal to or less than the predetermined threshold.

[0009] In order to solve the above problem, a program according to the present disclosure causes a computer to operate as the image processing device described above. Effect of the Invention

[0010] According to the image processing device, image processing method, and program of the present disclosure, in a moving image of an object captured while moving the viewpoint, a frame can be output at a timing when a specified part of the object has been displaced by a specified amount. [Brief description of the drawings]

[0011] [Figure 1] 1 is a diagram illustrating an example of the configuration of an image processing device according to a first embodiment of the present disclosure. [Figure 2A] 2 is a diagram showing an example of a detection result of feature points in a base time frame by a base time feature point detection unit shown in FIG. 1. FIG. [Figure 2B] 2 is a diagram showing an example of a detection result of feature points in elapsed time frames by the elapsed time feature point detection unit shown in FIG. 1. [Diagram 3] 2 is a diagram for explaining the superimposition of a base time frame and an elapsed time frame by the superimposition section shown in FIG. 1. FIG. [Figure 4] 2 is a diagram for explaining the determination of an overlap rate by an overlap rate determination unit shown in FIG. 1; FIG. [Diagram 5] 2 is a flowchart showing an example of the operation of the image processing device shown in FIG. [Figure 6] 2 is a diagram illustrating an example of a hardware configuration of the image processing device illustrated in FIG. 1. [Figure 7] FIG. 11 is a diagram illustrating an example of the configuration of an image processing device according to a second embodiment of the present disclosure. [Figure 8A] FIG. 8 is a diagram showing an example of setting of a feature point detection area in a base time frame by the feature point detection area setting section 1231 shown in FIG. 7. [Figure 8B] 8 is a diagram showing an example of setting of a feature point detection region in an elapsed time frame by the feature point detection region setting section 1241 shown in FIG. 7. FIG. [Figure 9] FIG. 13 is a diagram illustrating an example of the configuration of an image processing device according to a third embodiment of the present disclosure. [Figure 10] 10 is a diagram for explaining movement / rotation / scaling of an elapsed time frame performed by the movement / rotation / scaling unit shown in FIG. 9. FIG. [Figure 11] FIG. 11 is a diagram illustrating an example of the configuration of an image processing device according to a fourth embodiment of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] Hereinafter, embodiments of the present disclosure will be described with reference to the drawings.

[0013] (First embodiment) 1 is a diagram showing a configuration example of an image processing device 10 according to a first embodiment of the present disclosure. The image processing device 10 according to this embodiment cuts out a frame at a timing when a predetermined part of an object such as an infrastructure structure is displaced by a predetermined amount from a moving image captured by an imaging device mounted on a UAV or the like while moving the viewpoint, and outputs the frame.

[0014] As shown in FIG. 1, an image processing apparatus 10 according to this embodiment includes a moving image input unit 11, a feature point detection unit 12, an overlapping unit 13, an overlapping rate determination unit 14, and a storage unit 15.

[0015] Video input unit 11 receives video captured by a photographing device such as a digital video camera. The video is, for example, an object such as an infrastructure structure photographed by a photographing device mounted on a moving object such as a UAV while moving the viewpoint, and is composed of a plurality of frames (still images) arranged in time series. The resolution and frame rate of the video input to video input unit 11 are arbitrary. Video input unit 11 outputs the input video to feature point detection unit 12.

[0016] The feature point detection unit 12 detects feature points in a base time frame (first frame) that is a frame at a base time (first time) that is a base time, and an elapsed time frame (second frame) that is a frame at an elapsed time (second time) that is a predetermined time after the base time, in the moving image output from the moving image input unit 11. The feature point detection unit 12 includes a base time frame storage unit 121, an elapsed time frame storage unit 122, a base time feature point detection unit 123, and an elapsed time feature point detection unit 124.

[0017] The base time frame storage unit 121 receives a moving image from the video input unit 11. The base time frame storage unit 121 sets an arbitrary time t1 as a base time, and stores frames constituting the moving image at time t1 as base time frames. The base time frame storage unit 121 outputs the moving image input from the video input unit 11 to the elapsed time frame storage unit 122.

[0018] The elapsed time frame storage unit 122 receives an input of a moving image from the base time frame storage unit 121. The elapsed time frame storage unit 122 stores frames constituting the moving image at elapsed times that are a predetermined time Δt after time t1 (base time) as elapsed time frames. The elapsed time frame storage unit 122 may store any number of elapsed time frames. For example, the elapsed time frame storage unit 122 may store frames at multiple times that are different amounts of time elapsed from the base time, such as t1+Δt, t1+Δt1, t1+Δt2, . . . as elapsed time frames. The elapsed time frame storage unit 122 can store multiple elapsed time frames with different elapsed times from the base time as long as there are frames of the moving image.

[0019] As described above, an elapsed time frame is a frame at a time when a predetermined time Δt has elapsed from the base time. By defining an elapsed time frame as a frame chronologically later than the base time frame, it is possible to prevent a frame at a time before the base time from being saved as an elapsed time frame. This makes it possible to process frames in the time series in which they were captured. Furthermore, by processing frames chronologically, it is not necessary to process frames at a time before the base time, so that the amount of calculation can be reduced.

[0020] As shown in Fig. 2A, the base time feature point detection unit 123 detects feature points in the base time frame stored by the base time frame storage unit 121. A feature point is, for example, a pixel having brightness or color information that satisfies a certain condition in a frame (still image). Also, a feature point is, for example, a pixel whose feature amount calculated from brightness or color gradient information around a certain pixel satisfies a certain condition. The base time feature point detection unit 123 detects at least one or more feature points.

[0021] 2B, the elapsed time feature point detection unit 124 detects feature points in the elapsed time frames stored by the elapsed time frame storage unit 122. When multiple elapsed time frames are stored, the elapsed time feature point detection unit 124 detects feature points in each of the multiple elapsed time frames.

[0022] The overlapping unit 13 overlaps the base time frame and the elapsed time frame so that the distance between corresponding feature points in the base time frame and the elapsed time frame is minimized. The overlapping unit 13 includes a minimum distance search unit 131. The minimum distance search unit 131 includes a movement unit 1311.

[0023] The minimum distance search unit 131 receives the base time frame and the feature amount and pixel information of the feature point in the base time frame from the base time feature point detection unit 123. When there are multiple feature points, information on each of the multiple feature points is input to the minimum distance search unit 131.

[0024] The moving unit 1311 receives the elapsed time frames and the feature amounts and pixel information of the feature points in the elapsed time frames from the elapsed time feature point detection unit 124. When there are multiple feature points, information on each of the multiple feature points is input to the moving unit 1311. When feature points are detected for multiple elapsed time frames, the moving unit 1311 receives the multiple elapsed time frames, the elapsed times corresponding to each of the multiple elapsed time frames, and the information on the feature points in each of the multiple elapsed time frames.

[0025] The minimum distance search unit 131 searches for corresponding feature points between the base time frame and the elapsed time frame. Specifically, the minimum distance search unit 131 searches for feature points whose feature amounts are similar between the elapsed time frame and the base time frame.

[0026] The search for feature points by minimum distance search section 131 will be described with reference to FIGS. 2A and 2B.

[0027] In Fig. 2A, it is assumed that two feature points (feature point A and feature point B) are detected in the base time frame. Also, in Fig. 2B, it is assumed that two feature points (feature point A' and feature point B') are detected in the elapsed time frame. For example, minimum distance search unit 131 detects feature point A' as the feature point having the most similar feature amount to feature point A, and detects feature point B' as the feature point having the most similar feature amount to feature point B.

[0028] When the number of feature points in the elapsed time frame is smaller than the number of feature points in the base time frame, the minimum distance search unit 131 searches for corresponding feature points in the base time frame and the elapsed time frame in accordance with the number of feature points in the elapsed time frame. Also, when the number of feature points in the elapsed time frame is greater than the number of feature points in the base time frame, the minimum distance search unit 131 searches for corresponding feature points in the base time frame and the elapsed time frame in accordance with the number of feature points in the base time frame. In other words, the minimum distance search unit 131 searches for corresponding feature points in the base time frame and the elapsed time frame in accordance with the smaller of the number of feature points in the base time frame and the number of feature points in the elapsed time frame.

[0029] Next, as shown in Fig. 3, minimum distance search unit 131 sets a coordinate system (xy coordinate system) based on the base time frame, and causes movement unit 1311 to move the elapsed time frame in the x-axis direction and y-axis direction so that the distance (Euclidean distance) between corresponding feature points in the base time frame and the elapsed time frame is minimized. Minimum distance search unit 131 fixes the elapsed time frame at the point where the Euclidean distance is minimized. Note that when there are multiple elapsed time frames, minimum distance search unit 131 performs the above-mentioned process for each elapsed time frame.

[0030] Referring again to Figure 1, when the overlay unit 13 overlays the base time frame and the elapsed time frame so that the distance between corresponding feature points is minimized, it outputs information regarding the positions of the base time frame and the elapsed time frame fixed on the coordinate axis to the overlap rate determination unit 14.

[0031] Based on the information output from the overlapping unit 13, the overlapping rate determination unit 14 calculates the area (pixel area) of the overlapping region where the base time frame and the elapsed time frame overlap, as shown in Fig. 4. Then, the overlapping rate determination unit 14 determines whether the ratio of the area of ​​the overlapping region to the area of ​​the base time frame (overlapping rate) is equal to or less than a predetermined threshold. If the overlapping rate determination unit 14 determines that the overlapping rate is equal to or less than the predetermined threshold, it outputs the elapsed time frame (the frame at time t1+Δt) as an extracted frame to the storage unit 15 to store it. On the other hand, if the overlapping rate determination unit 14 determines that the overlapping rate is not equal to or less than the predetermined threshold, it does not output the elapsed time frame to the storage unit 15 (does not store it).

[0032] When the ratio of the area of ​​the overlapping region to the area of ​​the base time frame (overlapping rate) is equal to or less than a predetermined threshold, it is considered that a predetermined part of the object in the angle of view has moved by a certain amount between the base time (time t1) and the elapsed time (time t1+Δt). Therefore, when the ratio of the area of ​​the overlapping region to the area of ​​the base time frame is equal to or less than a predetermined threshold, the image processing device 10 according to this embodiment outputs the elapsed time frame to the storage unit 15 and stores it. In this way, in a moving image in which the object is photographed while the viewpoint is moving, a frame at a timing when a predetermined part of the object has been displaced by a predetermined amount can be output.

[0033] When the overlap rate determination unit 14 causes the storage unit 15 to store the elapsed time frame (the frame at time t1+Δt) as an extracted frame, it notifies the base point time frame storage unit 121 that the elapsed time frame has been stored. Upon receiving the notification from the overlap rate determination unit 14, the base point time frame storage unit 121 repeats the above-mentioned process, setting time t2 after time t1+Δt as a new base point time. That is, when the elapsed time frame is output (stored) as an extracted frame, the base point time frame storage unit 121 (feature point detection unit 12) determines a time after the time corresponding to the elapsed time frame as a new base point time, and extracts a frame at that time as a new base point time frame from the input video. This makes it possible to prevent the process from going back to a time before the base point time and repeating overlapping processes.

[0034] If the elapsed time frame at time t1+Δt is not stored in the storage unit 15 as the extracted frame, the frame at time t1+Δt1, which is later than time t1+Δt, is used as the elapsed time frame and the above-mentioned process is repeated.

[0035] Next, the operation of the image processing device 10 according to this embodiment will be described.

[0036] FIG. 5 is a flowchart showing an example of the operation of the image processing device 10 according to the present embodiment, and is a diagram for explaining an image processing method by the image processing device 10.

[0037] The video input unit 11 receives input of a video image of an object captured while moving the viewpoint (step S11).

[0038] The feature point detection unit 12 detects feature points in a base time frame (first frame) at the base time (first time) and an elapsed time frame (second frame) at an elapsed time (second time) a predetermined time has elapsed from the base time (step S12). Specifically, the base time frame storage unit 121 extracts and stores a base time frame from the input video. Furthermore, the elapsed time frame storage unit 122 extracts and stores an elapsed time frame from the input video. Then, the base time feature point detection unit 123 detects feature points in the stored base time frame. Furthermore, the elapsed time feature point detection unit 124 detects feature points in the stored elapsed time frame.

[0039] The superimposing unit 13 superimposes the base time frame and the elapsed time frame so that the distance between corresponding feature points in the base time frame and the elapsed time frame is minimized (step S13). Specifically, the minimum distance search unit 131 searches for feature points in the base time frame and feature points in the elapsed time frame that correspond to the feature points in the base time frame. Then, the minimum distance search unit 131 causes the movement unit 1311 to move the elapsed time frame on the coordinate system based on the base time frame so that the Euclidean distance between the corresponding feature points is minimized, and the base time frame and the elapsed time frame are superimposed.

[0040] The overlap rate determination unit 14 calculates the area of ​​the overlap region where the base time frame and the elapsed time frame overlap, and then determines whether the ratio of the area of ​​the overlap region to the area of ​​the base time frame (overlap rate) is equal to or smaller than a predetermined threshold (step S14).

[0041] When it is determined that the overlap rate is equal to or less than a predetermined threshold (step S14: Yes), the overlap rate determination unit 14 outputs the elapsed time frame as an extracted frame to the storage unit 15 to store it (step S15). After the elapsed time frame is stored in the storage unit 15 as an extracted frame, the base time is updated, and the process is repeated from step S12. Specifically, the base time is updated, and the process from step S12 onwards is performed on the new base time frame and the elapsed time frame.

[0042] If it is determined that the overlap rate is not equal to or less than the threshold value (step S14: No), the overlap rate determination unit 14 determines whether or not a predetermined threshold value T or more has elapsed since the base time, i.e., whether or not the elapsed time from the base time to the elapsed time is equal to or more than T (step S16).

[0043] When it is determined that a predetermined threshold value T or more has elapsed since the base time (step S16: Yes), the overlap rate determination unit 14 outputs the elapsed time frame as an extracted frame to the storage unit 15 to store it (step S15). This makes it possible to prevent the process from terminating without storing any extracted frame even if an error occurs in image recognition.

[0044] If it is determined that a predetermined threshold value T or more has not elapsed since the base time (step S16: No), the overlap rate determination unit 14 notifies the base time frame storage unit 121 that the elapsed time frame will not be stored as an extracted frame. This notification causes the processing to be repeated from step S12. Specifically, the elapsed time is updated, and the processing from step S12 onwards is performed on the base time frame and the new elapsed time frame.

[0045] Next, the hardware configuration of the image processing device 10 according to this embodiment will be described.

[0046] Fig. 6 is a diagram showing an example of a hardware configuration of an image processing device 10 according to this embodiment. Fig. 6 shows an example of a hardware configuration of the image processing device 10 when the image processing device 10 is configured by a computer capable of executing program instructions. Here, the computer may be a general-purpose computer, a dedicated computer, a workstation, a PC (Personal computer), an electronic notepad, etc. The program instructions may be program code, code segments, etc. for performing necessary tasks.

[0047] 6, the image processing device 10 has a processor 21, a ROM (Read Only Memory) 22, a RAM (Random Access Memory) 23, a storage 24, an input unit 25, a display unit 26, and a communication interface (I / F) 27. Each component is connected to be able to communicate with each other via a bus 29. The processor 21 is specifically a CPU (Central Processing Unit), an MPU (Micro Processing Unit), a GPU (Graphics Processing Unit), a DSP (Digital Signal Processor), a SoC (System on a Chip), or the like, and may be composed of multiple processors of the same type or different types.

[0048] The processor 21 is a control unit that controls each component and executes various arithmetic processing. That is, the processor 21 reads a program from the ROM 22 or the storage 24, and executes the program using the RAM 23 as a working area. The processor 21 controls each component and executes various arithmetic processing according to the program stored in the ROM 22 or the storage 24. In this embodiment, the ROM 22 or the storage 24 stores a program for operating a computer as the image processing device 10 according to the present disclosure. The program is read and executed by the processor 21, thereby realizing each component of the image processing device 10, that is, the feature point detection unit 12, the overlapping unit 13, and the overlap rate determination unit 14.

[0049] The program may be provided in a form stored in a non-transitory storage medium such as a CD-ROM (Compact Disk Read Only Memory), a DVD-ROM (Digital Versatile Disk Read Only Memory), a USB (Universal Serial Bus) memory, etc. The program may also be provided in a form downloaded from an external device via a network.

[0050] The ROM 22 stores various programs and various data. The RAM 23 temporarily stores programs or data as a working area. The storage 24 is configured with an HDD (Hard Disk Drive) or an SSD (Solid State Drive) and stores various programs including an operating system and various data. The ROM 22 or the storage 24 stores, for example, elapsed time frames output as cut-out frames.

[0051] The input unit 25 includes a pointing device such as a mouse, and a keyboard, and is used to perform various input operations.

[0052] The display unit 26 is, for example, a liquid crystal display, and displays various information. The display unit 26 may function as the input unit 25 by adopting a touch panel system.

[0053] The communication interface 27 is an interface for communicating with other devices (for example, a photographing device that photographs a moving image), and is, for example, an interface for a LAN.

[0054] A computer can be suitably used to function as each part of the image processing device 10 described above. Such a computer can be realized by storing a program describing the processing contents for realizing the functions of each part of the image processing device 10 in a storage unit of the computer, and reading and executing the program by a processor of the computer. In other words, the program can cause the computer to function as the image processing device 10 described above. The program can also be recorded on a non-transitory recording medium. The program can also be provided via a network.

[0055] Thus, the image processing device 10 according to this embodiment includes a feature point detection unit 12, a superimposition unit 13, and an overlap rate determination unit 14. The feature point detection unit 12 detects feature points in a base time frame at a base time and in an elapsed time frame at a predetermined time after the base time. The superimposition unit 13 superimposes the base time frame and the elapsed time frame so that the distance between corresponding feature points in the base time frame and the elapsed time frame is minimized. The overlap rate determination unit 14 determines whether the ratio (overlap rate) of the area of ​​the overlapping region where the base time frame and the elapsed time frame overlap with respect to the area of ​​the base time frame is equal to or less than a predetermined threshold, and outputs the elapsed time frame as a cut-out frame when it is determined that the overlap rate is equal to or less than the predetermined threshold.

[0056] When the area of ​​the overlapping region relative to the area of ​​the base time frame is equal to or less than a predetermined threshold, it is considered that a predetermined part of the object in the angle of view has moved a certain amount between the base time and the elapsed time. Therefore, the image processing device 10 according to the present embodiment outputs the elapsed time frame as an extracted frame when the ratio of the area of ​​the overlapping region relative to the area of ​​the base time frame is equal to or less than a predetermined threshold. In this way, it is possible to output a frame at a timing when a predetermined part of the object has been displaced a predetermined amount in a moving image in which the object is photographed while the viewpoint is moving.

[0057] Second embodiment FIG. 7 is a diagram showing an example of the configuration of an image processing device 10A according to the second embodiment of the present disclosure.

[0058] An image processing device 10A shown in Fig. 7 differs from the image processing device 10 shown in Fig. 1 in that the feature point detection unit 12 has been changed to a feature point detection unit 12A. The feature point detection unit 12A differs from the feature point detection unit 12 in that the base time feature point detection unit 123 has been changed to a base time feature point detection unit 123A, and that the elapsed time feature point detection unit 124 has been changed to an elapsed time feature point detection unit 124A. The base time feature point detection unit 123A includes a feature point detection area setting unit 1231. The elapsed time feature point detection unit 124A includes a feature point detection area setting unit 1241.

[0059] 8A, the feature point detection area setting unit 1231 sets a feature point detection area in which feature points are detected in the base time frame. The feature point detection area setting unit 1231 sets the feature point detection area in response to, for example, an input from a user via the input unit 25. The base time feature point detection unit 123A detects feature points in the feature point detection area set by the feature point detection area setting unit 1231 in the base time frame.

[0060] 8B, the feature point detection area setting unit 1241 sets a feature point detection area in which feature points are detected in the elapsed time frame. The feature point detection area setting unit 1241 sets the feature point detection area in response to, for example, an input from a user via the input unit 25. The elapsed time feature point detection unit 124A detects feature points in the feature point detection area set by the feature point detection area setting unit 1241 in the elapsed time frame.

[0061] 8A and 8B show an example in which the feature point detection region is rectangular, but the feature point detection region is not limited to this and may be set in any shape.

[0062] Thus, in this embodiment, the feature point detection unit 12A sets feature point detection areas, which are areas in which feature points are detected in frames (base time frames and elapsed time frames), and detects feature points in the feature point detection areas set for each of the base time frames and elapsed time frames.

[0063] Setting a feature point detection area can prevent a feature point detected in a base time frame from not being captured in an elapsed time frame due to movement of the viewpoint of the imaging device. For example, if a feature point is detected in a pixel near the edge of the frame in the base time frame, there is a high possibility that the feature point will not be captured in an elapsed time frame due to movement of the viewpoint of the imaging device. For this reason, it is effective to set the center of the frame, or an area where an object exists in the next frame when imaging is performed while the imaging device is moving in a fixed direction, as the feature point detection area.

[0064] (Third embodiment) FIG. 9 is a diagram showing an example of the configuration of an image processing device 10B according to the third embodiment of the present disclosure.

[0065] Image processing device 10B shown in Fig. 9 differs from image processing device 10 shown in Fig. 1 in that superimposition unit 13 has been changed to superimposition unit 13B. Superimposition unit 13B differs from superimposition unit 13 in that minimum distance search unit 131 has been changed to minimum distance search unit 131B. Minimum distance search unit 131B differs from minimum distance search unit 131 in that movement unit 1311 has been changed to movement / rotation / scale change unit 1312.

[0066] When moving the elapsed time frame on the coordinate system based on the base time frame so that the distance between corresponding feature points in the base time frame and the elapsed time frame is minimized, the movement / rotation / scale change unit 1312 rotates the elapsed time frame and / or changes the scale of the elapsed time frame as shown in Fig. 10. In other words, the superimposition unit 13B rotates the elapsed time frame and / or changes the scale of the elapsed time frame to superimpose the base time frame and the elapsed time frame so that the distance between corresponding feature points in the base time frame and the elapsed time frame is minimized.

[0067] By rotating the elapsed time frame, the positions of the feature points in the elapsed time frame also change in accordance with the rotation of the elapsed time frame. In addition, by changing the scale of the elapsed time frame (enlarging or reducing), the positions of the feature points in the elapsed time frame also change in accordance with the enlargement or reduction of the elapsed time frame.

[0068] Therefore, even when the viewpoint of the imaging device is rotated while imaging, or when the object is imaged obliquely at a constant elevation angle, it is possible to reduce errors in overlapping of corresponding feature points between the base time frame and the elapsed time frame. As a result, the accuracy of the overlap rate determination is improved, and it is possible to suppress overlaps and omissions of frames to be saved as extracted frames.

[0069] In the present embodiment, the superimposing unit 13 of the image processing device 10 according to the first embodiment is replaced with the superimposing unit 13B, but the present invention is not limited to this. The superimposing unit 13 of the image processing device 10A according to the second embodiment may be replaced with the superimposing unit 13B.

[0070] (Fourth embodiment) FIG. 11 is a diagram illustrating an example of the configuration of an image processing device 10C according to the fourth embodiment of the present disclosure.

[0071] An image processing device 10C shown in Fig. 11 differs from the image processing device 10 shown in Fig. 1 in that the overlap rate determination unit 14 is changed to an overlap rate determination unit 14C. The overlap rate determination unit 14C includes an averaging unit 141.

[0072] The averaging unit 141 calculates the average value of the area of ​​the overlapping area between the base time frame and each of the elapsed time frames at a plurality of elapsed times having different elapsed times from the base time. For example, assuming that the base time is time t1, the averaging unit 141 calculates the area of ​​the overlapping area with the base time frame not only for the elapsed time frame at time t1+Δ but also for the elapsed time frames at times t1+Δt1, t1+Δt2, and t1+Δt3. Then, the averaging unit 141 calculates the average value of the area of ​​the overlapping area between the base time frame and each of the elapsed time frames at times t1+Δ, t1+Δt1, t1+Δt2, and t1+Δt3. Note that the number of elapsed time frames for which the average value of the area of ​​the overlapping area with the base time frame is calculated may be any number equal to or greater than 2.

[0073] The overlap rate determination unit 14C determines whether the ratio of the average value of the area of ​​the overlapping area between the base time frame and each of the multiple elapsed time frames calculated by the averaging unit 141 to the area of ​​the base time frame is equal to or less than a predetermined threshold. If the overlap rate determination unit 14C determines that the ratio of the average value of the area of ​​the overlapping area between the base time frame and each of the multiple elapsed time frames to the area of ​​the base time frame is equal to or less than a predetermined threshold, it outputs at least one of the multiple elapsed time frames to the storage unit 15 as an extracted frame and stores it.

[0074] By using the average value of the overlapping area between the base time frame and each of multiple elapsed time frames, it is possible to reduce the occurrence of errors due to incorrect image recognition of a single frame.

[0075] In the present embodiment, the overlap rate determination unit 14 of the image processing device 10 according to the first embodiment is replaced by the overlap rate determination unit 14C, but the present invention is not limited to this. The overlap rate determination unit 14 of the image processing device 10A according to the second embodiment or the image processing device 10B according to the third embodiment may be replaced by the overlap rate determination unit 14C.

[0076] In addition, the image processing device 10A according to the second embodiment, the image processing device 10B according to the third embodiment, and the image processing device 10C according to the fourth embodiment can also be configured by a computer having the hardware configuration described with reference to FIG. 6.

[0077] The following supplementary notes are further disclosed regarding the above embodiment.

[0078] [Additional note 1] An image processing device that cuts out a frame at a timing when a predetermined part of an object is displaced by a predetermined amount from a moving image composed of a plurality of frames, the moving image being obtained by photographing an object while moving a viewpoint, and outputs the cut-out frame, Memory, A control unit connected to the memory; Equipped with The control unit is Detecting feature points in a first frame, which is a frame at a first time point serving as a base point, and a second frame, which is a frame at a second time point a predetermined time after the first time point; superimposing the first frame and the second frame so that a distance between corresponding feature points in the first frame and the second frame is minimized; An image processing device that determines whether a ratio of an area of ​​an overlapping region where the first frame and the second frame overlap to an area of ​​the first frame is equal to or less than a predetermined threshold, and outputs the second frame as the cut-out frame if it is determined that the ratio is equal to or less than the predetermined threshold.

[0079] [Additional note 2] In the image processing device according to claim 1, The control unit sets a feature point detection area in the frame in which the feature points are to be detected, and detects the feature points in the set feature point detection area.

[0080] [Additional note 3] In the image processing device according to claim 1, The control unit rotates the second frame and / or changes the scale of the second frame to superimpose the first frame and the second frame so that the distance is minimized.

[0081] [Additional note 4] In the image processing device according to claim 1, the control unit calculates an average value of the area of ​​the overlapping region between the first frame and the second frame at each of a plurality of times having different elapsed times from the base time, determines whether a ratio of the average value to the area of ​​the first frame is equal to or less than the predetermined threshold, and if it determines that the ratio is equal to or less than the predetermined threshold, outputs at least one elapsed time frame among the plurality of second frames as the cut-out frame.

[0082] [Additional note 5] In the image processing device according to claim 1, When the second frame is output as the cut-out frame, the control unit determines a time after the time corresponding to the second frame to be the base time, and extracts the frame at that time from the moving image as the first frame.

[0083] [Additional note 6] 1. An image processing method for cutting out and outputting a frame at a timing when a predetermined part of an object is displaced by a predetermined amount from a moving image composed of a plurality of frames, the moving image being obtained by photographing an object while moving a viewpoint, the method comprising: Detecting feature points in a first frame, which is a frame at a first time, and a second frame, which is a frame at a second time when a predetermined time has elapsed since the first time; superimposing the first frame and the second frame so that a distance between corresponding feature points in the first frame and the second frame is minimized; An image processing method comprising: determining whether a ratio of an area of ​​an overlapping region where the first frame and the second frame overlap to an area of ​​the first frame is equal to or less than a predetermined threshold; and outputting the second frame as the cut-out frame if it is determined that the ratio is equal to or less than the predetermined threshold.

[0084] [Additional note 7] A non-transitory storage medium storing a program executable by a computer, the non-transitory storage medium storing the program causing the computer to operate as the image processing device described in appended claim 1.

[0085] Although the above-mentioned embodiment has been described as a representative example, it is obvious to those skilled in the art that many modifications and substitutions can be made within the spirit and scope of the present disclosure. Therefore, the present invention should not be interpreted as being limited by the above-mentioned embodiment, and various modifications or changes can be made without departing from the scope of the claims. For example, it is possible to combine multiple building blocks shown in the configuration diagram of the embodiment into one, or to divide one building block. [Explanation of symbols]

[0086] 10, 10A, 10B, 10C Image processing device 11 Video input section 12,12A Feature point detection section 13,13B Overlapped part 14,14C Overlap rate judgment section 15 Preservation Department 121 Base time frame storage section 122 Elapsed time frame storage unit 123,123A Base point time feature point detection unit 124, 124A Elapsed time feature point detection unit 131,131B Minimum distance search unit 141 Average part 1231, 1241 Feature point detection area setting section 1311 Mobile Department 1312 Move / Rotate / Scale Change Part 21 Processors 22 ROM 23 RAM 24 Storage 25 Input section 26 Display section 27 Communication I / F 29 Bus

Claims

1. An image processing device that cuts out a frame at a timing when a predetermined part of an object is displaced by a predetermined amount from a moving image composed of a plurality of frames, the moving image being obtained by photographing an object while moving a viewpoint, and outputs the cut-out frame, a feature point detection unit that detects feature points in a first frame that is a frame at a first time point that is a base point, and a second frame that is a frame at a second time point that is a predetermined time after the first time point; a superimposing unit that superimposes the first frame and the second frame so as to minimize a distance between corresponding feature points in the first frame and the second frame; an overlap rate determination unit that determines whether a ratio of an area of ​​an overlap region where the first frame and the second frame overlap to an area of ​​the first frame is equal to or less than a predetermined threshold, and outputs the second frame as the cut-out frame if it is determined that the ratio is equal to or less than the predetermined threshold.

2. 2. The image processing device according to claim 1, The image processing device, wherein the feature point detection unit sets a feature point detection area in the frame in which the feature points are to be detected, and detects the feature points in the set feature point detection area.

3. 3. The image processing device according to claim 1, An image processing device, wherein the overlay unit rotates the second frame and / or changes the scale of the second frame to overlay the first frame and the second frame so that the distance is minimized.

4. 4. The image processing device according to claim 1, the overlap rate determination unit calculates an average value of the area of ​​the overlap region between the first frame and the second frame at each of a plurality of times that have elapsed time from the base time, determines whether a ratio of the average value to the area of ​​the first frame is equal to or less than the predetermined threshold, and if it determines that the ratio is equal to or less than the predetermined threshold, outputs at least one elapsed time frame among the plurality of second frames as the cut-out frame.

5. 5. The image processing device according to claim 1, When the second frame is output as the cut-out frame, the feature point detection unit determines a time after the time corresponding to the second frame to be the base time, and extracts the frame at that time from the moving image as the first frame.

6. 1. An image processing method for cutting out a frame at a timing when a predetermined part of an object is displaced by a predetermined amount from a moving image composed of a plurality of frames, the moving image being obtained by photographing an object while moving a viewpoint, and outputting the cutout frame, the method comprising: detecting feature points in a first frame, which is a frame at a first time, and a second frame, which is a frame at a second time when a predetermined time has elapsed since the first time; overlapping the first frame with the second frame such that a distance between corresponding feature points in the first frame and the second frame is minimized; determining whether a ratio of an area of ​​an overlapping region where the first frame and the second frame overlap to an area of ​​the first frame is equal to or less than a predetermined threshold, and outputting the second frame as the cut-out frame if it is determined that the ratio is equal to or less than the predetermined threshold.

7. A program for causing a computer to operate as the image processing device according to any one of claims 1 to 5.

Images