Still-picture generation device and program for generating still picture

Abstract

To generate a still picture of a material of a subject that is relatively moved against a camera from photographing image information from the camera that performs a photograph by using a fisheye lens.SOLUTION: A material of a subject that is relatively moved to a photographing position is photographed with a camera for performing a photographing with a fisheye lens. An image processing device comprises: a correction part that performs dewarping of photographing image information from a camera in a frame unit; a feature point detection part that detects a feature point from an image in the frame unit; a matching part that performs a matching between post- and pre-frames in regard to the feature point detected by the feature point detection part; a movement amount calculation part that calculates a movement amount of the feature point detection part of the post- and pre-frames on the basis of a matching result in the matching part; a coupling image generation part that generates a coupling image about an image in the frame unit on the basis of the movement amount calculated by the movement amount calculation part; and a coupling part that couples the coupling image generated by the coupling image generation part to the relative movement direction of the material of the subject.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a still image generating device and a still image generating program for generating a still image from captured image information from a camera that captures an image of a moving object. [Background technology]

[0002] Patent Document 1 (JP Patent Publication No. 4871678) is known as a technology for generating a still image from captured image information of a moving body. In Patent Document 1, the moving body is a vehicle, and the bottom surface of the vehicle is captured by a camera disposed near the road surface, and a still image of the entire bottom surface of the vehicle is generated from the captured image.

[0003] In Patent Document 1, since the distance between the bottom of the vehicle and the camera is short, it is difficult to view the entire bottom of the vehicle even if a wide-angle lens is used, so a fisheye lens is used to view the entire bottom of the vehicle. In Patent Document 1, a vehicle speed sensor is provided to detect the traveling speed of the vehicle, and the time required to travel a predetermined distance is calculated based on the traveling speed detected by the vehicle speed sensor, and a shooting command is given to the camera for each calculated time. Furthermore, in Patent Document 1, a still image of the entire bottom of the vehicle is generated by combining each of the multiple images taken by the camera after correcting the spherical aberration caused by the fisheye lens. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Patent No. 4871678 Summary of the Invention [Problem to be solved by the invention]

[0005] However, the still image generating device of Patent Document 1 requires a vehicle speed sensor in addition to the fisheye lens camera, which increases costs and requires installation space. In addition, when the moving object to be photographed is something other than a vehicle, it may be necessary to prepare a different sensor for each moving object.

[0006] In addition, in Patent Document 1, the time required to travel a predetermined distance is calculated based on the traveling speed detected by the vehicle speed sensor, and a shooting command is given to the camera for each calculated time. Therefore, it is assumed that the vehicle travels at a constant speed when traveling the predetermined distance, and there is a problem in that it is difficult to respond to changes in speed.

[0007] An object of the present invention is to provide a still image generating device that can solve the above problems. [Means for solving the problem]

[0008] In order to solve the above problem, the invention of claim 1 is as follows: a camera that uses a fisheye lens to capture an image of a subject object that moves relative to a capture position; and an image processing device that generates a still image of the entire image of the subject object from captured image information from the camera; The image processing device includes: a correction unit that dewarps the captured image information from the camera on a frame-by-frame basis; a feature point detection unit that detects feature points from the frame-by-frame images corrected by the correction unit; a matching unit that performs matching between previous and next frames regarding the feature points detected by the feature point detection unit; a movement amount calculation unit that calculates a movement amount of the feature point between the previous and next frames based on a matching result by the matching unit; a combination image generating unit that generates a combination image for the image in frame unit based on the movement amount calculated by the movement amount calculating unit; a combining unit that combines the images to be combined generated by the image to be combined generating unit in the relative movement direction of the subject; The present invention provides a still image generating device comprising:

[0009] In the still image generating device according to the invention of claim 1 having the above-mentioned configuration, feature points are extracted from frame-by-frame images obtained by dewarping and correcting image information captured using a fisheye lens, and a matching unit performs matching processing between the previous and next frames for the feature points. Then, based on the matching results, the amount of movement of the feature points between the previous and next frames is calculated. Based on the calculated amount of movement, a combination image is generated for the frame-by-frame images, and the generated combination image is combined in the relative movement direction of the subject object to generate a still image.

[0010] Therefore, according to the still image generating device of the invention of claim 1 having the above-mentioned configuration, it is not necessary to use a sensor such as a vehicle speed sensor, which is advantageous in terms of cost and does not require installation space. Also, since an image for combination is generated for an image on a frame-by-frame basis based on the amount of movement of a feature point between previous and next frames, and the generated image for combination is combined in the relative movement direction of the subject object, even if the speed of the relative movement of the subject object changes, an image for combination corresponding to the change can be obtained, and a good still image can be generated. Effect of the Invention

[0011] According to the present invention, since it is not necessary to use a sensor such as a vehicle speed sensor, it is advantageous in terms of cost and does not require installation space. Furthermore, even if the relative movement speed of the subject changes, a corresponding image to be combined can be obtained, and a good still image can be generated. [Brief description of the drawings]

[0012] [Figure 1] 1 is a diagram illustrating an example of the configuration of an embodiment of a still image generating device according to the present invention. [Diagram 2]3 is a diagram for explaining a subject object appearance / disappearance detection unit in the embodiment of the still image generating device according to the present invention. FIG. [Diagram 3] 1 is a diagram used to explain a dewarping correction unit in an embodiment of a still image generating device according to the present invention. FIG. [Figure 4] 11 is a diagram used to explain a center portion cutout unit in the embodiment of the still image generating device according to the present invention. FIG. [Diagram 5] 3 is a diagram used to explain a feature point detection unit and a matching unit in the embodiment of the still image generating device according to the present invention. FIG. [Figure 6] 1 is a diagram used to explain a movement amount calculation unit in the embodiment of the still image generating device according to the present invention. FIG. [Figure 7] 1 is a diagram used to explain a combination image generating section in an embodiment of a still image generating device according to the present invention. FIG. [Figure 8] 1 is a diagram used to explain a combination image generating section in an embodiment of a still image generating device according to the present invention. FIG. [Figure 9] 1 is a diagram used to explain a coupling unit in an embodiment of a still image generating device according to the present invention. FIG. [Figure 10] FIG. 11 is a diagram showing a part of a flowchart illustrating an example of the flow of operations of an image processing device in an embodiment of a still image generating device according to the present invention. [Figure 11] FIG. 11 is a diagram showing a part of a flowchart illustrating an example of the flow of operations of an image processing device in an embodiment of a still image generating device according to the present invention. [Figure 12] FIG. 11 is a diagram showing a part of a flowchart illustrating an example of the flow of operations of an image processing device in an embodiment of a still image generating device according to the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0013] Hereinafter, an embodiment of a still image generating device according to the present invention will be described with reference to the drawings.

[0014] Fig. 1 is a diagram showing an example of the configuration of an embodiment of a still image generating device according to the present invention. The still image generating device of the embodiment shown in Fig. 1 is configured with a camera 2 that captures an image of the bottom surface of the body of a moving automobile MV as a subject, and an image processing device 1 that generates a still image of the bottom surface of the body of the automobile MV from image information captured by the camera 2.

[0015] Camera 2 is installed below the road surface on which automobile MV travels, with the distance between the bottom of the body of automobile MV and camera 2 being short. Camera 2 uses a fisheye lens 2L as a shooting lens so that the entire bottom of the body of automobile MV can be included in the field of view even when the distance between the bottom of the body of automobile MV and camera 2 is short. In this example, as shown in Fig. 1, the shooting position of camera 2 is illuminated by lighting device 3 to provide a brightness suitable for shooting.

[0016] In this embodiment, the camera 2 captures a moving image of the underside of the body of the traveling automobile MV, and supplies the captured image information to the image processing device 1. The captured image information may be supplied from the camera 2 to the image processing device via a cable or wireless communication. If the process of generating a still image does not need to be performed in real time, a storage medium such as a card-type memory may be loaded into the camera 2, the captured image information may be stored in the storage medium, and the image processing device 1 may read out the captured image information from the storage medium and perform the process of generating a still image.

[0017] In the image processing device 1, the distortion inherent to the captured image caused by using the fisheye lens 2L is reduced by dewarping, and then an image of only the central part of the subject object, the automobile MV, in the direction of travel is cut out, and these cut-out images are combined in the direction of travel to generate a still image of the entire bottom surface of the body of the automobile MV as it travels.

[0018] In this example, when the traveling direction of the automobile MV, which is the subject object, is a straight traveling direction, the direction parallel to the vertical direction of the frame image acquired from the captured image information from the camera 2 is made to match the traveling direction of the automobile MV. In this way, if the traveling direction of the automobile MV, which is the subject object, is straight and parallel to the vertical direction of the frame image, the image to be cut out for joining may be the vertical center of the frame image.

[0019] However, the traveling direction of the automobile MV is not always a straight direction, and may be oblique to the straight direction. In this way, when the traveling direction of the automobile MV is oblique, as described below, the frame images are rotated so that the traveling direction and the combining direction coincide with each other, and then the combining image is cut out and combined.

[0020] The image processing device 1 is configured by connecting a control unit 10 with a frame-unit image acquisition unit 11, a subject object appearance / disappearance detection unit 12, a dewarping correction unit 13, a central portion cut-out unit 14, a feature point detection unit 15, a matching unit 16, a movement amount calculation unit 17, a combination image generation unit 18, and a combination unit 19.

[0021] In this example, the control unit 10 is configured as a computer, and controls the processing of each unit connected thereto. Each of the frame unit image acquisition unit 11, subject object appearance / disappearance detection unit 12, dewarping correction unit 13, central portion clipping unit 14, feature point detection unit 15, matching unit 16, movement amount calculation unit 17, image to be combined generation unit 18, and combining unit 19 can be configured as hardware or as a functional block executed by a software program.

[0022] The frame-unit image acquisition unit 11 acquires frame-unit image information of a moving image from the camera 2 in this example. In this case, the frames to be acquired may be all frames of the captured image information of the moving image, or may be every other frame or every several frames. In short, the frame-unit image acquisition unit 11 only needs to acquire frames necessary to generate a still image.

[0023] The subject object appearance / disappearance detection unit 12 detects whether or not an image of the bottom of the body of the automobile MV, which is the subject object in this example, has appeared in the image of a frame acquired by the frame unit image acquisition unit 11, and whether or not the image of the bottom of the body of the automobile MV, which had appeared in the image of the frame, has disappeared from the image of the frame. In this example, the detection in the subject object appearance / disappearance detection unit 12 is processed by AI (Artificial Intelligence) using images of the bottom of the body of the automobile MV that have been learned in the past.

[0024] In this embodiment, as described later, in order to perform still image generation processing with higher accuracy, processing is performed using only the image of the vertical center portion of the frame image corresponding to the straight running direction of the automobile, which is the subject object. Therefore, the subject object appearance / disappearance detection unit 12 does not detect the appearance of the subject object when the subject object begins to appear in the image frame of the fisheye lens 2L as shown in FIG. 2(A), but detects that the subject object has appeared in the image frame of the fisheye lens 2L when the subject object has appeared near the center of the image frame of the fisheye lens 2L as shown in FIG. 2(B). In addition, the subject object appearance / disappearance detection unit 12 detects that the subject object has disappeared from the image frame of the fisheye lens 2L when the subject object disappears from near the center of the image frame of the fisheye lens 2L.

[0025] The control unit 10 receives the detection result from the subject object appearance / disappearance detection unit 12, and when it detects that a subject object has appeared, starts a still image generation process by image combination as described below, and when it detects that the subject object has disappeared, ends the still image generation process. In addition, in this embodiment, when the control unit 10 starts the still image generation process, it resizes the image information on a frame-by-frame basis, for example by lowering the resolution, taking into consideration the performance of the information processing device 1 of this embodiment for the subsequent image processing.

[0026] The dewarp correction unit 13 dewarps the frame image (fisheye image) from a fisheye lens, which appears distorted and rounded as shown in Figure 3(A), and corrects it to a less distorted planar image (an image as if photographed from the front) as shown in Figure 3(B).

[0027] The central portion cropping unit 14 crops out only the central portion in the vertical direction of the frame image in this example, in the moving direction of the automobile MV, which is the subject object, from the frame image 201 (see FIG. 4(A)) dewarped and corrected by the dewarp correction unit 13, to obtain a central portion image 202 as shown in FIG. 4(B). Here, the vertical central portion image of the frame image means an image within a range of a predetermined vertical width W centered on the vertical center position of the frame, as shown in FIG. 4(A).

[0028] The reason for cutting out the central partial image 202 in this manner is to make it easier to detect feature points in the feature point detection unit 15 described below, and also to make it possible to eliminate image portions in the image of the underside of the vehicle body, which is the subject object, that have significant image distortion due to the perspective difference from the fisheye lens 2L, which is the shooting lens, when combining images to be combined in the direction of movement of the subject object, as described below.

[0029] In this embodiment, the feature point detection unit 15, the matching unit 16, the movement amount calculation unit 17, and the combination image generation unit 18 use the central portion image 202 cut out by the central portion cutout unit 14 as the processing target.

[0030] The feature point detection unit 15 detects feature points in the central portion image 202 cut out by the central portion cutout unit 14. The feature point detection process in the feature point detection unit 15 uses a known feature point detection process algorithm that is common in image processing, and the details are omitted here.

[0031] The matching unit 16 matches the same feature points in two frames before and after the feature points detected by the feature point detection unit 15. A general, well-known matching processing algorithm is also used for the matching process in the matching unit 16. Details of the algorithm are omitted here.

[0032] With reference to Fig. 5, the matching of identical feature points in two previous and next frames performed by the matching unit 16 will be described. That is, Fig. 5(A) shows an example of an image (central partial image) 202a of the previous frame, and Fig. 5(B) shows an example of an image (central partial image) 202b of the next frame. In the example of Fig. 5, the same feature points P1, P2, P3, P4, P5, and P6 detected in the image 202a of the previous frame in Fig. 5(A) are detected as feature points P1', P2', P3', P4', P5', and P6' in the image 202b of the next frame in Fig. 5(B). In order to clearly show the same feature points, Fig. 5 shows the same feature points in the previous and next frames connected by a dashed line.

[0033] The movement amount calculation unit 17 calculates the amount of movement between the same feature points matched by the matching unit 16. That is, the movement amount calculation unit 17 detects the amount of movement as the difference between the position coordinates in the previous frame of the feature points P1, P2, P3, P4, P5, and P6 detected in the image 202a of the previous frame and the position coordinates in the subsequent frame of the feature points P1', P2', P3', P4', P5', and P6' detected in the image 202b of the subsequent frame. In this case, the movement amount calculation unit 17 detects the amount of movement as the difference in position coordinates, and therefore detects it as a vector including the movement direction.

[0034] In Fig. 6(A), vector arrows are used to show an example of the amount of movement of feature points P1', P2', P3', P4', P5', and P6' detected in a subsequent frame relative to feature points P1, P2, P3, P4, P5, and P6 in a previous frame. For example, Fig. 6(A) shows a case where the moving direction of a car MV, which is a subject object, is a straight direction along the vertical direction of the frame.

[0035] Furthermore, in this embodiment, the movement amount calculation unit 17 also performs processing to find and remove outliers in the amount of movement (for example, see the amounts of movement indicated by dashed arrows Er1 and Er2 in FIG. 6(B)) that are due to erroneous detection of feature points by the feature point detection unit 15 or erroneous recognition. The method of detecting and removing these outliers can use a large difference in the amount of movement or a difference in the direction of movement. Since a well-known general algorithm can be used for this outlier removal processing, details thereof will be omitted here.

[0036] In this embodiment, the movement amount calculation unit 17 removes the outliers, calculates the average value of the calculated movement amounts, and outputs the average value of the movement amounts as a calculation result. By removing the outliers, the movement amount calculation unit 17 can improve the calculation accuracy of the movement amount.

[0037] The image to be combined generating unit 18 determines the image range to be combined according to the average value of the movement amount calculated by the movement amount calculating unit 17. Then, in this embodiment, as shown in Figures 7(A) and (B), the image to be combined generating unit 18 generates an image to be combined 203 (see Figure 7(B)) of the determined image range by cutting out from the central partial image 202 (see Figure 7(A)) cut out by the central portion cutting unit 14 as an image of a part in the combining direction (the vertical direction of the frame in this example).

[0038] In addition, if the traveling direction of the automobile MV is inclined relative to the vertical direction of the frame, that is, if the traveling direction of the automobile MV is not straight, as described above, the central partial image 202 is rotated and corrected so that the image combining direction matches the traveling direction of the automobile MV, and a part in the combining direction is cut out from the rotated and corrected central partial image 202 to generate the image to be combined 203.

[0039] In addition, for example, due to reasons such as dewarping correction not functioning properly, the amount of movement calculated by the movement amount calculation unit 17 may have a certain gap between the central part and both left and right end parts in the direction perpendicular to the joining direction, as shown by arrows Vc1, Vc2, and Vc3 in the central portion image of Figure 8(A).

[0040] In such a case, the image-to-be-combined generating unit 18 generates a curvature-corrected image 203 as shown in Fig. 8(B) based on the amount of movement calculated for the center portion and both left and right end portions, and then generates an image-to-be-combined 204 as shown in Fig. 8(C) from the curvature-corrected image 203. This makes it possible to make the image-to-be-combined 204 closer to an image captured from the front.

[0041] In this embodiment, the image to be combined generated by the combining image generating unit 18 is cut out from the central partial image and generated, but the image to be combined may also be cut out and generated from the image of the dewarped and corrected frame.

[0042] As shown in Fig. 9, the combining unit 19 sequentially combines the images to be combined 203 thus generated in the moving direction of the automobile MV, which is the subject object. In this case, the combining unit 19 performs image correction so that some images overlap at the combined portion and the combined portion becomes smooth. Note that in Fig. 9, the lower image is the combined image 205 obtained by combining the images to be combined up to that point.

[0043] [Flow of still image generation processing in image processing device 1] An example of the flow of the still image generating process in the information processing device 1 configured as above will be described with reference to the flowcharts of Fig. 10 to Fig. 12. Note that the following description is based on the assumption that each of the frame unit image acquiring unit 11, subject object appearance / disappearance detecting unit 12, dewarping correcting unit 13, central portion cropping unit 14, feature point detecting unit 15, matching unit 16, movement amount calculating unit 17, image to be combined generating unit 18, and combining unit 19 is configured as a functional block based on a software program included in the control unit 10. Therefore, the process of each step of the flowcharts shown in Fig. 10 to Fig. 12 will be described as being executed by the control unit 10.

[0044] The control unit 10 acquires captured image information from the camera 2 (step S101), and acquires image information in units of frames from the acquired captured image information (step S102). Next, the control unit 10 monitors the acquired image information in units of frames, and determines whether or not the subject object has been detected in the frame (step S103). As described with reference to FIG. 2, in this embodiment, the combining process is performed using the image information of the frame in the moving direction of the subject object, in this example, the image of the vertical center of the frame, so that when the subject object appears near the vertical center of the frame, it is determined that the appearance of the subject object has been detected, and when the subject object can only be detected in an area smaller than half the vertical area of ​​the frame, it is determined that the subject object has disappeared.

[0045] When it is determined in step S103 that the appearance of the subject object has been detected, the control unit 10 dewarps the frame image, which is a fisheye image, and corrects it to a planar image (step S106), as described with reference to Fig. 3. Next, the control unit 10 cuts out the vertical center portion, in this example, of the dewarped and corrected frame image 201, as shown in Fig. 4, to obtain a center portion image 202 (step S107).

[0046] Next, the control unit 10 detects feature points in the central partial image (step S108) as described with reference to Fig. 5. Then, the control unit 10 determines whether or not the same feature points as those detected in the image of the previous frame are present in the frame being processed (the above-mentioned subsequent frame) (step S111 in Fig. 11).

[0047] If it is determined in step S111 that the same feature points as those detected in the image of the previous frame are not present in the frame being processed, the control unit 10 retains the detected feature points as feature points of the previous frame (step S112), and then returns the process to step S102 to repeat the processes from step S102 onwards.

[0048] Furthermore, when it is determined in step S111 that the same feature points as those detected in the image of the previous frame are present in the frame being processed, the control unit 10 matches the feature points detected in the frame being processed with the feature points of the previous frame (step S113). Next, the control unit 10 calculates the amount of movement of each feature point together with the direction of movement from the coordinate difference between the matched feature points in the previous and previous frames (step S114).

[0049] Next, the control unit 10 performs a process of removing outliers from the amount and direction of movement of the feature points between the previous and next frames (step S115).Then, the control unit 10 determines whether or not there is still a movement amount even after removing the outliers (step S116). If there is no movement amount, that is, if the movement amount is zero, the control unit 10 performs the process of removing outliers from the feature points (step S117). The detected feature points are stored as feature points of the previous frame (step S117), and then the process returns to step S102 to repeat the processes from step S102 onwards.

[0050] Furthermore, when it is determined in step S116 that a movement amount exists even after the outliers are removed, the control unit 10 determines whether the movement direction after the outliers are removed is straight and coincides with the bonding direction (step S121 in FIG. 12). When it is determined in step S121 that the movement direction after the outliers are removed is not straight but is oblique to the bonding direction, the control unit 10 performs rotation correction on the image so that the movement direction after the outliers are removed coincides with the bonding direction (step S122).

[0051] In step S121, when it is determined that the movement direction after removing the outliers is straight and matches the combining direction, and following step S122, the control unit 10, in this example, cuts out the central portion from the central partial image in the direction along the movement direction determined based on the detected amount of movement, to generate an image for combining (step S123).

[0052] Next, the control unit 10 judges whether or not a combined image (combined with the image to be combined) is stored in the buffer memory (step S124), and when it is judged that the combined image is not stored in the buffer memory, it holds the image to be combined generated in step S123 in the buffer memory (step S125).When it is judged that the combined image is stored in the buffer memory, the control unit 10 combines the image to be combined generated in step S123 with the combined image and holds the combined image in the buffer memory (step S126).

[0053] After step S125 or step S126, the control unit 10 stores the feature points detected in step S108 in the buffer memory as feature points of the previous frame (step S127). After step S127, the control unit 10 returns the process to step S102 and repeats the processes from step S102 onward.

[0054] Then, when it is determined in step S103 of FIG. 10 that the object is no longer detected and the object has disappeared, the control unit 10 determines whether or not a combined image is stored in the buffer memory (step S104), and if the combined image is not stored, the control unit 10 returns to step S102 and repeats the processes from step S102 onward. Also, when it is determined in step S104 that the combined image is stored in the buffer memory, the control unit 10 saves the combined image as a still image file (step S105). The still image file is a still image of the object generated from the captured image information, and can be read out from the buffer memory and displayed, for example. After step S105, the control unit 10 returns to step S102 and repeats the processes from step S102 onward.

[0055] [Effects of the Still Image Generation Device of the Embodiment] According to the embodiment of the still image generating device described above, a sensor for detecting the running movement of the subject object is not required, which is advantageous in terms of cost and does not require installation space.

[0056] Furthermore, in the embodiment of the still image generating device described above, of the planar images obtained by dewarping image information captured by a camera using a fisheye lens, only the central part in the direction of movement of the subject object, which is the image part with the least distortion, is used to combine the images to generate a still image of the subject object, thereby making it possible to obtain a relatively accurate and good still image.

[0057] In addition, in the embodiment of the still image generating device described above, feature points of images on a frame-by-frame basis are detected, and an image to be combined is generated based on the amount of movement of the feature points in the previous and next frames, and a still image is generated by combining the image to be combined in the direction of movement, thereby achieving the remarkable effect of being able to respond to changes in the movement speed of the subject object.

[0058] Furthermore, in the embodiment of the still image generating device described above, when the direction of movement of a feature point between previous and next frames differs from the combining direction, the image is rotated and corrected in accordance with the moving direction so that the combining direction and the moving direction coincide with each other. This makes it possible to generate good still images even when the subject object does not travel in a straight line but travels at an angle.

[0059] Furthermore, in the embodiment of the still image generating device described above, when the dewarping of the fisheye image from camera 2 does not function properly and a difference occurs in the amount of movement of feature points between the central part and both ends, the image is curved and corrected according to the amount of movement to make it closer to a planar image captured from the front, so that when the images to be combined are combined, they can be combined more neatly.

[0060] Furthermore, in the embodiment of the still image generation device described above, outliers are removed from the amount and direction of movement of feature points between previous and next frames. This improves the accuracy of the amount and direction of movement for obtaining a combination image used to generate a still image, resulting in a better generated still image.

[0061] [Other embodiments or modifications] In the above example, a case where a still image is generated from captured image information of a video was described; however, this invention is not limited to videos and can also be applied to the case where a single still image is generated from image information of multiple frames captured in rapid succession by a fisheye lens camera.

[0062] Furthermore, the subject object to be photographed is not limited to the underside of the car body in the above example, but may be any object that can be photographed with a fisheye lens camera and a still image can be obtained from the captured image information.

[0063] In addition, in the above embodiment, a subject object that moves relative to a fixed camera position is photographed by a camera, but the present invention can also be applied to a case in which the subject object is placed in a fixed position and photographed by moving the camera.

[0064] Furthermore, the camera is not limited to photographing the bottom side of a relatively moving object, but the present invention can also be applied to photographing the side, top, front, back, etc. of an object.

[0065] In addition, in the above-described embodiment, the direction parallel to the vertical direction of the frame of the captured image is the relative movement direction (combining direction) of the subject object. However, in this invention, even when the subject object moves relatively with the direction parallel to the horizontal direction of the frame being the straight direction, a still image can be generated by performing the combining process in a similar manner by setting the image combining direction to the horizontal direction. [Explanation of symbols]

[0066] 1...image processing device, 2...camera, 2L...fisheye lens, 3...illumination device, 10...control unit, 11...frame-by-frame image acquisition unit, 12...subject object appearance / disappearance detection unit, 13...dewarping correction unit, 14...central portion extraction unit, 15...feature point detection unit, 16...matching unit, 17...movement amount calculation unit, 18...combination image generation unit, 19...combination unit

Claims

1. a camera that uses a fisheye lens to capture an image of a subject object that moves relative to a capture position; and an image processing device that generates a still image of the entire image of the subject object from captured image information from the camera; The image processing device includes: a correction unit that dewarps the captured image information from the camera on a frame-by-frame basis; a feature point detection unit that detects feature points from the frame-by-frame images corrected by the correction unit; a matching unit that performs matching between previous and next frames regarding the feature points detected by the feature point detection unit; a movement amount calculation unit that calculates a movement amount of the feature point between the previous and next frames based on a matching result by the matching unit; a combination image generating unit that generates a combination image for the image in frame unit based on the movement amount calculated by the movement amount calculating unit; a combining unit that combines the images to be combined generated by the image to be combined generating unit in the relative movement direction of the subject; A still image generating device comprising:

2. a cutout unit that cuts out a center portion of the subject object in the relative movement direction from the frame unit image corrected by the correction unit, The feature point detection unit detects feature points in the central image in the relative movement direction of the images in the frame units.

2. The still image generating device according to claim 1.

3. the movement amount calculation unit also calculates movement directions of the feature points in the previous and next frames based on a matching result by the matching unit; a rotation correction unit that, when the movement direction of the feature point is an oblique direction with respect to the vertical or horizontal direction of the frame, rotates and corrects the image of the frame in accordance with the calculated movement direction of the feature point; 3. The still image generating device according to claim 1 or 2.

4. When the amount of movement calculated by the movement amount calculation unit has a difference of a predetermined value or more between a center portion and both end portions in a vertical direction or a horizontal direction of the image of the frame that intersects with the relative movement direction of the subject object, the image for combination is obtained by correcting the curvature of the image based on the amount of movement calculated by the movement amount calculation unit.

3. The still image generating device according to claim 1 or 2.

5. An outlier is detected from the movement amount calculated by the movement amount calculation unit and is removed, and the image-to-be-combined generating unit generates an image to be combined from the images in the frame units based on an average value of the movement amounts excluding the outlier.

3. The still image generating device according to claim 1 or 2.

6. The image processing device includes: a subject appearance / disappearance detection unit that detects the appearance of the subject from the photographed image information from the camera and detects the disappearance of the subject from the photographed image information from the camera, A process of generating the still image from captured image information from the camera during the period from when the subject appearance / disappearance detection unit detects the appearance of the subject to when the subject disappears 3. The still image generating device according to claim 1 or 2.

7. A computer is provided in an image processing device that generates a still image of a whole image of a subject object that moves relative to a shooting position from captured image information from a camera that uses a fisheye lens to capture the subject object, A correction unit that dewarps the captured image information from the camera on a frame-by-frame basis; a feature point detection unit that detects feature points from the frame-by-frame images corrected by the correction unit; a matching unit that performs matching between previous and next frames regarding the feature points detected by the feature point detection unit; a movement amount calculation unit that calculates movement amounts of the feature points between the previous and next frames based on a matching result by the matching unit; a combination image generating unit that generates a combination image for the image in frame unit based on the movement amount calculated by the movement amount calculating unit; a combining unit that combines the images to be combined generated by the image to be combined generating unit in the relative movement direction of the subject; A still image generation program to function as a still image generator.

8. The computer further comprises: a subject appearance detection unit that detects whether an image of the subject appears in an image captured by the camera; a subject disappearance detection unit that detects whether or not an image of the subject has disappeared from an image captured by the camera; 8. The still image generating program according to claim 7, for causing the still image generating program to function as a still image generating program.

Images