Method of photo localization in space using video motion analysis

Abstract

The present invention relates to a method of photo localization in a space using video motion analysis and, more specifically, to a method for calculating the spatial position of a photographic image by using a video filmed by a video camera and a photographic image captured by a photographic camera during filming of the video. According to the present invention, the method of photo localization in a space using video motion analysis has an advantage of effectively finding a relative position of the photographic image in a three-dimensional shape, which can be identified by using video image frames, of a filming space.

Description

A method for finding the spatial location of a photographic image by dynamic analysis of a video.

[0001] The present invention relates to a method for finding the spatial position of a photographic image by dynamic analysis of a video, and more particularly, to a method for calculating the spatial position of a photographic image by using a video captured by a video camera and a photographic image captured by a photo camera during the capture of the video.

[0002] There are a variety of services available using photographic images. Because photographic images are typically captured in still images, they possess relatively high clarity. Therefore, using these relatively high-clarity images, it is possible to provide a variety of services.

[0003] For example, the position and orientation of each photographic image and the projection center point can be calculated using camera intrinsic parameters such as the relative position between the photographic image and the projection center point of the camera that took the photograph and the projection plane, and the size of the projection plane (CMOS image sensor, CCD image sensor, etc.). In this way, photographic images taken at different positions or angles for the same object can be used for various purposes such as constructing a 3D shape of the object, calculating the size or distance of the object, or providing 360 VR services.

[0004] This method of using photographic images allows for a higher quality service by taking as many photos as possible from various angles and positions. However, the drawback of taking such a large number of photos is that it requires a relatively large amount of time and effort.

[0005] Videos are composed of image frames captured continuously over time, offering the advantage of being able to capture a lot of image information in a relatively short period of time. In particular, wide-angle or 360-degree cameras offer the advantage of a wide field of view, allowing for a large amount of image information to be captured in a relatively short period of time. Conversely, when the cameraman moves while holding the video camera, the sharpness of the image frames in the captured video deteriorates depending on the camera's speed and rotation, resulting in blurriness. Video data captured with a moving camera is more likely to have low-sharpness image frames. Low-sharpness images lack clear outlines of objects, making it difficult to accurately identify the shape and outline of objects depicted in the image. Furthermore, low-sharpness images containing text also make it difficult to discern the text within the image.

[0006] When extracting and using image frames from data recorded in the form of a video in this way, various restrictions arise in specific use due to the problem of reduced clarity as described above.

[0007] For example, when extracting a 3D shape of a captured object using image frames from a video, the resulting 3D shape may be less accurate. Furthermore, when calculating the distance between two points in space using image frames from a video, the resulting calculations may be inaccurate.

[0008] Additionally, when implementing 360 VR video using image frames from 360-degree videos, if the area captured with low clarity is displayed on the screen of the display device, it can be difficult to confirm the specific shape of the desired object or object, such as text, shapes, structures, or people. For example, when trying to check the progress of building construction through a video, it can be difficult to confirm whether the main structures have been constructed accurately according to the drawings due to the low clarity of the image frames.

[0009] Recently, technologies that capture video footage of construction sites or building interiors to create and utilize three-dimensional visual documentation of those spaces are becoming increasingly widespread. These technologies are used for real estate brokerage and to document construction status at construction sites.

[0010] However, due to the limitations of video footage as described above, it may be difficult to effectively use it for purposes such as visual documentation.

[0011] To compensate for the limitations of such video, photos taken with high-resolution cameras are sometimes used. While these images offer high resolution and clarity, they have a relatively narrow field of view and struggle to capture a large number of photos in a short period of time. This limits the number of photos that can be captured, making it difficult to grasp the overall shape of a three-dimensional space.

[0012] For this purpose, while shooting a video, there may be cases where important parts of Visual Documentation are taken separately and used with a high-resolution camera.

[0013] When video and photo images are used together like this, it is difficult to determine the relationship between the video image frames and the photo images and calculate the shooting location of the photo images or the spatial location of the photo images. When examining the similarity between all image frames of the video and the photo images to determine the relationship between the video and photo images, the computational load and computational time increase dramatically.

[0014] Therefore, a method is needed to simultaneously take advantage of both video and photographic images by effectively understanding the relationship between video image frames and photographic images while reducing computational overhead and computational time.

[0015] The present invention has been devised to satisfy the above-described need, and its purpose is to provide a method for finding the spatial location of a photograph image by dynamic analysis of a video, which can effectively identify the relationship between a video and a photograph image by using a photograph image taken simultaneously during video shooting and quickly find the location of the photograph image in the shooting space.

[0016] In order to solve the above-described object, the present invention provides a method for finding a spatial position of a photograph image by dynamic analysis of a video using a video captured by a video camera and a photograph image captured by a photograph camera during the video capture, the method comprising: (a) a step in which a video receiving module receives a video captured by the video camera as a target video and receives and stores a camera intrinsic parameter of the video camera as video camera information; (b) a step in which a photo receiving module receives a target image in the form of a photograph image captured by the photo camera and receives and stores a camera intrinsic parameter of the photo camera as photo camera information; (c) a step in which a video position calculation module calculates a relative position of each image frame of the target video with respect to a shooting space using the image frame of the target video and the video camera information; (d) a step in which a low-speed image selection module selects, as low-speed image frames, image frames captured while the target camera moves relatively slowly compared to other image frames among the image frames of the target video; And (e) a step of a similar image selection module comparing the target image with the low-speed image frames and selecting image frames having a high similarity to the target image among the low-speed image frames as similar image frames; and (f) a step of a photo position calculation module calculating the position of the target image using the video camera information of the similar image frames, the positions of the similar image frames calculated by the video position calculation module, and the photo camera information of the target image.

[0017] The method of finding the spatial location of a photographic image by dynamic analysis of a video of the present invention has the advantage of quickly and effectively acquiring a large amount of image information from a recorded video, while also separately acquiring photographic images captured during video shooting, thereby effectively identifying the relationship between video image frames.

[0018] In addition, the method of finding the spatial position of a photographic image by dynamic analysis of a video of the present invention has the advantage of being able to effectively find the relative position of a photographic image in a three-dimensional shape of a shooting space that can be determined using video image frames.

[0019] FIG. 1 is a block diagram of a device for implementing an example of a method for finding a spatial location of a photographic image by dynamic analysis of a video according to the present invention.

[0020] FIG. 2 is a flowchart illustrating an example of a method for finding a spatial location of a photographic image by dynamic analysis of a video according to the present invention.

[0021] FIG. 3 and FIG. 4 are drawings for explaining a process of implementing a method for finding a spatial location of a photographic image by dynamic analysis of a video according to the present invention.

[0022] Hereinafter, with reference to the attached drawings, a method for finding the spatial location of a photographic image by dynamic analysis of a video according to an embodiment of the present invention will be described.

[0023] FIG. 1 is a block diagram of a device for implementing an example of a method for finding a spatial location of a photographic image by dynamic analysis of a video according to the present invention, and FIG. 2 is a flowchart for implementing an example of a method for finding a spatial location of a photographic image by dynamic analysis of a video according to the present invention.

[0024] Referring to FIG. 1, a device for implementing a method for finding a spatial position of a photograph image by dynamic analysis of a video according to the present invention comprises a video receiving module (100), a photo receiving module (200), a video position calculation module (300), a low-speed image selection module (400), a similar image selection module (500), and a photo position calculation module (600).

[0025] The video receiving module (100) receives and stores a target video in the form of a captured video. The target video received by the video receiving module (100) includes various types of videos composed of image frames that can be rendered over time. In the present embodiment, a video captured by a mobile device such as a mobile phone or a 360-degree camera is used as the target video. In some cases, the target video may be a 2D video captured by a mobile phone or camcorder, or a video captured and combined as a combination of hemispherical image frames to create a 360-degree VR video, or a video captured as spherical image frames. Any video data captured using a camera lens and capable of being sequentially played over time can be the target video. Hereinafter, a camera that captured such a target video will be referred to as a video camera. The video receiving module (100) receives the target video along with the camera intrinsic parameters of the video camera and stores them as video camera information. The relative position between the projection center point of the camera that took the photo or video and the projection plane and the size of the projection plane (CMOS image sensor, CCD image sensor, etc.) correspond to the camera-specific parameters.

[0026] Video receiving modules can also receive and store dynamic information about the movement of the camera at the time of capturing the target video. The dynamic information is stored as values ​​that can be mapped chronologically to the target video. This camera dynamic information is measured by various types of sensors, such as an inertial measurement unit (IMU), an acceleration sensor, a geomagnetic sensor, and an angular displacement sensor. Typically, this dynamic information is measured by a dynamic sensor installed on a mobile device with a camera installed. In the present embodiment, values ​​measured over time by an inertial measurement sensor are used as dynamic information.

[0027] The video receiving module (100) receives and stores such dynamic information as a value corresponding to the playback time of the target video.

[0028] The photo receiving module (200) receives and stores a target image in the form of a photo image captured by a camera. Hereinafter, the camera that captured such a target image will be referred to as a photo camera. The photo camera is a camera that a user carries along with a video camera and captures a target video while shooting the video. While carrying the video camera and shooting the target video, the user uses the photo camera to capture the target image when character recognition or a clear image is required. Typically, the photo camera is a separate device from the video camera, but in some cases, the video camera and the photo camera may be installed separately on the same mobile device.

[0029] The photo receiving module (200) receives and stores the camera-specific parameters of the photo camera along with the target image as photo camera information.

[0030] The video position calculation module (300) calculates the relative position of each image frame of the target video with respect to the shooting space using the image frame of the target video and the video camera information. The video position calculation module (300) can calculate the position of the camera that captured each image frame by using the relationship between the video camera information and the video image frames. In addition, the video position calculation module (300) can also calculate the relative position of the video image frames in the space where the target video was captured. The position of the video camera and the positions of the image frames can be calculated using known methods such as SfM (Structure-from-Motion) or SLAM (Simultaneous Localization and Mapping).

[0031] In some cases, the video position calculation module (300) can additionally use dynamic information of the video camera to calculate the position of the camera. The video position calculation module (300) can also calculate the position at which the target video was shot based on the time of the video by using the dynamic information received by the video receiving module (100). The video position calculation module (300) can calculate the position at which the target video was shot by using sensor measurement values ​​such as an inertial measurement sensor and an acceleration sensor at the time of shooting the target video, and if necessary, can also calculate the direction of each image frame of the target video. For example, since the amount of change in displacement can be calculated by integrating the acceleration twice, the position at which the corresponding image frame was shot can be calculated by using the value calculated in this way. The direction of the camera can also be calculated by using the measurement values ​​of an angular velocity sensor such as a gyroscope. In some cases, various other methods such as Visual Odometry or Machine Learning may be used to calculate the position and direction for the image frame of the target video.

[0032] The low-speed image selection module (400) selects image frames that are captured while moving relatively slowly compared to other image frames of the target video as low-speed image frames. The low-speed image selection module (400) can calculate the speed, angular speed, and velocity of the video camera over time using the values ​​calculated by the video position calculation module (300). The low-speed image selection module (400) selects all or part of the image frames captured while the video camera is moving at a relatively low speed and angular speed as low-speed image frames. Various criteria and methods can be used for selecting such low-speed image frames.

[0033] When the speed and angular velocity of the video camera are within the reference speed and reference angular velocity, respectively, the low-speed image selection module (400) may select at least some of the image frames of the target video mapped to the corresponding time as low-speed image frames. The values ​​of the reference speed and reference angular velocity may be designated as appropriate values ​​in advance.

[0034] Alternatively, if it is determined that the video camera has stayed within the reference area range for a preset reference time, the low-speed image selection module (400) may select at least some of the image frames captured during that time as low-speed image frames. The reference period and the range of the reference area may also be preset to appropriate values.

[0035] If the video receiving module (100) also receives dynamic information, the low-speed image selection module (400) may additionally utilize the dynamic information. The low-speed image selection module (400) may utilize the dynamic information to identify a section in which the video camera moves relatively slowly and select image frames captured in that section as low-speed image frames.

[0036] Once low-speed image frames are selected in this manner, the similar image selection module (500) compares the target image with the low-speed image frames and selects at least some of the low-speed image frames with high similarity as similar image frames. In selecting the similar image frames, the similar image selection module (500) may use an image retrieval algorithm such as BoW or VLAD.

[0037] The present invention performs similarity determinations on only low-speed image frames, rather than comparing all image frames of a target video with the target image. Therefore, the present invention can drastically reduce the time and computational effort required to select similar image frames from a target video.

[0038] The photo position calculation module (600) calculates the position of the target image using the video camera information of the similar image frames, the positions of the similar image frames calculated by the video position calculation module (300), and the photo camera information of the target image. The photo position calculation module (600) can calculate the position of the target image from the similar image frames and the positions of the similar image frames using a method such as the PnP (Perspective-n-Point) algorithm. In addition, the photo position calculation module (600) can also calculate the average distance from the target image to the subject.

[0039] The display module (700) displays a target image on a display device. At this time, the display module (700) can display the target image on the display device in various ways. When displaying image frames of a target video, the display module (700) can display the target image by overlapping it with an image frame that is most similar to the target image. In addition, the display module (700) can display the target image on the display device by arranging the target image in a corresponding position and direction in a three-dimensional space implemented by calculating using the video image frames. In addition, the display module (700) can display the target image on the display device by arranging the target image in a position corresponding to the average distance between the target image and the subject calculated in advance. Alternatively, the display module (700) can display the target image on the display device by aligning the position and direction of the target image with a three-dimensional object implemented by the image frames of the target video.

[0040] Hereinafter, a process of performing a method for finding a spatial location of a photographic image by dynamic analysis of a video according to the present invention using a device configured as described above will be described.

[0041] First, the video receiving module (100) receives and stores the target video and video camera information (step (a); S100). As described above, the target video may be of various types depending on the intended use and purpose. In some cases, the video receiving module (100) also receives and stores dynamic information regarding the shooting of the target video.

[0042] The photo receiving module (200) also receives and stores target images and photo camera information (step (b); S200). While there may be multiple target images, the present embodiment will describe the case where the present invention is implemented for a single target image as an example. If there are multiple target images, the process described in this embodiment can be repeated for each target image.

[0043] Next, the video position calculation module (300) calculates the relative position of each image frame of the target video with respect to the shooting space (step (c); S300). As described above, by using a method such as SfM or SLAM using the video camera information and the image frames, the position and direction of the video camera can be calculated, and in some cases, the shape of the three-dimensional object captured can also be constructed using the two-dimensional image frames. At this time, in some cases, the position and direction of the video camera can be calculated by additionally using the dynamic information received and stored in step (a).

[0044] In this way, when the positions and directions of the image frames of the target video are calculated, the low-speed image selection module (400) selects the low-speed image frames based on this (step (d); S400). As described above, various criteria and methods can be used to select image frames (low-speed image frames) captured while the video camera moves relatively slowly among the image frames of the target video.

[0045] When low-speed image frames are selected in this way, the similar image selection module (500) compares the low-speed image frames with the target image and selects image frames with a high similarity to the target image as similar image frames (step (e); S500). As described above, various known algorithms can be used to determine similarity. In addition, various criteria for selecting similar image frames can also be selected. By specifying a standard similarity in advance, only image frames exceeding the standard similarity can be selected as similar image frames by the similar image selection module (500).

[0046] The photo position calculation module (600) calculates the position and direction of the target image using similar image frames and video camera information, the position and direction of the similar image frames, the target image, and the photo camera information of the target image (step (f) S600). As described above, by using a method such as the PnP algorithm, the position and direction of the target image can be calculated using the previously calculated information. In this way, if the positions of the image frames of the target video and the corresponding position and direction of the target image are known, they can be used for various purposes.

[0047] In particular, the present invention effectively overcomes the limitations of video, which allows for obtaining a large amount of image information in a relatively short period of time but makes it difficult to identify specific and accurate information about a specific point due to resolution limitations. By taking high-resolution photographs as target images for points requiring character recognition or accurate shape identification, the shortcomings of target videos can be compensated for. In particular, by calculating the position and direction of the target image to correspond to the position and direction of the image frames of the target video, there is an advantage in that the corresponding target image can be easily identified while confirming the image frames of the target video.

[0048] Furthermore, as described above, since the similarity test is performed by comparing only low-speed image frames with the target image, rather than comparing all image frames of the target video with the target image, the computational load can be drastically reduced. Therefore, there is the advantage of conveniently mapping photographic images, captured separately from the necessary portions during video recording, to the image frames of the video.

[0049] When using the method of finding the spatial location of a photographic image by dynamic analysis of a video of the present invention, the user can capture a relatively high-resolution photographic image with a camera while moving relatively slowly or stationary when shooting a video of an area requiring further confirmation, and use it together with the target video. The present invention is characterized by correlating the target video and the target image by taking advantage of the user's photographing movements.

[0050] As an example of a method of using the result of calculating the position and direction of the target image in step (f), the display module (700) displays the target image on a display device in conjunction with the target video (step (g); S700).

[0051] The display module (700) may display the target image by superimposing it on one of the low-speed image frames corresponding to the position and direction of the target image, or the display module (700) may display the target image on a display device by aligning it with a three-dimensional object implemented using image frames of the target video.

[0052] In some cases, the path within the building where the target video was filmed may be displayed on a display device as shown in FIG. 3, and when a user selects a point on the path, the display module (700) may be operated to display a target image at a location corresponding to the selected point on the display device as shown in FIG. 4.

[0053] As described above, the method of providing various services using target videos and target images in the present invention can be applied to all types of videos, including general two-dimensional videos as well as 360-degree VR videos.

[0054] Although the present invention has been described above with preferred examples, the scope of the present invention is not limited to the embodiments described above.

[0055] For example, although the display module (700) has been described above as displaying a target image on a display device through step (g), the present invention can also be implemented without including step (g). The calculated position and direction of the target image can be output as result values ​​and stored, and the values ​​can be used in various ways by other services. Even if the target image is displayed on a display device through step (g), various display methods other than the method described above as an example can be used.

[0056] In addition, although it has been described above that dynamic information from a video camera is received and stored in step (a) and that the dynamic information is used in step (c), it is also possible to not use the dynamic information. In other words, it is also possible to implement the present invention using only video camera information and video without receiving or using the dynamic information.

[0057] While video cameras and photo cameras were described separately above, in some cases, both can be installed and used on a single mobile device. In some cases, the video camera and photo camera may even be the same. In such cases, the camera may operate by shooting a video and then instantly capturing a high-resolution photo.

Claims

1. A method for finding the spatial location of a photograph image by dynamic analysis of a video using a video captured by a video camera and a photograph image captured by a photo camera during the video capture, (a) A step in which a video receiving module receives a video captured by the video camera as a target video and receives and stores camera intrinsic parameters of the video camera as video camera information; (b) a step of receiving a target image in the form of a photographic image captured by a photographic camera by a photographic receiving module and receiving and storing camera-specific parameters of the photographic camera as photographic camera information; (c) a step of calculating the relative position of each image frame of the target video with respect to the shooting space using the image frame of the target video and the video camera information by the video position calculation module; (d) a step of a low-speed image selection module selecting image frames captured while the target camera moves relatively slowly compared to other image frames among the image frames of the target video as low-speed image frames; and (e) a step of the similar image selection module comparing the target image with the low-speed image frames and selecting image frames having a high similarity to the target image among the low-speed image frames as similar image frames; and (f) A method for finding the spatial position of a photograph image by dynamic analysis of a video, comprising: a step of calculating the position of the target image by using the video camera information of the similar image frames and the positions of the similar image frames calculated by the video position calculation module and the photograph camera information of the target image; 2. In paragraph 1, The step (c) above is a method for finding the spatial location of a photograph image by dynamic analysis of a video performed by the video location calculation module using one of SfM (Structure-from-Motion) and SLAM (Simultaneous Localization and Mapping).

3. In paragraph 1, The step (f) above is a method for finding the spatial position of a photograph image by dynamic analysis of a video in which the photograph position calculation module calculates the position of the target image using a PnP (Perspective-n-Point) algorithm.

4. In paragraph 1, The above step (a) is a method for finding the spatial location of a photographic image by dynamic analysis of a video, in which the video receiving module additionally receives and stores the measurement value of the dynamic sensor of the video camera corresponding to the playback time of the target video as dynamic information.

5. In paragraph 4, The step (a) above is a method for finding the spatial location of a photographic image by dynamic analysis of a video that receives a measurement value of an inertial measurement sensor (IMU) as the dynamic information as a measurement value of the dynamic sensor.

6. In paragraph 4, The step (c) above is a method for finding the spatial position of a photograph image by dynamic analysis of a video, wherein the video position calculation module additionally uses the dynamic information to calculate the positions of image frames of the target video.

7. In paragraph 4, The step (d) above is a method for finding the spatial location of a photographic image by dynamic analysis of a video in which the low-speed image selection module selects the low-speed image frame using the dynamic information.

8. In paragraph 4, The step (d) above is a method for finding the spatial location of a photographic image by dynamic analysis of a video, wherein the low-speed image selection module selects at least some of the image frames of the target video corresponding to the playback time as low-speed image frames when the measurement value of the dynamic sensor is within the reference speed and reference angular speed.

9. In paragraph 1, The step (d) above is a method for finding the spatial location of a photographic image by dynamic analysis of a video, wherein the low-speed image selection module selects at least some of the image frames as low-speed image frames when the video camera moves at a speed lower than a preset reference speed.

10. In paragraph 1, The step (d) is a method for finding the spatial location of a photographic image by dynamic analysis of a video, wherein, when the low-speed image selection module determines that the video camera has stayed within a reference area range for a preset reference time, at least some of the image frames captured during that time are selected as low-speed image frames.

11. In paragraph 1, (g) A method for finding a spatial location of a photographic image by dynamic analysis of a video, further comprising: a step of displaying the target image on a display device by a display module; 12. In paragraph 11, (g) Step, A method for finding the spatial location of a photographic image by dynamic analysis of a video in which the display module superimposes the target image on one of the low-speed image frames using the location of the target image calculated in the step (f).

13. In paragraph 11, (g) Step, A method for finding the spatial position of a photographic image by dynamic analysis of a video in which the display module aligns the target image with a three-dimensional object implemented by image frames of the target video using the position of the target image calculated in the step (f).

Images