Information processing device, information processing method, and information processing program

Abstract

To provide an information processing apparatus, an information processing method, and an information processing program for recording displacement of a target without adding a facility such as a sensor.SOLUTION: In a green of a golf course, an information processing apparatus 30 for detecting a position of a hole cup which is changed every day includes: an image acquisition unit configured to acquire a sequence of images captured while changing a viewpoint; a feature point extraction unit configured to extract a plurality of feature points from the sequence of images; a target detection unit configured to detect a target feature point indicating a detection target from among the plurality of feature points; a key point setting unit configured to set a plurality of key points from among the plurality of feature points; and a position information generation unit configured to generate position information of the detection target on the basis of relative positions between the plurality of key points and the target feature point. Each of the key points is a feature point indicating the same feature out of the feature points extracted from each of a plurality of sets of the sequence of images.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present disclosure relates to an information processing device, an information processing method, and an information processing program. [Background technology]

[0002] Generally, the position of the hole cup in a golf course is changed daily so as not to locally damage the grass of the green. Patent Document 1 discloses a method of recording the changing hole cup position by arranging a sensor at the hole cup position. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2018-191985 A Summary of the Invention [Problem to be solved by the invention]

[0004] When a sensor is used to detect the hole cup position, equipment such as a receiver for receiving a signal transmitted from the sensor is required. Such equipment requires cost and effort to install and maintain. In addition, depending on the object to record the displacement, it may not be possible to attach a sensor. Therefore, it is desirable to be able to record the displacement of the object without adding equipment such as a sensor.

[0005] An object of the present disclosure is to provide an information processing device, an information processing method, and a program capable of recording the displacement of an object. [Means for solving the problem]

[0006] An information processing device according to one aspect of the present disclosure includes an image acquisition unit configured to acquire a series of images captured while changing a viewpoint, a feature point extraction unit configured to extract a plurality of feature points from the series of images, a target detection unit configured to detect a target feature point indicating a detection target from the plurality of feature points, a key point setting unit configured to set a plurality of key points from the plurality of feature points, and a position information creation unit configured to create position information of the detection target based on relative positions between the plurality of key points and the target feature point, wherein each of the key points is a feature point indicating the same feature among the feature points extracted from each of a plurality of sets of the series of images.

[0007] An information processing method according to one embodiment of the present disclosure includes having one or more computers execute the following: acquiring a series of primary images taken while changing a viewpoint; extracting a plurality of primary feature points from the series of primary images; acquiring a series of secondary images taken while changing a viewpoint so as to include a detection target; extracting a plurality of secondary feature points from the series of secondary images; detecting a secondary target feature point indicative of the detection target from the plurality of secondary feature points; setting a plurality of key points, each of which is one of the plurality of secondary feature points that indicates the same feature as one of the plurality of primary feature points; and generating position information of the detection target based on relative positions of the plurality of key points and the secondary target feature points.

[0008] An information processing program according to one embodiment of the present disclosure is a program for causing one or more computers to execute the following: acquiring a series of primary images taken while changing a viewpoint; extracting a plurality of primary feature points from the series of primary images; acquiring a series of secondary images taken while changing a viewpoint so as to include a detection target; extracting a plurality of secondary feature points from the series of secondary images; detecting a secondary target feature point indicative of the detection target from the plurality of secondary feature points; setting a plurality of key points, each of which is one of the plurality of secondary feature points that indicates the same feature as one of the plurality of primary feature points; and generating position information of the detection target based on relative positions of the plurality of key points and the secondary target feature points. Effect of the Invention

[0009] According to the present disclosure, it becomes possible to record the displacement of an object. [Brief description of the drawings]

[0010] [Figure 1] FIG. 1 is a schematic diagram illustrating the configuration of an information processing device according to an embodiment. [Diagram 2] FIG. 2 is a functional block diagram of the information processing device of FIG. [Diagram 3] FIG. 3 is a schematic diagram for explaining the displacement of a detection target and a method for capturing a moving image. [Figure 4] FIG. 4 is a schematic diagram showing a state during video shooting. [Diagram 5] FIG. 5 is a schematic diagram illustrating a plurality of primary feature points extracted from a primary video. [Figure 6] FIG. 6 is a schematic diagram illustrating a plurality of secondary feature points extracted from the secondary video and a plurality of set key points. [Figure 7] FIG. 7 is a schematic diagram illustrating the generated three-dimensional data. [Figure 8] FIG. 8 is a plan view generated from three-dimensional data. [Figure 9]FIG. 9 is a flow diagram illustrating an information processing method according to the embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] Examples of the information processing device, information processing method, and information processing program of the present disclosure will be described below with reference to the drawings. The present invention is not limited to these examples, but is defined by the claims, and is intended to include all modifications within the meaning and scope of the claims.

[0012] [Information Processing System] An information processing device 30 for detecting the position of a hole cup 13, which changes daily, within a green 12 of a golf course 11 illustrated in FIG. 1 will be described. The position of the hole cup 13 is detected from a series of images 15 (see FIG. 4) taken by a photographing device 21 operated by a photographer 20. A "series of images" refers to a plurality of still images or videos taken while changing the viewpoint. The photographing device 21 may be, for example, but is not limited to, a portable digital camera, a video camera, a smartphone, or a tablet. It is preferable that the photographing device 21 be capable of taking videos. Hereinafter, the "series of images 15" may be simply referred to as images 15 or video 15.

[0013] The video 15 captured by the cameraman 20 may be stored in a user terminal 22. The user terminal 22 may be placed, for example, in a management facility 14 arranged alongside the golf course 11. The user terminal 22 is, for example, a personal computer. The user terminal 22 may be configured to communicate with the information processing device 30 via a network 23. In this case, the video 15 stored in the user terminal 22 may be transmitted to the information processing device 30 via the network 23.

[0014] The network 23 includes, for example, the Internet, a Wide Area Network (WAN), a Local Area Network (LAN), a provider terminal, a wireless communication network, a wireless base station, a dedicated line, etc. When the photographing device 21 is a terminal having a communication function, for example, a smartphone or a tablet, the photographed image may be transmitted directly to the information processing device 30 without going through the user terminal 22.

[0015] The information processing device 30 is, for example, a server or a computer including a processor 31, a memory 32, and a communication IF (interface) 33. The information processing device 30 may further include an input device 34, an output device 35, and a storage 36. The processor 31, the memory 32, the communication IF 33, the input device 34, the output device 35, and the storage 36 are connected to each other by a communication bus 37.

[0016] The processor 31 is, for example, a processing circuit configured to execute various software processes. The processing circuit may include a dedicated hardware circuit (e.g., ASIC, etc.) that processes at least a part of the software processes. In other words, the software processes may be executed by a processing circuitry that includes at least one of one or more software processing circuits and one or more dedicated hardware circuits.

[0017] The processor 31 is, for example, a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor unit (MPU), a field-programmable gate array (FPGA), or other arithmetic device. The processor 31 executes a series of instructions included in a program stored in the memory 32 or the storage 36 in response to a given signal or in response to the establishment of a predetermined condition.

[0018] The memory 32 is, for example, a random access memory (RAM) or other volatile memory. The memory 32 is configured to temporarily store programs and data. The programs are read from, for example, the storage 36. The data stored in the memory 32 may include data received by the information processing device 30 and data generated by the processor 31.

[0019] The communication IF 33 is configured to connect to the network 23. The communication IF 33 is configured to communicate with other devices connected to the network 23. The communication IF 33 is realized, for example, as a LAN or other wired communication IF. The communication IF 33 can also be realized, for example, but not limited to, Wi-Fi (registered trademark), Bluetooth (registered trademark), or other wireless communication IF.

[0020] The input device 34 is, for example, a keyboard and a mouse, and may include a button, a key, a switch, a touch pad, or a microphone. The output device 35 may be, for example, a liquid crystal monitor or an organic EL (Electro Luminescence) monitor, or may be a touch screen having a touch panel that also serves as the input device 34. The output device 35 may be a speaker or a terminal for audio output.

[0021] The storage 36 is a type of memory and may be, for example, a ROM (Read-Only Memory), a hard disk drive, a flash memory, or other non-volatile storage device. The storage 36 may be a removable storage device such as a memory card. The storage 36 is configured to permanently store programs and data. The storage 36 may store one or more applications 38 for implementing various functions and data 41 to 46, which will be described later. The location information data 46 may include map data of the golf course 11.

[0022] 2, the information processing device 30 may include an image acquisition unit 51, a feature point extraction unit 52, an object detection unit 53, a key point setting unit 54, a three-dimensional data generation unit 55, and a position information creation unit 56. The image acquisition unit 51, the feature point extraction unit 52, the object detection unit 53, the key point setting unit 54, the three-dimensional data generation unit 55, and the position information creation unit 56 may be realized as functions of an application 38 executed by the processor 31, or may be realized as a plurality of devices (e.g., computers) 51-56.

[0023] For example, the computer 51 for acquiring the video 15, the computer 52 for extracting the feature points 62, the computer 54 for setting the key points 64, the computer 55 for generating the three-dimensional data 45, and the computer 56 for creating the position information may communicate with each other through the network 23. Alternatively, the information processing device 30 may be realized by implementing some of the image acquisition unit 51, the feature point extraction unit 52, the object detection unit 53, the key point setting unit 54, the three-dimensional data generation unit 55, and the position information creation unit 56 as one device (computer). In this case, the information processing system including the computers 51 to 55 becomes the information processing device 30 as a whole.

[0024] The image acquisition unit 51 is configured to acquire the video 15 transmitted from the user terminal 22 or the information processing device 30 and store it as image data 41 in the storage . The feature point extraction unit 52 is configured to extract a plurality of feature points 62 (indicated by white circles in Figs. 5 and 6) from the video 15. A method for extracting feature points from an image is disclosed in, for example, Japanese Patent No. 4689758, but is not limited to this. The feature points 62 are characteristic parts in an image, and typical examples include the corners or edges of an object. The feature point extraction unit 52 stores information related to the extracted feature points 62 in the storage 36 as feature point data 42.

[0025] The target detection unit 53 refers to the target data 43 registered in advance and detects one or more target feature points 62t (see FIG. 6) indicating the detection target from among the multiple feature points 62. The target data 43 includes information indicating the feature of the detection target. The detection target in this example is the hole cup 13, but in addition to this, or instead of this, the pin 24 indicating the position of the hole cup 13 (see FIG. 6) may also be the detection target. The target feature points 62t may also be stored in the storage 36 as feature point data 42 together with the other feature points 62.

[0026] The key point setting unit 54 is configured to set a plurality of key points 64 (indicated by black circles in white circles in FIG. 6) from among the extracted plurality of feature points 62. A key point 64 is a feature point 62 that indicates the same feature among the feature points 62 extracted from each of a plurality of videos 15 (a plurality of sets of a series of images 15). For example, a primary feature point 62f extracted from a video 15 shot the previous day and a secondary feature point 62s extracted from a video 15 shot today are matched, and the feature point 62 that indicates the same feature becomes the key point 64.

[0027] Specifically, if a part (e.g., a fruit) of an object (e.g., a tree) is photographed on different days, that part will show the same characteristics in multiple videos. Conversely, feature points extracted from multiple videos of different objects will not show the same characteristics. In other words, the setting of key point 64 means that the same object (part) is photographed across multiple videos.

[0028] Of the extracted feature points 62, the target feature point 62t is excluded from candidates for the key point 64. In other words, the feature points 62 remaining after excluding the target feature point 62t from the multiple feature points 62 become candidates for the key point 64. The key point setting unit 54 stores information related to the set key point 64 in the storage 36 as key point data 44.

[0029] The key points 64 are not limited to matching the latest feature points 62 with the previous feature points 62, but may be matched with feature points 62 from a specified number of times before, or may be matched with feature points 62 a specified number of times (e.g., three times). In this case, the storage 36 may store at least feature point data 42 related to feature points 62 for a specified number of times and key point data 44 related to key points 64. The key point setting unit 54 may be configured to match feature points 62 extracted from different videos 15, respectively, and delete key points 64 that do not match after a specified number of times of matching (e.g., one or more times).

[0030] In the case where the matching of the feature points 62 is performed only with the previous feature points 62, the storage 36 may store the feature point data 42 related to the previous feature points 62 and the key point data 44 related to the multiple key points 64 set by the key point setting unit 54 from the previous feature points 62. Then, when the image acquisition unit 51 acquires a new series of images 15, the feature point extraction unit 52 extracts new multiple feature points 62 from the new series of images 15, the target detection unit 53 detects new target feature points 62t indicating the detection target from the new multiple feature points 62, and the key point setting unit 54 matches the new feature points 62 with the previous feature points 62, and updates the key point data 44 with the feature points 62 indicating the same feature as the new key points 64.

[0031] The three-dimensional data generating unit 55 is configured to generate three-dimensional data 45 (see FIG. 7) from the video 15, i.e., a series of two-dimensional images. This is a process called SfM (Structure From Motion), and reproduces the three-dimensional structure of the photographed subject. The three-dimensional data generating unit 55 stores the generated three-dimensional data 45 in the storage 36. The three-dimensional data 45 includes a target feature point 62t and a plurality of key points 64 within the reproduced three-dimensional structure.

[0032] The position information creation unit 56 is configured to detect a position Td (see FIG. 8) of the detection target based on the relative positions of the multiple key points 64 and the target feature point 62t. The position information creation unit 56 may be configured to create position information of the detection target from the three-dimensional data 45. For example, the position information creation unit 56 may be configured to generate a plan view 46 (see FIG. 8) including the position Td of the detection target from the three-dimensional data 45.

[0033] 8, plan view 46 shows detection position Td of hole cup 13 within green 12. It is preferable that plan view 46 also shows boundary line Gd of green 12. If the entire boundary line of green 12 is captured in image 15, boundary line Gd can be easily reproduced in three-dimensional data 45. Furthermore, position information creation unit 56 may be configured to plot detection position Td on map data of the area (golf course 11) in which the detection target exists by overlaying plan view 46 on the map data.

[0034] [Photographing the detected object] 3, in a golf course 11, the position of a hole cup 13 is changed frequently (e.g., daily) within the green 12, mainly to prevent localized damage to the grass of the green 12. In this example, a video 15 captured by a cameraman 20 using a camera device 21 is used to detect the position of the hole cup 13, which is the detection target.

[0035] 3 and 4, photographer 20 captures video 15 by capturing an environment including a detection target (hole cup 13) while changing the viewpoint. Photographer 20 may capture an image each time the position of hole cup 13 is changed. Photographer 20 should capture an image so that video 15 includes hole cup 13, the boundary line of green 12, and one or more reference objects 16-18 that exist around hole cup 13 and green 12.

[0036] The objects 16-18 are preferably objects that are fixed in the surrounding environment of the detection target and whose position, shape, and size are unlikely to change. The objects 16-18 may include, for example, trees, buildings, or structures. In this example, the objects 16-18 are trees around the green 12. The object 16 is, for example, a deciduous tree, and the objects 17 and 18 are, for example, evergreen trees. Depending on the shape and size of the objects 16-18, the photographer 20 may take pictures with a vertical or horizontal angle of view.

[0037] Preferably, the video 15 includes the entire boundary of the green 12. Therefore, the photographer 20 may take pictures while circling the green 12. Alternatively, the photographer 20 may stand within the green 12 and take pictures while changing the shooting direction 360 degrees. After finishing taking pictures, the photographer 20 saves the video 15 on the user terminal 22, or transmits it to the user terminal 22 or the information processing device 30.

[0038] The image acquisition unit 51 receives the transmitted video 15 and stores it in the storage 36. Here, the video 15 captured the first time (first time) is referred to as a primary video (series of primary images) 15f, and the video 15 captured the next time (second time or set time) is referred to as a secondary video (series of secondary images) 15s. In addition, the feature points 62 extracted from the primary video 15f are referred to as primary feature points 62f, and the feature points 62 extracted from the secondary video 15s are referred to as secondary feature points 62s. Furthermore, the target feature points 62t detected from the primary feature points 62f are referred to as primary target feature points 62tf, and the target feature points 62t detected from the secondary feature points 62s are referred to as secondary target feature points 62ts.

[0039] [Detection method for target] FIG. 9 illustrates a method for detecting the position of the hole cup 13. First, in step S11, the image acquisition unit 51 acquires the primary moving image 15f. In step S12, the feature point extraction unit 52 extracts a plurality of primary feature points 62f from the primary moving image 15f (see FIG. 5).

[0040] In step S13, the target detection unit 53 detects a target feature point 62tf indicating the hole cup 13 from among the multiple primary feature points 62f (see FIG. 5). In FIG. 5, the detected target feature point 62tf is marked by surrounding it with a dashed square. If it is not necessary to detect the position of the hole cup 13 for the first time, the detection in step S13 may be omitted. In other words, the target feature point 62tf does not have to be detected in step S13.

[0041] Next, in step S14, the image acquisition unit 51 acquires the secondary video 15s. In step S15, the feature point extraction unit 52 extracts a plurality of secondary feature points 62s from the secondary video 15s (see FIG. 6). In step S16, the target detection unit 53 detects secondary target feature points 62ts indicating the hole cup 13 from the plurality of secondary feature points 62s (see FIG. 6). In FIG. 6, the detected target feature points 62ts are marked by surrounding them with dashed squares.

[0042] In step S17, the key point setting unit 54 sets a plurality of key points 64 from among the plurality of secondary feature points 62s (see FIG. 6). Each key point 64 is one of the plurality of secondary feature points 62s that shows the same feature as one of the plurality of primary feature points 62f. Therefore, for example, feature points 62s that were only captured the first time are excluded, and feature points 62s that constantly exist around the hole cup 13 for at least a certain period of time become key points 64.

[0043] In step S18, the three-dimensional data generating unit 55 generates three-dimensional data 45 from the secondary video 15s (see FIG. 7). Furthermore, in step S19, the position information creating unit 56 generates position information, and the process ends. More specifically, the position information creating unit 56 detects the position Td of the hole cup 13 at the time of acquiring the secondary video 15s, based on the relative positions of the key points 64 and the secondary target feature points 62ts. Then, the position information creating unit 56 generates a plan view 46 (see FIG. 8) from the three-dimensional data 45. This plan view 46 shows the boundary line Gd of the green 12 as well as the position Td of the hole cup 13. Therefore, the position Td of the hole cup 13 within the green 12 at the time of the second shooting is grasped.

[0044] As for the tertiary moving image (series of tertiary images) 15t captured the third time (update time) after the second time, the image acquisition unit 51 acquires it as in step S14, and as in step S15, the feature point extraction unit 52 extracts a plurality of tertiary feature points 62. Also, as in step S16, the target detection unit 53 detects a tertiary target feature point 62t from among the plurality of tertiary feature points 62.

[0045] Then, as in step S17, the key point setting unit 54 sets the key points 64, thereby updating the key points 64. In this manner, the key points 64 are updated every time a new video 15 is acquired. In this disclosure, "setting" the key points 64 includes "updating" the key points 64, but the term "update" may be used to emphasize that this is setting for the second or subsequent times.

[0046] Finally, similarly to step S19, the position information creation unit 56 generates position information of the hole cup 13 at the time of the third photographing based on the relative positions between the updated multiple key points 64 and the tertiary target feature points 62t. Similarly, for the images 15 photographed at the fourth photographing (second update) and thereafter after the third photographing, the key points 64 are updated and the position information is generated.

[0047] It is preferable that the primary video 15f is carefully shot, for example, along the entire perimeter of the green 12 so as to include the entire interior of the green 12. This allows all of the feature points 62f around the green 12 to be grasped without omission, and allows the three-dimensional data 45 to be generated more precisely.

[0048] In contrast, the shooting time of video 15 shot the second or subsequent times may be shorter than the shooting time of primary video 15f shot the first time. For example, it is sufficient to obtain image 15 that includes some of objects 16-18, part of the boundary of green 12, and hole cup 13, and it is not necessary to shoot the entire circumference of green 12 or the entire green 12.

[0049] [Effects of the present disclosure] According to the method disclosed herein, the image 15 can be obtained by photographing the surroundings of the green 12 on foot, so there is no need to photograph the hole cup 13 from the air using a drone, for example. Therefore, the cost of photography is low. Also, from the second time onwards, the key points 64 can be set even in a short video 15, so photography is not time-consuming.

[0050] Since the hole cup 13 is small relative to the green 12, it may be difficult to capture the hole cup 13 together with the boundary of the green 12 in order to grasp its position within the green 12. In contrast, the objects 16-18 around the hole cup 13 and the green 12 are easy to capture in the image 15. For example, the objects 16-18 that are constantly present around the green 12, such as trees, buildings, and structures, have fixed positions. Therefore, by matching feature points 62 extracted from different videos 15 and using the constantly present feature points 62 as key points 64, it becomes possible to detect the position Td of the hole cup 13 based on its relative position with the key points 64.

[0051] For example, in the first time when the key points 64 are not set, the position of the hole cup 13 itself shown in the three-dimensional data 45 created from the primary video 15 may be plotted on the map data. From the second time onwards, by setting the key points 64 based on the objects 16 to 18, it becomes possible to detect the position Td with higher accuracy.

[0052] When the objects 16-18 are natural objects, such as trees, their appearance may change depending on the season. Therefore, the number and positions of the feature points 62 extracted from the objects 16-18 may change. For example, in the case of a deciduous tree, the number of feature points 62 may increase as the trunk is exposed due to falling leaves. Also, in the case of a fruit tree, the number of feature points 62 may increase as the tree bears fruit, and decrease as the fruit falls. Even in cases other than natural objects, the feature points 62 may change if, for example, the decoration of a building is changed, the arrangement of objects is changed, or a structure deteriorates.

[0053] In this regard, by setting the feature points 62 that are continuously present across multiple shots as the key points 64, the key points 64 are updated sequentially. If the key points 64 are set from multiple feature points 62 extracted from trees, the key points 64 are updated in accordance with seasonal changes in the trees. As a result, even if the feature points 62 of the objects 16 to 18 change over time, it is possible to detect the position Td of the hole cup 13 using the updated key points 64 as a reference point.

[0054] Even if an object 16-18 exists in the same position for a certain period of time, it may not be captured well depending on the shooting conditions of the image 15. Therefore, the key point 64 once set may be deleted if it does not match after a specified number of matchings (e.g., 3 to 5 times), rather than being deleted if it does not match after one matching. In this case, it is preferable to leave the feature points 62 corresponding to the images 15 for the specified number of times as the feature point data 42.

[0055] [Effects of this disclosure] According to the present disclosure, the following effects can be achieved. (1) The position of the hole cup 13, which is the detection target, can be detected based on the relative positions of the multiple key points 64 and the target feature point 62t. By performing such detection sequentially, the displacement of the hole cup 13 can be recorded.

[0056] (2) By generating the three-dimensional data 45 including the multiple key points 64 and the target feature points 62t, the relative positions of the multiple key points 64 and the target feature points 62t can be visually grasped.

[0057] (3) The plan view 46 generated from the three-dimensional data 45 makes it possible to more clearly illustrate the planar arrangement of the hole cups 13. (4) By overlaying the map data and the plan view 46 and plotting the position of the hole cup 13 on the map data, the geographic information of the hole cup 13, that is, the absolute position information, can be obtained.

[0058] (5) Since the key point data 44 is updated every time the key point setting unit 54 sets a new key point 64, the position of the hole cup 13 can be detected based on the key point 64 that corresponds to changes in the surrounding environment.

[0059] (6) The key point setting unit 54 deletes key points 64 that do not match after a specified number of matching attempts. This makes it possible to update the key points 64 to correspond to the surrounding environment while preventing the deletion of key points 64 that were temporarily not captured.

[0060] (7) The storage 36 stores feature point data 42 relating to feature points 62 used a prescribed number of times to set key points 64. Feature points 62 that are not used to set key points 64 can be deleted, so that an increase in the required capacity of the storage 36 can be suppressed.

[0061] (8) Since the feature points 62 are extracted from one or more objects 16 to 18, the position of the detection target can be detected based on the objects 16 to 18 that exist around the detection target.

[0062] (9) The key points 64 are updated when the feature points 62 disappear or increase due to a change in the objects 16 to 18. Therefore, even if the objects 16 to 18 are trees and the condition of the leaves, fruits, flowers, branches, or trunks of the trees changes with the seasons, the key points 64 can be updated in response to the changes.

[0063] (10) When the detection target is a hole cup 13 on a golf course 11, if the video 15 is shot so as to include the boundary line of the green 12 and the boundary line Gd of the green 12 is shown in the plan view 46, it is easy to grasp the detection position Td of the hole cup 13.

[0064] (11) Videos are a series of images that can be treated as a single file, making them easy to handle, such as when transferring and storing the data. (12) If three-dimensional data 45 that reproduces the environment including the detection target is created from the first captured video 15f (a series of primary images), the second and subsequent videos 15s can be captured so as to extract some feature points 62. Therefore, the capture time for the second and subsequent videos 15 can be shorter than the capture time for the first video 15f, thereby reducing the effort required for capture.

[0065] [Example of change] This embodiment can be modified as follows: This embodiment and the following modifications can be combined with each other to the extent that there is no technical contradiction.

[0066] The detection target is not limited to the hole cup 13. For example, the technology disclosed herein may be used to detect the displacement of equipment, tools, living things, humans, a specific person whose face has been recognized, or a vehicle. In this case, the movement history may be recorded by continuously photographing the same target, or different targets may be detected as long as they are pre-registered targets (i.e., targets of the same type). Alternatively, it is possible to detect the presence or absence of a detection target, or a change in the appearance position. Such a method of use is useful, for example, for the maintenance and management of goods or facilities, the ecological survey of wildlife, or confirmation of the entrance and exit of people.

[0067] The detection target does not have to appear in all images 15 (for example, a series of primary images). For example, a building, structure, or facility may be photographed as part of a regular inspection, with cracks or damaged areas in the building or structure as the detection target. In this case, when an abnormality such as a crack or damaged area occurs, the abnormality can be detected as the detection target. This method of use is useful for the maintenance and management of buildings, structures, or facilities. For example, when inspecting a structure such as a tunnel or road, a video 15 may be taken by a camera attached to a vehicle. In addition, if fallen objects on the road are the detection target, this is useful for removing the fallen objects.

[0068] The sequence of images 15 does not necessarily have to be taken manually, but may be taken, for example, by a camera or surveillance camera attached to a drone, an autonomous vehicle or an animal, or may be taken automatically, periodically, or when movement or sound is detected by a camera whose shooting direction can be changed.

[0069] A mark or a light source may be attached to the detection target so that the detection target is more likely to appear in the series of images 15. The pin 24 is an example of a mark for indicating the position of the hole cup 13 that is the detection target.

[0070] The primary image 15f, secondary video 15s, and tertiary video 15t do not have to be taken in this order (chronological order), and can be used for detection in the order in which the displacement is to be understood. In other words, "primary" does not mean that it was taken at the earliest time, but that it is basic data for restoring the three-dimensional structure of the environment including the detection target. In contrast to this "primary," "secondary" means that it is data for first setting the key point 64, and "tertiary" means that it is data for first updating the key point 64. Also, "fourth" and onwards mean that it is data for updating the key point 64, just like the tertiary.

[0071] The series of images (or video) 15 may be captured by a 3D camera. In this case, the information processing device 30 does not need to include the three-dimensional data generating unit 55. The three-dimensional data 45 may be overlaid with 3D map data. In this case, the creation of the plan view 46 may be omitted. Alternatively, if it is desired to detect the height position of the detection target instead of its planar position, the position information creation unit 56 may create an elevation view instead of the plan view 46. In addition, the position information creation unit 56 may create various other drawings to indicate the position of the detection target.

[0072] Aspects grasped based on the above embodiment and modified examples are listed below. [1] an image acquisition unit configured to acquire a series of images taken while varying a viewpoint; a feature point extractor configured to extract a plurality of feature points from the sequence of images; an object detection unit configured to detect an object feature point indicating a detection object from among the plurality of feature points; a key point setting unit configured to set a plurality of key points from the plurality of feature points; a position information creation unit configured to create position information of the detection target based on relative positions of the plurality of key points and the target feature points; Each of the key points is a feature point that indicates the same feature among the feature points extracted from each of the plurality of sets of the series of images. Information processing device [2] the images are two-dimensional images, and further comprising a three-dimensional data generator configured to generate three-dimensional data from the series of images; the three-dimensional data includes the plurality of key points and the target feature points; The position information creation unit is configured to create position information of the detection target from the three-dimensional data. The information processing device according to [1] above.

[0073] [3] The position information creation unit is configured to generate a plan view including the position of the detection target from the three-dimensional data. The information processing device according to [2] above.

[0074] [4]1 or more memory, The memory stores map data of an area in which the detection target is present, the position information creation unit is configured to plot the position of the detection target on the map data by superimposing the map data and the plan view. The information processing device according to [3] above.

[0075] [5]1 or more memory, The memory includes: Previous feature point data relating to the plurality of feature points; keypoint data relating to the plurality of keypoints; When the image acquisition unit acquires a new series of images, the feature point extraction unit extracts new feature points from the new series of images; the target detection unit detects a new target feature point indicating the detection target from among the new plurality of feature points; the key point setting unit is configured to match the new plurality of feature points with the previous plurality of feature points, set a feature point showing an identical feature as a new key point, and update the key point data. An information processing device according to any one of the above [1] to [4].

[0076] [6] The keypoint setting unit is configured to match the feature points extracted from each of the series of images a specified number of times, and delete the keypoints that do not match in the specified number of matches. An information processing device according to any one of the above [1] to [5].

[0077] [7]1 or more memory, The memory stores feature point data relating to the feature points for the specified number of times and key point data relating to the key points. The information processing device according to [6] above.

[0078] [8] The series of images is taken to include one or more objects present in the vicinity of the detection target; the feature point extraction unit is configured to extract the plurality of feature points from the one or more objects; An information processing device according to any one of the above [1] to [7].

[0079] [9] the one or more objects include one or more trees; the key points are set from the plurality of feature points extracted from the tree; The key points are updated according to seasonal changes of the trees. The information processing device according to [8] above.

[0080]

[10] The detection object is at least one of a hole cup of a golf course or a pin set in the hole cup, The series of images is taken so as to include the boundary line of the green on which the hole cup is located. An information processing device according to any one of the above [1] to [9].

[0081]

[11] The sequence of images is a video. An information processing device according to any one of the above [1] to

[10] .

[12] On one or more computers, acquiring a series of primary images taken from different viewpoints; Extracting a plurality of primary feature points from the series of primary images; acquiring a series of secondary images taken from different viewpoints including the detected object; Extracting a plurality of secondary feature points from the series of secondary images; detecting a secondary object feature point indicative of the detection object from among the plurality of secondary feature points; Setting a plurality of key points, each of which is one of the plurality of secondary feature points that exhibits the same feature as one of the plurality of primary feature points; generating position information of the detection target based on relative positions of the plurality of key points and the second target feature points; An information processing method, comprising:

[0082]

[13] One or more computers, acquiring a series of tertiary images captured while changing a viewpoint so as to include the detection target; Extracting a plurality of tertiary feature points from the series of tertiary images; detecting a tertiary object feature point indicative of the detection object from among the plurality of tertiary feature points; updating the plurality of key points, each of the updated key points being one of the plurality of tertiary feature points that exhibits the same features as one of the plurality of secondary feature points; generating position information of the detection target based on relative positions of the plurality of key points and the tertiary target feature point;

[0033] The information processing method according to

[12] above, comprising executing the steps:

[0083]

[14] the series of primary images is a primary moving image; said series of secondary images being a secondary moving image; The shooting time of the second video is shorter than the shooting time of the first video. The information processing method according to

[12] or

[13] above.

[0084]

[15] On one or more computers, acquiring a series of primary images taken from different viewpoints; Extracting a plurality of primary feature points from the series of primary images; acquiring a series of secondary images taken from different viewpoints including the detected object; Extracting a plurality of secondary feature points from the series of secondary images; detecting a secondary object feature point indicative of the detection object from among the plurality of secondary feature points; Setting a plurality of key points, each of which is one of the plurality of secondary feature points that exhibits the same feature as one of the plurality of primary feature points; generating position information of the detection target based on relative positions of the plurality of key points and the second target feature points; An information processing program for executing the above. [Explanation of symbols]

[0085] 11...golf course, 12...green, 13...hole cup, 14...management facility, 15...series of images (video), 15f...series of primary images, 15s...series of secondary images, 15t...series of tertiary images, 16-18...reference objects, 20...photographer, 21...photographing device, 22...user terminal, 23...network, 24...pin, 30...information processing device, 31...processor, 32...memory, 33...communication IF, 34...input device, 35...output device, 36...storage, 37...communication bus, 38...application, 41...image Image data, 42...feature point data, 43...target data, 44...keypoint data, 45...3D data, 46...positional information data, 51...image acquisition unit, 52...feature point extraction unit, 53...target detection unit, 54...keypoint setting unit, 55...3D data generation unit, 56...positional information creation unit, 62...feature point, 62f...primary feature point, 62s...secondary feature point, 62t...target feature point, 62tf...primary target feature point, 62tf...target feature point, 62ts...secondary target feature point, 64...key point, Gd...boundary line, Td...detection position.

Claims

1. an image capture unit configured to capture a sequence of images taken while varying the viewpoint; a feature point extractor configured to extract a plurality of feature points from the sequence of images; an object detection unit configured to detect an object feature point indicating a detection object from among the plurality of feature points; a key point setting unit configured to set a plurality of key points from the plurality of feature points; a position information generating unit configured to detect a position of the detection target based on relative positions of the plurality of key points and the target feature point; Each of the key points is a feature point that indicates the same feature among the feature points extracted from each of the plurality of sets of the series of images. Information processing device.

2. the images are two-dimensional images, and further comprising a three-dimensional data generator configured to generate three-dimensional data from the series of images; the three-dimensional data includes the plurality of key points and the target feature points; The position information creation unit is configured to create a diagram showing the position of the detection target from the three-dimensional data. The information processing device according to claim 1 .

3. The position information creation unit is configured to generate a plan view including the position of the detection target from the three-dimensional data. The information processing device according to claim 2 .

4. One or more memories, The memory stores map data of an area in which the detection target is present, the position information creation unit is configured to generate a diagram in which the position of the detection target is plotted on the map data by superimposing the map data and the plan view. The information processing device according to claim 3 .

5. One or more memories, The memory includes: Previous feature point data relating to the plurality of feature points; keypoint data relating to the plurality of keypoints; When the image acquisition unit acquires a new series of images, the feature point extraction unit extracts new feature points from the new series of images; the target detection unit detects a new target feature point indicating the detection target from among the new plurality of feature points; the key point setting unit is configured to match the new plurality of feature points with the previous plurality of feature points, set a feature point showing the same feature as the previous feature point as a new key point, and update the key point data. The information processing device according to claim 1 .

6. The key point setting unit is configured to perform matching of the feature points extracted from each of the series of images a specified number of times, and to delete the key points that do not match in the specified number of times of matching. The information processing device according to claim 1 .

7. One or more memories, The memory stores feature point data relating to the feature points for the specified number of times and key point data relating to the key points. The information processing device according to claim 6.

8. The series of images is taken so as to include one or more objects present in the vicinity of the detection target, the feature point extraction unit is configured to extract the plurality of feature points from the one or more objects; The information processing device according to any one of claims 1 to 7.

9. the one or more objects include one or more trees; the key points are set from the plurality of feature points extracted from the tree; The key points are updated according to seasonal changes of the trees. The information processing device according to claim 8.

10. the detection object is at least one of a hole cup of a golf course or a pin set in the hole cup, The series of images is taken so as to include the boundary line of the green on which the hole cup is located. The information processing device according to claim 1 .

11. The sequence of images is a video. The information processing device according to claim 1 .

12. On one or more computers, acquiring a series of primary images taken while varying the viewpoint; extracting a plurality of primary feature points from the series of primary images; acquiring a series of secondary images taken while varying the viewpoint to include the detected object; extracting a plurality of secondary feature points from the series of secondary images; detecting a secondary object feature point indicative of the detection object from among the plurality of secondary feature points; Setting a plurality of key points, each of which is one of the plurality of secondary feature points that exhibits the same feature as one of the plurality of primary feature points; and Detecting a position of the detection target based on relative positions of the plurality of key points and the second target feature point; An information processing method, comprising:

13. The one or more computers, acquiring a series of tertiary images captured while changing a viewpoint so as to include the detection target; extracting a plurality of tertiary feature points from the series of tertiary images; detecting a tertiary object feature point indicative of the detection object from among the plurality of tertiary feature points; updating the plurality of key points, each of the updated key points being one of the plurality of tertiary feature points that exhibits the same features as one of the plurality of secondary feature points; Detecting a position of the detection target based on relative positions of the plurality of key points and the third target feature point; [0033] The information processing method according to claim 12.

14. the sequence of primary images is a primary motion picture; the series of secondary images is a secondary moving image; A shooting time of the second video is shorter than a shooting time of the first video. The information processing method according to claim 12.

15. On one or more computers, acquiring a series of primary images taken while varying the viewpoint; extracting a plurality of primary feature points from the series of primary images; acquiring a series of secondary images taken while varying the viewpoint to include the detected object; extracting a plurality of secondary feature points from the series of secondary images; detecting a secondary object feature point indicative of the detection object from among the plurality of secondary feature points; Setting a plurality of key points, each of which is one of the plurality of secondary feature points that exhibits the same feature as one of the plurality of primary feature points; and Detecting a position of the detection target based on relative positions of the plurality of key points and the second target feature point; An information processing program for executing the above.

Images