Information processing device, information processing method, and information processing program

Abstract

To provide an information processing device capable of acquiring a position of a mobile body, an information processing method, and an information processing program.SOLUTION: An information processing device comprises: a storage unit which stores map information on a map; an acceptance unit which accepts image information on an image captured by an imaging unit arranged in the mobile body; a first acquisition unit which acquires a ratio between a first distance on the map to a first edge part of the image on a further side from the mobile body from the center position of the image and a second distance on the map to a second edge part of the image on a closer side to the mobile body from the center position of the image on the basis of the image information accepted by the acceptance unit; and a second acquisition unit which acquires a position of the mobile body on the basis of the ratio acquired by the first acquisition unit, an imaging angle when the ground is captured by the imaging unit, and the height of the imaging unit.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present disclosure relates to an information processing device, an information processing method, and an information processing program. [Background technology]

[0002] Conventionally, examples of mobile bodies include aircraft and vehicles, etc. As described in Patent Document 1, such mobile bodies (aircraft in this patent document) acquire their own positions by receiving signals transmitted from GPS satellites, for example. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2004-157887 Summary of the Invention [Problem to be solved by the invention]

[0004] In recent years, flying objects (mobile objects) such as drones have appeared, and cameras mounted on these drones are sometimes used to photograph ground scenery, etc. Furthermore, such drones are not limited to photographing scenery, but are also used for a variety of purposes, including exterior inspections of facilities. As drone applications continue to expand, there are cases where it is desirable to obtain the drone's flight position, but equipping a drone with a GPS receiver poses problems such as increased weight and power consumption.In addition, there are also cases where it is desirable to obtain the position of other moving objects other than drones, such as vehicles traveling indoors, without using GPS.

[0005] The present disclosure provides an information processing device, an information processing method, and an information processing program capable of acquiring the position of a moving object. [Means for solving the problem]

[0006] An information processing device of one embodiment includes a memory unit that stores map information related to a map, a reception unit that receives image information related to an image captured by an imaging unit disposed on a moving body, a first acquisition unit that acquires, based on the image information received by the reception unit, a ratio between a first distance on the map from the center position of the image to a first edge of the image farther from the moving body and a second distance on the map from the center position of the image to a second edge of the image closer to the moving body, and a second acquisition unit that acquires the position of the moving body based on the ratio acquired by the first acquisition unit, the imaging angle when capturing an image of the ground by the imaging unit, and the height of the imaging unit. [Effects of the Invention]

[0007] According to one aspect, based on a reception unit that receives image information and image information regarding an image captured by an imaging unit disposed on the moving body and received by the reception unit, a ratio is obtained between a first distance on a map from the center position of the image to a first edge of the image farther from the moving body and a second distance on the map from the center position of the image to a second edge of the image closer to the moving body, and the position of the moving body can be obtained based on the ratio, the imaging angle when capturing an image of the ground by the imaging unit, and the height of the imaging unit. [Brief explanation of the drawings]

[0008] [Figure 1] FIG. 1 is a diagram illustrating an information processing device according to an embodiment. [Figure 2] FIG. 1 is a block diagram illustrating an information processing device according to an embodiment. [Figure 3] FIG. 2 is a first diagram showing an example of the relationship between a map image and a camera image. [Figure 4] FIG. 10 is a second diagram showing an example of the relationship between a map image and a camera image. [Figure 5] FIG. 10 is a diagram illustrating an example of the relationship between the imaging angle (pitch) and the ratio. [Figure 6] 10A and 10B show the correlation between the imaging angle (pitch) and the ratio (division distance ratio) relative to the imaging angle of view, where (A) shows the case where the imaging angle of view is 30°, (B) shows the case where the imaging angle is 45°, and (C) shows the case where the imaging angle is 60°. [Figure 7] 1 is a flowchart illustrating an information processing method according to an embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0009] An embodiment will be described below.

[0010] [Overview of information processing device 1] First, an overview of an information processing device 1 according to an embodiment will be described. FIG. 1 is a diagram illustrating an information processing device 1 according to an embodiment.

[0011] The information processing device 1 may be configured as, for example, a position estimation device that estimates the position of a moving object on which the imaging unit 21 is disposed. The information processing device 1 may also be configured as, for example, a position tracking device that tracks the moving position of the moving object. Here, tracking the moving position of the moving object may refer to tracking the position of the moving object, for example. The information processing device 1 may be, for example, a computer such as a server, a desktop, a laptop, a tablet, or a smartphone, or may be any of various computers mounted on a mobile object.

[0012] The moving body may be, for example, various aircraft such as the drone 100, various vehicles such as an unmanned transport vehicle, an autonomous vehicle, and a vehicle operated by a driver, as well as a mobile robot, etc. The moving body may be remotely controlled, may be operated by a person on board, or may be capable of moving automatically.

[0013] The moving object includes an imaging unit 21. The imaging unit 21 may be, for example, a camera capable of capturing still images or moving images and generating image information. Here, when capturing still images, the imaging unit 21 may capture the still images at predetermined intervals. As an example of this case, the imaging unit 21 may capture the still images at predetermined time intervals. For example, the imaging unit 21 can capture images while the moving object is moving. As an example of this case, if the moving object is a flying drone 100, the imaging unit 21 can capture images of the ground.

[0014] When the imaging unit 21 generates image information, it can transmit the image information to the information processing device 1. In this case, the imaging unit 21 may transmit the image information while the moving object is moving, or may transmit the image information when the moving object has finished moving (for example, when it has arrived at a destination, etc.). Furthermore, the imaging unit 21 may, for example, record the image information in a recording medium such as a memory. For example, when the recording medium is inserted into an interface or the like (not shown), the information processing device 1 may acquire the image information recorded in the recording medium.

[0015] For example, when the information processing device 1 receives image information, it acquires, based on the image information and map information corresponding to the position of the scenery recorded in the image information, a ratio (division distance ratio) between a first distance on the map from the center position of the image to a first edge of the image farther from the moving body and a second distance on the map from the center position of the image to a second edge of the image closer to the moving body. As a general example, the height direction (vertical direction) of an image based on image information is considered to be the distance direction between the side farther from the moving object (upward in the image) and the side closer to the moving object (downward in the image). Therefore, the information processing device 1 acquires a ratio (division distance ratio) between the distance in the actual scenery from the center position of the image based on image information to the midpoint of the edge in the upward direction of the image (first distance) and the distance in the actual scenery from the center position of the image to the midpoint of the edge in the downward direction of the image (second distance).

[0016] Furthermore, the information processing device 1 acquires, for example, the imaging direction of the imaging unit 21 (for example, the direction of the center position imaged by the imaging unit 21), the angle (imaging angle) formed between the imaging unit 21 and the direction perpendicular to and downward from the imaging unit 21. Furthermore, the information processing device 1 acquires, for example, the height of the imaging unit 21 from a reference plane. Here, the reference plane may be the ground, for example, in the case where the mobile object is a drone 100 capable of flying. Furthermore, the reference plane may be, for example, the road surface on which the mobile object travels, in the case where the mobile object is a vehicle capable of running.

[0017] The information processing device 1 acquires the position of the moving body based on the above-mentioned ratio (division distance ratio), the imaging angle, and the height of the imaging unit 21. That is, the information processing device 1 acquires the position of the moving body by performing calculations using, for example, the ratio (division distance ratio), the imaging angle, and the height of the imaging unit 21. The position of the moving body may be, for example, coordinates based on longitude and latitude, or may be a relative position (relative coordinates) with respect to a reference position (reference coordinates). Note that the position of the moving body is not limited to the above-mentioned example, and various other means may be used as long as the position of the moving body can be identified.

[0018] [Details of information processing device 1] Next, the information processing device 1 according to an embodiment will be described in detail. FIG. 2 is a block diagram illustrating an information processing device 1 according to an embodiment.

[0019] The information processing device 1 includes, for example, a communication unit 22, a storage unit 23, a display unit 24, an imaging unit 21, and a control unit 11. The communication unit 22, the storage unit 23, and the display unit 24 may be an embodiment of an output unit. The control unit 11 includes, for example, a reception unit 12, a first acquisition unit 13, a second acquisition unit 14, and an output control unit 15. The control unit 11 may be configured, for example, by an arithmetic processing unit of the information processing device 1. The control unit 11 (for example, an arithmetic processing unit) may realize the functions of each unit (for example, the reception unit 12, the first acquisition unit 13, the second acquisition unit 14, and the output control unit 15) by, for example, appropriately reading and executing various programs stored in the storage unit 23.

[0020] The communication unit 22 is capable of transmitting and receiving various information to and from, for example, a device external to the information processing device 1 (external device).

[0021] The storage unit 23 may store, for example, various types of information and programs. Examples of the storage unit 23 may include a memory, a solid state drive, and a hard disk drive. The storage unit 23 stores map information related to a map. The map may be, for example, a map on which target objects (target positions) serving as landmarks are recorded. As an example, the map may be a map on which landmarks such as roads and buildings are recorded. Here, the target objects (target positions) may be, for example, objects (positions) that do not move and objects (positions) whose shapes change relatively little over time. The map information may also include distance information related to distance and position information related to position. In this case, the position information may be, for example, coordinate information related to longitude and latitude, and coordinate information related to relative position coordinates based on predetermined coordinates (reference coordinates).

[0022] The display unit 24 is capable of displaying, for example, various characters, symbols, images, and the like.

[0023] As described above, the imaging unit 21 may be a camera or the like disposed on a moving body. The imaging unit 21 may be disposed on the information processing device 1, for example, in other words, the information processing device 1 may be disposed on a moving body. Alternatively, the imaging unit 21 may be disposed outside the information processing device 1, for example, in other words, the information processing device 1 and the moving body may be configured as separate entities. The imaging unit 21 captures an image of a subject and generates image information.

[0024] The reception unit 12 receives image information relating to an image captured by an imaging unit 21 disposed in a moving object. The reception unit 12 may receive the image information, for example, via the communication unit 22. Alternatively, when a recording medium on which the image information is recorded is inserted into an interface or the like (not shown) of the information processing device 1, the reception unit 12 may receive the image information from the recording medium.

[0025] The first acquisition unit 13 acquires, based on the image information received by the reception unit 12, the ratio (division distance ratio) between a first distance on the map from the center position of the image to a first edge of the image farther from the moving body and a second distance on the map from the center position of the image to a second edge of the image closer to the moving body. As a general example, an image based on image information can be considered to be rectangular. Also, as a general example, the height direction (vertical direction) of the image can be considered to be the distance direction between the side farther from the moving object (upward in the image) and the side closer to the moving object (downward in the image). Therefore, the first acquisition unit 13 acquires a ratio (division distance ratio) between the distance from the center position of the image to the edge in the upward direction (first distance) and the distance from the center position of the image to the edge in the downward direction (second distance). In this case, the first distance may correspond to, for example, the distance between positions C and T illustrated in FIG. 3. Furthermore, the second distance may correspond to, for example, the distance between positions C and B illustrated in FIG. 3.

[0026] In this case, the first acquisition unit 13 may, for example, refer to map information stored in the storage unit 23 and acquire a ratio (division distance ratio) based on the actual distance (distance on the map) of the scenery recorded in the image (scenery captured by the imaging unit 21). That is, the first acquisition unit 13 may, for example, acquire a ratio (division distance ratio) between a first distance as the distance in the actual scenery (distance on the map) from the center position of the image to a first edge of the image farther from the moving object, and a second distance as the distance in the actual scenery (distance on the map) from the center position of the image to a second edge of the image closer to the moving object, based on the image information and map information corresponding to the position of the scenery recorded in the image information. In this case, the first distance may correspond to, for example, the distance d_TC illustrated in FIG. 4. The second distance may correspond to, for example, the distance d_CB illustrated in FIG. 4.

[0027] Here, a configuration will be described in which the imaging angle and the height of the imaging unit 21 are known and the imaging angle and the height of the imaging unit 21 are input to the information processing device 1 by an input unit (not shown) arranged in the information processing device 1 to acquire the ratio (division distance ratio). Note that the input unit may be, for example, a keyboard and a mouse.

[0028] 3 is a first diagram showing an example of the relationship between a map image and a camera image. Here, an image captured by the imaging unit 21 is referred to as a "camera image." FIG. 4 is a second diagram showing an example of the relationship between the map image and the camera image.

[0029] As shown in Figure 3, in the camera image, the upper left (far point on the left side) is LT, the lower left (near point on the left side) is LB, the upper right (far point on the right side) is RT, the lower right (near point on the right side) is RB, the midpoint is C, the midpoint of the left edge (LT-LB) is LM, the midpoint of the right edge (RT-RB) is RM, the midpoint of the upper edge (LT-RT) is MT, and the midpoint of the lower edge (LB-RB) is MB. The camera image can be considered as a plane perpendicular to the line segment from the camera position P to the midpoint C. In this case, the map image can be considered as a plane that intersects the camera image at an angle. Therefore, if we consider that the map image intersects with the camera image at the center C and midpoints LM and RM, the positions on the map image corresponding to the upper left LT, upper right RT, and midpoint MT recorded in the camera image are on an extension line from the camera position P, and the positions on the map image corresponding to the lower left LB, lower right RB, and midpoint MB are on the line segment from the camera position P to the camera image, and can be considered to be the lower left LBs, lower right RBs, and midpoint B. In the example shown in FIG. 3, the yaw and pitch of the imaging unit 21 are set to 0°, and only the pitch is taken into consideration.

[0030] As a specific example (first example) of this case, when the pitch (imaging angle) of the imaging unit 21 and the height of the imaging unit 21 are known, the first acquisition unit 13 can calculate the distance on the map based on the resolution and angle of view of the imaging unit 21. The resolution of the imaging unit 21 may be, for example, the number of pixels in the vertical and horizontal directions of the camera image. The angle of view of the imaging unit 21 is determined, for example, by the planar size of the imaging element and the focal length of the lens, and may be a known value.

[0031] As a first example, if the number of pixels (resolution) in the vertical direction (height direction) of the camera image is 2160, the number of pixels (resolution) in the horizontal direction (width direction) is 3840, and the horizontal angle of view of the imaging unit 21 is 80°, the vertical angle of view of the imaging unit 21 is 45° (=(80° / 3840)×2160). For example, if the imaging angle (pitch) is input as 30° via an input unit (not shown), the angle (ANG_TC) (see FIG. 4) formed by the midpoint C of the camera image, the position P of the imaging unit 21, and the position MT of the camera image (the position T on the map corresponding to the position MT) is determined to be 22.5° (=the vertical angle of view 45° / 2). Similarly, the angle (ANG_CB) (see FIG. 4) formed by the midpoint C of the camera image, the position P of the imaging unit 21, and the position MB of the camera image (the position B on the map corresponding to the position MB) is determined to be 22.5° (=the vertical angle of view 45° / 2). Similarly, the angle (ANG_MR) between the midpoint C of the camera image, the position P of the imaging unit 21, and the position RM of the camera image is determined to be 40° (= horizontal angle of view 80° / 2). Also, the angle (ANG_BP0) between the position MB of the camera image (position B on the map corresponding to position MB), the position P of the imaging unit 21, and the position P0 directly below the imaging unit 21 is determined to be 7.5° (= 30° - 22.5°).

[0032] Furthermore, for example, if the height of the imaging unit 21 is input as 100 m via an input unit (not shown), the distance (d_BP0) from position B on the map to position P0 directly below the imaging unit 21 (see FIG. 4) is calculated as 13.16525 m (100 × tan 22.5°). Similarly, the distance (d_CB) from midpoint C to position B on the map (see FIG. 4) is calculated as 44.56978 m (= 100 × tan 30° - 13.16525 m). Similarly, the distance (d_TC) from midpoint C to position T on the map (see FIG. 4) is calculated as 72.58751 m (= 100 × tan (30° - 22.5°) - 13.16525 m - 44.56978 m). The first acquisition unit 13 can acquire the ratio (divided distance ratio) dr (=d_CB / d_TC) based on the distance (d_CB) and the distance (d_TC).

[0033] Furthermore, when the imaging angle (pitch) is known, the first acquisition unit 13 can calculate the ratio (division ratio) based on the imaging angle of view. A specific example (second example) of this case will be described below. FIG. 5 is a diagram showing an example of the relationship between the imaging angle (pitch) and the ratio.

[0034] The ratio on the map (for example, the division distance ratio dr (=d_CB / d_TC) between the distance (d_TC) from the center C to the position T and the distance (d_CB) from the center C to the position B) correlates with the imaging angle (pitch). As illustrated in FIG. 5, the ratio (division distance ratio) changes depending on the imaging angle (pitch).

[0035] In this case, when the imaging angle (pitch) is input, the first acquisition unit 13 can calculate the ratio (division distance ratio) based on the angle of view of the imaging unit 21. Here, assuming that the imaging angle (pitch) is 45° and the camera height is 1 m, the angle (ANG_TC, ANG_CB) formed by the two angles is calculated as in the first example. In addition, the angle (ANG_BP0) is calculated as 22.5° (= 45° - 22.5°).

[0036] Furthermore, the distance (d_BP0) is calculated as 0.414214m (1 x tan 22.5°). Similarly, the distance (d_CB) is calculated as 0.585786m (= 1 x tan 45° - 0.414214m). Similarly, the distance (d_TC) is calculated as 0.414214m (= 1 x tan (45° - 22.5°) - 0.414214m - 0.585786m). Therefore, the first acquisition unit 13 can obtain the ratio (divided distance ratio) (dr) as 0.414214 (=0.585786 / 0.414214).

[0037] There is a correlation between the imaging angle (pitch) and the ratio (division distance ratio) relative to the imaging angle of view, and this correlation can be approximated by, for example, a cubic to quintic equation. As a specific example of the correlation, approximation by a cubic equation or the like will be described with reference to FIG.

[0038] Fig. 6 is a diagram showing the correlation between the imaging angle (pitch) and the ratio (division distance ratio) relative to the imaging angle of view, where Fig. 6(A) shows the case where the imaging angle of view is 30°, Fig. 6(B) shows the case where the imaging angle is 45°, and Fig. 6(C) shows the case where the imaging angle is 60°.

[0039] The correlation between the imaging angle (pitch) and the ratio (division distance ratio) when the imaging angle shown in FIG. 6(A) is 30° is, for example, y=-16.088x 3 -31.111x 2 It can be approximated by a cubic equation of -30.055x+75.456, and the coefficient of that equation (division distance coefficient) can be obtained. The correlation between the imaging angle (pitch) and the ratio (division distance ratio) when the imaging angle is 45° shown in FIG. 6(B) is, for example, y=-1.3813x 3 -24.054x 2 It can be approximated by a cubic equation of -43.874x+67.033, and the coefficient of that equation (division distance coefficient) can be obtained. The correlation between the imaging angle (pitch) and the ratio (division distance ratio) when the imaging angle is 60° shown in FIG. 6(C) is, for example, y=-6.4077x 3 -16.132x 2 It can be approximated by a cubic equation of -51.185x+59.292, and the coefficient of that equation (division distance coefficient) can be obtained.

[0040] The second acquisition unit 14 acquires the height of the moving object by calculation using the ratio (division distance ratio), the imaging angle, and the height of the imaging unit 21. Because the second acquisition unit 14 can acquire the ratio (division distance ratio) based on the angle of view of the imaging unit 21, it can calculate the imaging angle (pitch) and the height of the imaging unit 21 (moving object) by calculating the distance (d_CB) and the distance (d_TC).

[0041] In the second example described above, once one of the distances (d_BP0), (d_CB), and (d_TC) is determined, the second acquisition unit 14 can calculate the height of the imaging unit 21 (moving object). Here, for example, if the distance (d_CBm) is input as 50 m via an input unit (not shown) as the actual distance of the distance (d_CB) corresponding to the second example, the height h of the imaging unit 21 is calculated as 85.355 m (= 50 / 0.585786).

[0042] Furthermore, the second acquisition unit 14 can calculate the imaging angle and height by utilizing the correlation between the imaging angle (pitch) and the ratio (division distance ratio) relative to the imaging angle of view described above. That is, when the ratio (division distance ratio) is input based on the actual distance (d_TC) between the center C of the image and position T and the actual distance (d_CB) between the center C of the image and position B based on the image information, the second acquisition unit 14 can acquire the imaging angle (pitch) corresponding to the imaging angle of view based on the correlation described above. Furthermore, since the actual distance (d_CB) and the actual distance (d_TC) are known, the second acquisition unit 14 can calculate the height h of the imaging unit 21 in the same way as when the second example described above is used.

[0043] In this case, the second acquisition unit 14 can calculate the distance (d_BP0) using the calculation formula (h × tan(ANG_BP0)) based on the height h of the imaging unit 21 and the angle (ANG_BP0) formed between position B, the imaging unit 21, and position BP0. The angle (ANG_BP0) can be calculated by subtracting the angle (ANG_CB) formed between position C, the imaging unit 21, and position B from the imaging angle (pitch). The angle (ANG_CB) is half the angle of view (FOV) of the imaging unit 21 in the vertical direction. The angle of view (FOV) can be calculated based on the angle of view (ANG) of the imaging unit 21 in the horizontal direction and the number of pixels in each of the vertical and horizontal directions of the image sensor.

[0044] That is, the second acquisition unit 14 acquires the height from the ground to the imaging unit 21 based on the ratio acquired by the first acquisition unit 13 and the imaging angle when the ground is imaged by the imaging unit 21. The second acquisition unit 14 may also acquire the height from the ground to the imaging unit 21 based on the ratio acquired by the first acquisition unit 13 and the correlation between the imaging angle of the imaging unit 21 with respect to the imaging angle of view of the imaging unit 21 and the ratio. Furthermore, the second acquisition unit 14 may acquire the height from the ground to the imaging unit 21 based on the ratio of the distance on the map as a ratio and the imaging angle. In this case, the second acquisition unit 14 may acquire a third distance (d_BP0) from the second edge (corresponding position B on the map) to a position P0 directly below the moving body (imaging unit 21) based on the angle (ANG_CB) formed between the center position C of the image based on the image information, the position P of the imaging unit 21, and the midpoint MB of the second edge of the camera image (corresponding position B on the map), the imaging angle (pitch), and the height from the ground to the imaging unit 21.

[0045] When acquiring the height of the imaging unit 21 described above, the second acquisition unit 14 may further acquire the position of the imaging unit 21. Here, the second acquisition unit 14 may use the coordinates of position T, center C, and position B, and the distances (d_CB), (d_TC), and (d_BP0) to acquire the coordinates of position P0 located on a line passing through these points. As an example, the second acquisition unit 14 may use the third distance (d_BP0) to acquire the position of the moving object on the map (the position (P0) directly below the imaging unit 21).

[0046] Furthermore, the second acquisition unit 14 acquires the position of the moving object based on the ratio (division distance ratio) acquired by the first acquisition unit 13, the imaging angle when the imaging unit 21 images the ground, and the height of the imaging unit 21. The imaging angle may be, for example, the imaging angle when the imaging unit 21 images a landscape. That is, the imaging angle may be, for example, the angle formed between the imaging direction of the imaging unit 21 (for example, the direction of the center position imaged by the imaging unit 21), the imaging unit 21, and the direction perpendicular to and downward from the imaging unit 21.

[0047] The second obtaining unit 14 may use projective transformation to obtain coordinates on the map from the position of the image. That is, the second acquisition unit 14 may estimate the position (BP0) of the moving body on the map by projecting multiple positions recorded in an image based on the image information received by the reception unit 12 onto a position on a map based on the map information. First, in this case, the second acquisition unit 14 generates a projective transformation matrix by associating a plurality of positions recorded in the camera image with a plurality of positions on the map corresponding to the plurality of positions. The plurality of positions on the camera image may be represented by x and y coordinates on the camera image. The multiple positions on the map may be x, y coordinates on the map. The x, y coordinates on the map may be longitude and latitude position coordinates. For example, when associating a position in a camera image with a position on a map, the second acquisition unit 14 may generate a projective transformation matrix by using at least four positions.

[0048] The projective transformation matrix is ​​expressed by the following formula (1): where (x, y) may be, for example, the x and y coordinates of the camera image, and (x', y') may be, for example, the x and y coordinates of the map (e.g., longitude and latitude).

[0049]

number

[0050] The second acquisition unit 14 may use, for example, a projective transformation matrix to project the positions of features such as buildings and intersections from the scenery recorded in the camera image onto a map and acquire the coordinates of the features. The second acquisition unit may acquire the coordinates of position T, center C, and position B based on the coordinates of the features on the map, and may acquire the height of the imaging unit 21 and the position BP0 of the imaging unit 21 on the map based on the division distance ratio acquired as described above.

[0051] Since it is considered that the scenery recorded in the camera image also moves when the moving object moves, the second acquisition unit 14 can acquire the position on the map according to the movement of the moving object by using a projective transformation matrix. That is, the second acquisition unit 14 identifies at least four points in the camera image and acquires multiple corresponding positions on the map based on the projective transformation matrix. Therefore, by tracking the movement of at least four points in the camera image, the movement of the position on the map can be tracked. In other words, the second acquisition unit 14 can track the position of the moving object according to the change in the movement of the position recorded in the image information received by the receiving unit 12 as the moving object moves.

[0052] In this case, the second acquisition unit 14 first identifies, for example, at least four positions in the camera image that will serve as measurement points. Next, the second acquisition unit 14, for example, crops the camera image to include each of the identified positions. By generating multiple cropped images corresponding to each position, the second acquisition unit 14 can reduce the calculation processing burden compared to tracking at least four positions using the entire camera image. The second acquisition unit 14 tracks the movement of each position using the multiple cropped images. Note that if a target position falls outside the range of the cropped image due to movement of the moving object, the second acquisition unit 14 may move the crop range to include the position, or may identify a new position and generate a cropped image that includes the position.

[0053] The second acquisition unit 14 may use various tracking algorithms to track the position, for example. The tracking algorithm may be, for example, an algorithm that tracks the movement of an object by searching for a similar part of the object in an image. As a specific example, the tracking algorithm may be a CV tracker such as KCF, MedianFlow, or CSRT, or may be optical flow.

[0054] Here, the second acquisition unit 14 may update the projection transformation matrix at the position to which the moving body (imaging unit 21) has moved, for example, because when the moving body (imaging unit 21) moves, the tracking point (identified position) on the camera image corresponding to the position on the map moves. Furthermore, for example, when the moving body (imaging unit 21) moves, the identified position also moves, and the tracking of the position may deviate. In this case, the second acquisition unit 14 may correct the movement amount of a position that deviates relatively greatly from the average movement amount based on the average movement amount around the median of all of the multiple positions identified by at least four points. Furthermore, for example, if the multiple positions are significantly shifted relative to each other during the process of tracking the multiple positions, the second acquisition unit 14 may identify at least four points in the camera image and four corresponding points on the map each time, and update the projection transformation matrix.

[0055] The output control unit 15 may control the output unit to output at least one selected from the group of the position of the moving object, the height of the moving object, and the tracking result of the position of the moving object acquired by the second acquisition unit 14. Here, the output unit may be, for example, the communication unit 22, the storage unit 23, the display unit 24, etc. That is, the output control unit 15 may control the communication unit 22 to transmit at least one piece of information selected from the group of the position of the moving object, the height of the moving object, and the tracking result of the position of the moving object acquired by the second acquisition unit 14 to an external device (external device) of the information processing device 1. Here, the external device may be, for example, a server, a user terminal, etc. The user terminal may be, for example, a terminal used by a user of the information processing device 1, and specific examples thereof include a desktop, a laptop, a tablet, and a smartphone. The output control unit 15 may control the memory unit 23 to store at least one piece of information selected from the group of the position of the moving body, the height of the moving body, and the tracking results of the position of the moving body acquired by the second acquisition unit 14. The output control unit 15 may control the display unit 24 to display at least one selected from the group of the position of the moving body, the height of the moving body, and the tracking result of the position of the moving body acquired by the second acquisition unit 14.

[0056] The information processing device 1 described above can estimate the position of a flying drone 100 as a moving object, for example. In this case, the information processing device 1 can perform position estimation by patterning each of the following cases: when the position of the drone 100 is fixed and the imaging unit 21 is rotated; when the center of the imaging unit 21 is fixed and the drone 100 is moved; and when the drone 100 is simply moved linearly. Furthermore, the information processing device 1 can estimate the position of a traveling vehicle, for example, as a moving body. In this case, the imaging unit 21 may be an in-vehicle camera mounted on the vehicle. In this case, the information processing device 1 can perform position estimation as if the imaging unit 21 is simply moving, for example, by fixing the height and imaging angle (pitch) of the imaging unit 21. Furthermore, the information processing device 1 can estimate the position of a traveling robot or unmanned vehicle as a moving body, for example. In this case, the information processing device 1 can estimate the position assuming that there are frequent rotations (yaw changes) but little change in the height and pitch / roll of the imaging unit 21.

[0057] [Information processing method] Next, an information processing method according to an embodiment will be described. FIG. 7 is a flowchart illustrating an information processing method according to an embodiment.

[0058] In step ST101, the receiving unit 12 receives image information relating to an image captured by the imaging unit 21 disposed in the moving object.

[0059] In step ST102, the first acquisition unit 13 acquires, based on the image information received in step ST101, the ratio (division distance ratio) between the first distance (d_TC) on the map from the center position of the image to the first edge of the image farther from the moving body and the second distance (d_CB) on the map from the center position of the image to the second edge of the image closer to the moving body.

[0060] For example, when image information is received by the receiving unit 12, the first acquiring unit 13 identifies at least four positions in an image (camera image) based on the image information. The first acquiring unit 13 projects the identified at least four positions onto a map using a projection transformation matrix. In this case, the first acquisition unit 13 may calculate the distance and ratio (division distance ratio) based on, for example, the coordinates of each of the position T, center C, and position B on the map, the coordinates (e.g., longitude and movement) of the four corners of the map (positions on the map corresponding to the upper left LT, lower left LB, upper right RL, and lower right RB of the camera image), and the resolution of the camera image.

[0061] In step ST103, the second acquisition unit 14 acquires the height from the ground to the imaging unit 21 based on the ratio acquired in step ST102 and the imaging angle when the ground is imaged by the imaging unit 21. The second acquisition unit 14 may also acquire the height from the ground to the imaging unit 21 based on the ratio acquired in step ST102 and the correlation between the imaging angle of the imaging unit 21 with respect to the imaging angle of view of the imaging unit 21 and the ratio. Furthermore, the second acquisition unit 14 may acquire the height from the ground to the imaging unit 21 based on the ratio of the distance on the map as a ratio and the imaging angle. In this case, the second acquisition unit 14 may acquire a third distance (d_BP0) from the second edge (corresponding position B on the map) to a position P0 directly below the moving body (imaging unit 21) based on the angle (ANG_CB) formed between the center position C of the image based on the image information, the position P of the imaging unit 21, and the midpoint MB of the second edge of the camera image (corresponding position B on the map), the imaging angle (pitch), and the height from the ground to the imaging unit 21.

[0062] The second acquisition unit 14 estimates the position of the imaging unit 21 (moving object) by using, for example, the projected map. In this case, the second acquisition unit 14 may calculate the imaging angle (pitch) and height of the imaging unit 21. Note that, for example, when the imaging angle of view is unknown, the second acquisition unit 14 may estimate the imaging angle of view as follows. That is, since the second acquisition unit 14 can estimate, for example, that the above-mentioned projective transformation matrix is ​​correct, the second acquisition unit 14 can estimate the imaging angle of view by changing the imaging angle up and down without changing the height of the imaging unit 21 and comparing the mapping coordinates (coordinates on the map) when projecting the camera image onto the map with the coordinates calculated based on the imaging angle and the height of the imaging unit 21.

[0063] The second acquisition unit 14 may acquire the position of the moving object on the map (the position (P0) directly below the imaging unit 21) using the third distance (d_BP0). Furthermore, the second acquisition unit 14 acquires the position of the moving object based on the ratio (division distance ratio) acquired by the first acquisition unit 13, the imaging angle when the imaging unit 21 images the ground, and the height of the imaging unit 21. The second acquisition unit 14 may estimate the position (BP0) of the moving object on the map by projecting multiple positions recorded in an image based on the image information onto a position on the map based on the map information.

[0064] In step ST104, the second acquisition unit 14 tracks the position of the moving object according to the change in the movement of the position estimated in step ST103.

[0065] Each of the above-described units of the information processing device 1 may be realized as a function of a computer's arithmetic processing unit, etc. That is, the reception unit 12, the first acquisition unit 13, the second acquisition unit 14, and the output control unit 15 (control unit 11) of the information processing device 1 may be realized as a reception function, a first acquisition function, a second acquisition function, and an output control function (control function) by a computer's arithmetic processing unit, etc. The information processing program can cause a computer to realize each of the above-described functions. The information processing program may be recorded on a computer-readable non-transitory recording medium, such as a memory, a solid-state drive, a hard disk drive, or an optical disk. Furthermore, as described above, each unit of the information processing device 1 may be realized by an arithmetic processing device of a computer or the like. The arithmetic processing device or the like is configured by, for example, an integrated circuit or the like. Therefore, each unit of the information processing device 1 may be realized as a circuit that constitutes the arithmetic processing device or the like. That is, the reception unit 12, first acquisition unit 13, second acquisition unit 14, and output control unit 15 (control unit 11) of the information processing device 1 may be realized as a reception circuit, first acquisition circuit, second acquisition circuit, and output control circuit (control circuit) that constitute the arithmetic processing device of a computer or the like. The communication unit 22, storage unit 23, and display unit 24 (output unit) of the information processing device 1, and the imaging unit 21 may be realized as a communication function, a storage function, and a display function (output function) including the functions of an arithmetic processing device, etc. The communication unit 22, storage unit 23, and display unit 24 (output unit), and the imaging unit 21 of the information processing device 1 may be realized as a communication circuit, a storage circuit, and a display circuit (output circuit), and an imaging circuit by being configured with, for example, an integrated circuit, etc. The communication unit 22, storage unit 23, and display unit 24 (output unit), and the imaging unit 21 of the information processing device 1 may be realized as a communication device, a storage device, and a display device (output device), and an imaging device by being configured with, for example, a plurality of devices.

[0066] The information processing device 1 can be configured by combining one or any combination of the above-described multiple units. In this disclosure, the term "information" is used, but the term "information" can be replaced with "data" and the term "data" can be replaced with "information."

[0067] [Aspects and Effects of the Present Embodiment] Next, one aspect of this embodiment and the effects of each aspect will be described. Note that this embodiment is not limited to the aspects described below, and may be realized by appropriately combining the above-mentioned parts. Furthermore, the effects described below are examples, and the effects of each aspect are not limited to those described below.

[0068] (Aspect 1) An information processing device of one embodiment includes a memory unit that stores map information related to a map, a reception unit that receives image information related to an image captured by an imaging unit disposed on a moving body, a first acquisition unit that acquires, based on the image information received by the reception unit, a ratio between a first distance on the map from the center position of the image to a first edge of the image farther from the moving body and a second distance on the map from the center position of the image to a second edge of the image closer to the moving body, and a second acquisition unit that acquires the position of the moving body based on the ratio acquired by the first acquisition unit, the imaging angle when capturing an image of the ground by the imaging unit, and the height of the imaging unit. As a result, when the imaging angle and the height of the imaging unit are input, for example, the information processing device can acquire the position of the moving body by performing calculations using the input information and the ratio (division distance ratio) acquired by the first acquisition unit. Furthermore, the information processing device can acquire the position of a mobile object by calculation based on image information, even if the mobile object does not have a GPS receiver. Therefore, the information processing device can reduce the weight and power consumption of the mobile object compared to when the mobile object has a GPS receiver. Furthermore, the information processing device can acquire the position of the mobile object even when the mobile object is moving in an environment where it cannot receive GPS signals.

[0069] (Aspect 2) An information processing device of one embodiment includes a memory unit that stores map information related to a map, a reception unit that receives image information related to an image captured by an imaging unit disposed on a moving body, a first acquisition unit that acquires, based on the image information received by the reception unit, a ratio between a first distance on the map from the center position of the image to a first edge of the image farther from the moving body and a second distance on the map from the center position of the image to a second edge of the image closer to the moving body, and a second acquisition unit that acquires the height from the ground to the imaging unit and the position of the moving body based on the ratio acquired by the first acquisition unit and the imaging angle when capturing an image of the ground by the imaging unit. As a result, when an imaging angle is input, for example, the information processing device can obtain the height of the moving body and further obtain the position of the moving body by performing calculations using the input content and the ratio (division distance ratio) obtained by the first acquisition unit.

[0070] (Aspect 3) An information processing device of one embodiment includes a memory unit that stores map information related to a map, a reception unit that receives image information related to an image captured by an imaging unit disposed on a moving body, a first acquisition unit that acquires, based on the image information received by the reception unit, a ratio between a first distance on the map from the center position of the image to a first edge of the image farther from the moving body and a second distance on the map from the center position of the image to a second edge of the image closer to the moving body, and a second acquisition unit that acquires the height from the ground to the imaging unit and the position of the moving body based on the ratio acquired by the first acquisition unit and a correlation between the imaging angle of the imaging unit relative to the imaging angle of view of the imaging unit and the ratio. As a result, the information processing device can acquire the imaging angle and the height of the imaging unit by performing calculations using, for example, the correlation described above. Furthermore, the information processing device can acquire the position of the moving object by performing calculations using the ratio (division distance ratio), the imaging angle, and the height of the imaging unit.

[0071] (Aspect 4) In one embodiment of the information processing device, the second acquisition unit acquires the height from the ground to the imaging unit based on the ratio of the distances on the map as a ratio and the imaging angle, and acquires a third distance from the second edge to directly below the moving body based on the center position of the image based on the image information, the angle between the imaging unit and the second edge of the image, the imaging angle, and the height from the ground to the imaging unit, and may use the third distance to acquire the position of the moving body on the map. This allows the information processing device to obtain the position of the moving object through calculation.

[0072] (Aspect 5) In one embodiment of the information processing device, the second acquisition unit may estimate the position of the moving body on the map by projecting multiple positions recorded in an image based on image information received by the reception unit onto positions on a map based on map information. This allows the information processing device to obtain the position of the moving object through calculation.

[0073] (Aspect 6) In the information processing device of one aspect, the second acquisition unit may track the position of the moving object according to a change in the movement of the position recorded in the image information received by the reception unit as the moving object moves. This allows the information processing device to track the position of a moving object.

[0074] (Aspect 7) In one aspect of the information processing method, a computer having a memory unit that stores map information about a map includes a receiving step in which it receives image information about an image captured by an imaging unit disposed in a moving body, a first acquisition step in which it acquires, based on the image information received in the receiving step, a ratio between a first distance on the map from the center position of the image to a first edge of the image farther from the moving body and a second distance on the map from the center position of the image to a second edge of the image closer to the moving body, and a second acquisition step in which it acquires the position of the moving body based on the ratio acquired in the first acquisition step, the imaging angle when capturing an image of the ground by the imaging unit, and the height of the imaging unit. Execute. As a result, the information processing method can achieve the same effects as the information processing device of the above-described aspect.

[0075] (Aspect 8) An information processing program of one embodiment causes a computer to realize a memory function that stores map information related to a map, a reception function that receives image information related to an image captured by an imaging function disposed in a moving body, a first acquisition function that acquires, based on the image information received by the reception function, a ratio between a first distance on the map from the center position of the image to a first edge of the image farther from the moving body and a second distance on the map from the center position of the image to a second edge of the image closer to the moving body, and a second acquisition function that acquires the position of the moving body based on the ratio acquired by the first acquisition function, the imaging angle when capturing an image of the ground by the imaging function, and the height of the imaging function. As a result, the information processing program can achieve the same effects as the information processing device of the above-described aspect.

[0076] (Aspect 9) In one aspect of the information processing method, a computer having a memory unit that stores map information related to a map executes a reception step of receiving image information related to an image captured by an imaging unit disposed on a moving body, a first acquisition step of acquiring, based on the image information received in the reception step, a ratio between a first distance on the map from the center position of the image to a first edge of the image farther from the moving body and a second distance on the map from the center position of the image to a second edge of the image closer to the moving body, and a second acquisition step of acquiring the height from the ground to the imaging unit and the position of the moving body based on the ratio acquired in the first acquisition step and the imaging angle when capturing an image of the ground by the imaging unit. As a result, the information processing method can achieve the same effects as the information processing device of the above-described aspect.

[0077] (Aspect 10) An information processing program of one embodiment causes a computer to realize the following: a storage function for storing map information related to a map; a reception function for receiving image information related to an image captured by an imaging function disposed in a moving body; a first acquisition function for acquiring, based on the image information received by the reception function, a ratio between a first distance on the map from the center position of the image to a first edge of the image farther from the moving body and a second distance on the map from the center position of the image to a second edge of the image closer to the moving body; and a second acquisition function for acquiring the height from the ground to the imaging function and the position of the moving body based on the ratio acquired by the first acquisition function and the imaging angle when capturing an image of the ground by the imaging function. As a result, the information processing program can achieve the same effects as the information processing device of the above-described aspect.

[0078] (Aspect 11) In one embodiment of the information processing method, a computer having a memory unit that stores map information about a map performs the following steps: a reception step in which the computer receives image information about an image captured by an imaging unit disposed on a moving body; a first acquisition step in which the computer acquires, based on the image information received in the reception step, a ratio between a first distance on the map from the center position of the image to a first edge of the image farther from the moving body and a second distance on the map from the center position of the image to a second edge of the image closer to the moving body; and a second acquisition step in which the computer acquires the height from the ground to the imaging unit and the position of the moving body based on the ratio acquired in the first acquisition step and the correlation between the imaging angle of the imaging unit and the ratio relative to the imaging angle of the imaging unit. As a result, the information processing method can achieve the same effects as the information processing device of the above-described aspect.

[0079] (Aspect 12) An information processing program of one embodiment causes a computer to realize the following: a storage function for storing map information related to a map; a reception function for receiving image information related to an image captured by an imaging function disposed in a moving body; a first acquisition function for acquiring, based on the image information received by the reception function, a ratio between a first distance on the map from the center position of the image to a first edge of the image farther from the moving body and a second distance on the map from the center position of the image to a second edge of the image closer to the moving body; and a second acquisition function for acquiring the height from the ground to the imaging function and the position of the moving body based on the ratio acquired by the first acquisition function and the correlation between the imaging angle of the imaging function and the ratio relative to the imaging angle of view of the imaging function. As a result, the information processing program can achieve the same effects as the information processing device of the above-described aspect. [Explanation of symbols]

[0080] 1. Information processing equipment 11 Control section 12 Reception 13 First acquisition part 14 Second acquisition part 15 Output control section 21 Imaging unit 22 Communications Department 23 Memory section 24 Display

Claims

1. An input unit to which an imaging angle when imaging the ground by an imaging unit and a height of the imaging unit are input; a storage unit that stores map information relating to a map; a receiving unit that receives image information relating to an image captured by the imaging unit disposed in the moving object; a first acquisition unit that projects at least four positions in an image based on image information received by the reception unit onto the map, and acquires a ratio between a first distance on the map from the center position of the image to a first edge of the image farther from the moving body and a second distance on the map from the center position of the image to a second edge of the image closer to the moving body based on the coordinates of a center position on the projected map, a first position farther from the moving body on the projected map, and a second position closer to the moving body on the projected map, as well as the coordinates of four corners on the map; a second acquisition unit that acquires the position of the moving body based on the ratio acquired by the first acquisition unit, the imaging angle when the ground is imaged by the imaging unit, and the height of the imaging unit, and acquires a third distance from the second position corresponding to the second edge to a position directly below the moving body based on the angle formed between the center position on the map, the position of the imaging unit, and the second position corresponding to the midpoint of the second edge of the image, the imaging angle when the ground is imaged by the imaging unit, and the height of the imaging unit, and acquires coordinates of the position of the moving body on a straight line passing through the first position, the center position, and the second position on the map, using the first distance, the second distance, and the third distance; An information processing device comprising:

2. The second acquisition unit tracks the position of the moving object in accordance with changes in the positional movement recorded in the image information received by the reception unit as the moving object moves. The information processing device according to claim 1 .

3. A computer comprising: an input unit for inputting an imaging angle when imaging the ground by an imaging unit and a height of said imaging unit; and a storage unit for storing map information relating to a map, a receiving step of receiving image information relating to an image captured by the imaging unit disposed in the moving object; a first acquisition step of projecting onto the map at least four positions in an image based on the image information received in the receiving step, and acquiring a ratio of a first distance on the map from the center position of the image to a first edge of the image farther from the moving body and a second distance on the map from the center position of the image to a second edge of the image closer to the moving body based on the coordinates of a center position on the projected map, a first position farther from the moving body on the projected map, and a second position closer to the moving body on the projected map, as well as the coordinates of four corners on the map; a second acquisition step of acquiring the position of the moving body based on the ratio acquired in the first acquisition step, the imaging angle when the ground is imaged by the imaging unit, and the height of the imaging unit, wherein a third distance is acquired from the second position corresponding to the second edge to a position directly below the moving body based on the angle formed between the center position on the map, the position of the imaging unit, and the second position corresponding to the midpoint of the second edge of the image, the imaging angle when the ground is imaged by the imaging unit, and the height of the imaging unit, and acquiring coordinates of the position of the moving body on a straight line passing through the first position, the center position, and the second position on the map using the first distance, the second distance, and the third distance; An information processing method that performs the above.

4. A computer having an input unit into which an imaging angle when imaging the ground by an imaging function and a height of the imaging function are input, a memory function for storing map information relating to a map; a receiving function for receiving image information relating to an image captured by the imaging function disposed in the moving object; a first acquisition function that projects at least four positions in an image based on image information received by the reception function onto the map, and acquires the ratio of a first distance on the map from the center position of the image to a first edge of the image farther from the moving body and a second distance on the map from the center position of the image to a second edge of the image closer to the moving body based on the coordinates of a center position on the projected map, a first position farther from the moving body on the projected map, and a second position closer to the moving body on the projected map, as well as the coordinates of four corners on the map; a second acquisition function that acquires the position of the moving body based on the ratio acquired by the first acquisition function, the imaging angle when the ground is imaged by the imaging function, and the height of the imaging function, and acquires a third distance from the second position corresponding to the second edge to a position directly below the moving body based on the angle formed between the center position on the map, the position of the imaging function, and the second position corresponding to the midpoint of the second edge of the image, the imaging angle when the ground is imaged by the imaging function, and the height of the imaging function, and acquires coordinates of the position of the moving body on a straight line passing through the first position, the center position on the map, and the second position, respectively, using the first distance, the second distance, and the third distance; An information processing program that makes this possible.

Images