Information processing device, information processing method, and information processing program

Abstract

To provide an information processing device, an information processing method, and an information processing program, capable of identifying a position of a subject.SOLUTION: An information processing device comprises: a first obtainment section for obtaining map information in which photographing longitude and latitude information is recorded; a second obtainment section for obtaining image information added with longitude and latitude information based on the longitude and latitude information and the image information photographed by a photographing unit; a third obtainment section for obtaining the photographing longitude and latitude information pertaining to longitude and latitude of the photographing unit; a fourth obtainment section for obtaining a distance pertaining to the photographing unit in the image information generated by the photographing unit, based on the image information added with the longitude and latitude information and the photographing longitude and latitude information; and a fifth obtainment section for obtaining a distance to a subject from the photographing unit, based on the image information added with the longitude and latitude information and the distance pertaining to the photographing unit.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to an information processing device, an information processing method, and an information processing program. [Background technology]

[0002] Conventionally, there is a technology that recognizes a subject based on an image captured by an imaging unit and obtains the position of the subject, etc. Patent Document 1 describes a method of creating a difference image from two images captured by an imaging unit (a reference image serving as a standard and an image in which the subject appears), recognizing the subject from the difference image, and acquiring the position of the recognized subject based on the angle of view, installation angle, and installation position of the imaging unit. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2016-181786 A Summary of the Invention [Problem to be solved by the invention]

[0004] However, although Patent Document 1 describes acquiring the position of the subject based on the angle of view, installation angle, and installation position of the imaging unit, it does not describe how to specifically acquire the position of the subject based on the angle of view, installation angle, and installation position of the imaging unit.

[0005] An object of the present invention is to provide an information processing device, an information processing method, and an information processing program capable of identifying the position of a subject. [Means for solving the problem]

[0006] An information processing device of one embodiment includes an imaging unit that images a subject and generates image information, a first acquisition unit that acquires map information in which latitude and longitude information regarding longitude and latitude is recorded, including an imaging range in which the subject is imaged by the imaging unit, a second acquisition unit that acquires image information with latitude and longitude information by corresponding longitude and latitude within the imaging range in the map information to a predetermined position in the image information based on the latitude and longitude information in the map information acquired by the first acquisition unit and image information captured by the imaging unit, a third acquisition unit that acquires imaging latitude and longitude information regarding the longitude and latitude of the imaging unit based on at least four points in the image information and at least four points in the map information corresponding to the at least four points, a fourth acquisition unit that acquires a distance related to the imaging unit in the image information generated by the imaging unit based on the image information with the latitude and longitude information acquired by the second acquisition unit and the imaging latitude and longitude information acquired by the third acquisition unit, and a fifth acquisition unit that acquires a distance from the imaging unit to the subject based on the image information with the latitude and longitude information acquired by the second acquisition unit and the distance related to the imaging unit acquired by the fourth acquisition unit.

[0007] In one embodiment of the information processing device, the second acquisition unit may associate a longitude and latitude based on the longitude and latitude information with a specified position of the image information based on associating a plurality of positions that are characteristic of the image information with a plurality of positions that correspond to features in the map information.

[0008] In the information processing device of one aspect, the second acquisition unit may associate longitude and latitude with each of a plurality of pixels constituting the image information.

[0009] In one embodiment of the information processing device, when the image information records a rectangular imaging range having a vertical and horizontal direction, the third acquisition unit may set at least four points in the image information by setting at least two points on each of at least two vertical lines in the imaging range.

[0010] An information processing device of one embodiment may be provided with a generation unit that generates a movement path of the moving body based on image information with longitude and latitude information acquired by the second acquisition unit and a distance from the imaging unit to the subject acquired by the fifth acquisition unit, when the subject imaged by the imaging unit is a moving body.

[0011] An information processing device of one embodiment may be provided with a determination unit that, when a moving body imaged by an imaging unit is temporarily blocked by an obstruction, predicts the destination of the moving body based on the movement path generated by the generation unit, and determines whether the moving body before being temporarily blocked by the obstruction and the moving body after being temporarily blocked by the obstruction are the same.

[0012] An information processing device of one embodiment may also be provided with a sixth acquisition unit that acquires the longitude and latitude of the position of the subject imaged by the imaging unit based on image information accompanied by latitude and longitude information acquired by the second acquisition unit and the distance from the imaging unit to the subject acquired by the fifth acquisition unit.

[0013] In one aspect of the information processing method, a computer includes an imaging step of imaging an object and generating image information with an imaging unit, a first acquisition step of acquiring map information in which latitude and longitude information regarding longitude and latitude is recorded, the map information including an imaging range in which the object is imaged in the imaging step, a second acquisition step of acquiring image information with latitude and longitude information by associating the longitude and latitude within the imaging range in the map information with a predetermined position in the image information based on the latitude and longitude information in the map information acquired in the first acquisition step and image information captured in the imaging step, and a third acquisition step of acquiring imaging latitude and longitude information regarding the longitude and latitude of the imaging unit based on at least four points in map information corresponding to the imaging unit; a fourth acquisition step of acquiring a distance related to the imaging unit in the image information generated by the imaging step based on the image information with the latitude and longitude information acquired by the second acquisition step and the imaging latitude and longitude information acquired by the third acquisition step; and a fifth acquisition step of acquiring a distance from the imaging unit to the subject based on the image information with the latitude and longitude information acquired by the second acquisition step and the distance related to the imaging unit acquired by the fourth acquisition step.

[0014] An information processing program according to one embodiment includes a computer, the computer having an imaging function for imaging a subject and generating image information, a first acquisition function for acquiring map information in which latitude and longitude information regarding longitude and latitude is recorded, the map information including an imaging range in which the subject is imaged by the imaging function, a second acquisition function for acquiring image information to which latitude and longitude information is added by associating the longitude and latitude within the imaging range in the map information with a predetermined position in the image information based on the latitude and longitude information in the map information acquired by the first acquisition function and image information captured by the imaging function, and a second acquisition function for acquiring image information to which latitude and longitude information is added, the computer having a ... a third acquisition function that acquires imaging latitude and longitude information regarding the longitude and latitude of the imaging function based on at least four points in map information acquired by the second acquisition function; a fourth acquisition function that acquires a distance related to the imaging function in image information generated by the imaging function based on the image information with the latitude and longitude information acquired by the second acquisition function and the imaging latitude and longitude information acquired by the third acquisition function; and a fifth acquisition function that acquires a distance from the imaging function to a subject based on the image information with the latitude and longitude information acquired by the second acquisition function and the distance related to the imaging function acquired by the fourth acquisition function. Effect of the Invention

[0015] An information processing device of one embodiment includes a first acquisition unit that acquires map information in which imaging latitude and longitude information is recorded, a second acquisition unit that acquires image information with latitude and longitude information added based on the latitude and longitude information and image information captured by the imaging unit, a third acquisition unit that acquires imaging latitude and longitude information related to the longitude and latitude of the imaging unit, a fourth acquisition unit that acquires a distance related to the imaging unit in the image information generated by the imaging unit based on the image information with latitude and longitude information and the imaging latitude and longitude information, and a fifth acquisition unit that acquires the distance from the imaging unit to the subject based on the image information with latitude and longitude information and the distance related to the imaging unit, thereby making it possible to identify the position of the subject captured by the imaging unit. Furthermore, the information processing method and the information processing program according to one aspect can achieve the same effects as the information processing device according to the above aspect. [Brief description of the drawings]

[0016] [Figure 1] FIG. 1 is a diagram illustrating an information processing device according to an embodiment. [Diagram 2] FIG. 1 is a block diagram illustrating an information processing device according to an embodiment. [Diagram 3] 1A and 1B are diagrams for explaining contents related to processing performed by the second acquisition unit 12. (A) is a diagram for explaining contents related to an image captured by the imaging unit, and (B) is a diagram for explaining contents related to map information. [Figure 4] 11A and 11B are diagrams for explaining contents related to a process performed by a fourth acquisition unit, where (A) is a diagram for explaining contents related to an image captured by an imaging unit, and (B) is a diagram for explaining contents related to map information. [Diagram 5] 13 is a diagram for explaining contents related to a process performed by a fourth acquisition unit. FIG. [Figure 6] 1A is a diagram for explaining the contents related to the processing performed by the generation unit, and FIG. 1B is a diagram for explaining the contents related to the map information. [Figure 7] 1 is a first flowchart illustrating an information processing method according to an embodiment. [Figure 8] 11 is a second flowchart illustrating an information processing method according to an embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0017] Hereinafter, one embodiment of the present invention will be described. In this specification, the term "information" is used, but the term "information" can be replaced with "data" and the term "data" can be replaced with "information".

[0018] First, an overview of an information processing device 1 according to an embodiment will be described. FIG. 1 is a diagram illustrating an information processing device 1 according to an embodiment.

[0019] The information processing device 1 acquires image information generated by an imaging unit 21 that captures an image of a subject. The imaging unit 21 may be, for example, a fixed camera such as a surveillance camera. The fixed camera may be, for example, a camera that captures an image of a predetermined location for a certain period of time, or may be a movable camera.

[0020] The information processing device 1 acquires position information (for example, information related to longitude and latitude) for an arbitrary position in an image captured by the imaging unit 21, and the distance from the imaging unit 21 to the arbitrary position in the image. In this case, first, the information processing device 1 associates multiple positions A within the imaging range captured by the imaging unit 21 with multiple positions B in the map information corresponding to each of the multiple positions A, and identifies the longitude and latitude within the image captured by the imaging unit 21. Next, the information processing device 1 uses the longitude and latitude in the identified image to identify the longitude and latitude of the imaging unit 21. The information processing device 1 acquires a distance related to the imaging unit 21 in the image captured by the imaging unit 21 based on the longitude and latitude in the identified image and the longitude and latitude of the imaging unit 21. The information processing device 1 stores information (distance-latitude information) related to the longitude and latitude of an arbitrary position in the identified image and the distance related to the imaging unit 21 in the image in the storage unit 22 (see FIG. 2).

[0021] When the information processing device 1 captures a new image of a subject using the imaging unit 21, the information processing device 1 acquires the position of the subject (for example, longitude and latitude) and the distance from the imaging unit 21 to the subject based on the distance and latitude information stored in the storage unit 22.

[0022] The above-mentioned information processing device 1 may be, for example, a server, a desktop, a laptop, a tablet, or the like.

[0023] Next, the information processing device 1 according to an embodiment will be described in detail. FIG. 2 is a block diagram illustrating an information processing device 1 according to an embodiment.

[0024] The information processing device 1 includes an imaging unit 21, a storage unit 22, a communication unit 23, a display unit 24, and a control unit 10. The control unit 10 may be realized as one function of, for example, an arithmetic processing device. The control unit 10 may function as a first acquisition unit 11, a second acquisition unit 12, a third acquisition unit 13, a fourth acquisition unit 14, a fifth acquisition unit 15, a generation unit 16, a determination unit 17, a sixth acquisition unit 18, and an output control unit 19.

[0025] The imaging unit 21 captures an image of a subject to generate image information. The imaging unit 21 may be, for example, a camera disposed in the information processing device 1. Alternatively, the imaging unit 21 may be an external camera disposed outside the information processing device 1. In this case, the control unit 10 may acquire image information generated by the external camera via the communication unit 23. The imaging unit 21 may capture, for example, still images of the subject that are continuous in time (captured at predetermined time intervals) or a moving image.

[0026] The storage unit 22 stores, for example, various information and programs. The storage unit 22 may store, for example, map information in which latitude and longitude information relating to longitude and latitude is recorded. The storage unit 22 may store distance and longitude information. The distance and longitude information stored in the storage unit 22 may include, for example, image information (information in which pixels of the imaging unit 21 are associated with latitude and longitude information) to which latitude and longitude information is added acquired by a second acquisition unit 12 described later, and information on the distance related to the imaging unit 21 in image information (image) acquired by a fourth acquisition unit 14 described later.

[0027] The communication unit 23 communicates with, for example, devices external to the information processing device 1 (for example, an external camera, an external server, a terminal, and the like).

[0028] The display unit 24 displays, for example, various characters, symbols, images, and the like.

[0029] The first acquisition unit 11 acquires map information in which latitude and longitude information relating to longitude and latitude is recorded, including an imaging range in which an object is imaged by the imaging unit 21. The first acquisition unit 11 may acquire map information stored in the storage unit 22, for example, or may acquire map information stored in an external server or the like outside the information processing device 1 via the communication unit 23.

[0030] Fig. 3 is a diagram for explaining the contents related to the processing performed by the second acquisition unit 12. Fig. 3(A) is a diagram for explaining the contents related to the image captured by the imaging unit 21, and Fig. 3(B) is a diagram for explaining the contents related to the map information. That is, Fig. 3(B) is a diagram showing an example of a map based on the map information, including an imaging range captured by the imaging unit 21.

[0031] The second acquisition unit 12 acquires image information with the latitude and longitude information by associating the longitude and latitude within the imaging range in the map information with a predetermined position in the image information, based on the latitude and longitude information in the map information acquired by the first acquisition unit 11 and the image information captured by the imaging unit 21. Specifically, the second acquisition unit 12 may associate the longitude and latitude based on the latitude and longitude information with the predetermined position in the image information, based on associating a plurality of characteristic positions in the image information with a plurality of positions corresponding to the characteristics in the map information.

[0032] That is, the second acquisition unit 12 extracts, for example, a plurality of characteristic positions of a subject from an image of the subject captured by the imaging unit 21. As a specific example, when an intersection having a pedestrian crossing is captured by the imaging unit 21, the second acquisition unit 12 extracts a plurality of corners of the pedestrian crossing as characteristic positions. In this case, as an example, the second acquisition unit 12 may extract four positions A to D of the characteristic subject (see FIG. 3(A)). Note that the second acquisition unit 12 is not limited to the above-mentioned pedestrian crossing, and may extract various objects (fixed objects such as stationary objects) recorded in an image captured by the imaging unit 21 and recorded on a road map as characteristics.

[0033] Furthermore, the second acquisition unit 12 extracts, for example, a plurality of positions of features recorded in the map information. In this case, the second acquisition unit 12 extracts, for example, a same position of the same feature from the map information for a plurality of positions of the feature extracted from the image captured by the imaging unit 21. As a specific example, when extracting four corners A to D of a pedestrian crossing recorded in the image captured by the imaging unit 21 (see FIG. 3(A)), the second acquisition unit 12 extracts corners A' to D' of the pedestrian crossing recorded in the map information that are the same as the corners A to D (see FIG. 3(B)).

[0034] The second acquisition unit 12 associates the position information (longitude and latitude) recorded in the map information with the image (image information), for example, by associating multiple positions of a feature in the image captured by the imaging unit 21 with the same multiple positions of the same feature recorded in the map information. As a specific example, the second acquisition unit 12 associates the position information (longitude and latitude) recorded in the map information with the image (image information) by performing coordinate transformation using a projective transformation matrix based on the four corners A to D of the crosswalk recorded in the image captured by the imaging unit 21 and the corners A' to D' of the crosswalk recorded in the map information.

[0035] That is, the second acquisition unit 12 calculates a projective transformation matrix from a plurality of positions of features in an image captured by the imaging unit 21 to the map information, and performs coordinate transformation from the image captured by the imaging unit 21 to the map information. In addition, the second acquisition unit 12 calculates a projective transformation matrix from a plurality of positions of features recorded in the map information to the image captured by the imaging unit 21, and performs coordinate transformation from the map information to the image captured by the imaging unit 21.

[0036] The projective transformation matrix is ​​expressed by the following formula (1): Here, (x, y) may be, for example, the longitude and latitude before transformation, and (x', y') may be, for example, the longitude and latitude after transformation.

[0037]

number

[0038] The second acquisition unit 12 may associate longitude and latitude with each of a plurality of pixels constituting the image information. That is, the second acquisition unit 12 obtains the longitude and latitude for each pixel of the map information, for example, and generates information on the longitude and latitude corresponding to each pixel of the image captured by the imaging unit 21.

[0039] In this case, for example, when the longitude and latitude are not recorded for all pixels of the map information, the second acquisition unit 12 can obtain the longitude and latitude for each pixel by using information of a plurality of points where the longitude and latitude are recorded in the map information. As a specific example, when the map information records the longitude and latitude information of four points (Lon0,Lat0), (Lon0,Lat1), (Lon1,Lat0), and (Lon1,Lat1), the second acquisition unit 12 can calculate the longitude and latitude for each pixel in the map information (for example, information on the amount of change in longitude and latitude for each pixel) using the following formulas (2) and (3). Longitude per pixel = (Lon1-Lon0) / (distance between two points (number of pixels)) …(2) Latitude per pixel = (Lat1 - Lat0) (distance between two points (number of pixels)) ... (3) Furthermore, the second acquisition unit 12 can calculate the longitude and latitude for each pixel in the image captured by the imaging unit 21 by using the above-mentioned projective transformation matrix. The longitude is calculated based on the amount of change in latitude and longitude and the X coordinate (horizontal coordinate of the image) of the image, with the top left corner of the image captured by the imaging unit 21 as the starting point (starting longitude). Similarly, the latitude is calculated based on the amount of change in latitude and the Y coordinate (vertical coordinate of the image) of the image, with the top left corner of the image captured by the imaging unit 21 as the starting point (starting latitude).

[0040] Fig. 4 is a diagram for explaining the contents related to the processing performed by the fourth acquisition unit 14. Fig. 4(A) is a diagram for explaining the contents related to the image captured by the imaging unit 21, and Fig. 4(B) is a diagram for explaining the contents related to the map information.

[0041] The third acquisition unit 13 acquires imaging latitude and longitude information relating to the longitude and latitude of the imaging unit 21 based on at least four points in the image information and at least four points in the map information corresponding to the at least four points. In this case, when the image information records a rectangular imaging range having vertical and horizontal directions, the third acquisition unit 13 may set at least four points in the image information by setting at least two points on each of at least two lines in the vertical direction of the imaging range.

[0042] That is, the third acquisition unit 13 generates, for example, a plurality of line segments on a map based on map information, which pass through at least two points in the image captured by the imaging unit 21. Specifically, the third acquisition unit 13 extracts, for example, at least two points P1, P2 that overlap one edge in the vertical direction in the image captured by the imaging unit 21 (see FIG. 4(A)). The third acquisition unit 13 generates, for example, a line segment P1'-P2' that passes through points P1', P2' obtained by projecting the at least two points P1, P2 onto the map (see FIG. 4(B)). Similarly, the third acquisition unit 13 extracts at least two points that overlap with the other vertical edge of the image captured by the imaging unit 21, or at least two points P3, P4 on a vertical center line that passes through the center of the image captured by the imaging unit 21 (see FIG. 4(A)). The third acquisition unit 13 generates, for example, a line segment P3'-P4' that passes through points P3', P4' that are obtained by projecting the at least two points P3, P4 onto a map (see FIG. 4(B)). The third acquisition unit 13 acquires information on the longitude and latitude of the position (imaging latitude and longitude information) of the intersection P0 of the line segment P1'-P2' and the line segment P3'-P4' in the map information, which is the position where the imaging unit 21 is located (see FIG. 4(B)). Also, the direction of the camera can be obtained from the line segment P3'-P4' in the map information.

[0043] FIG. 5 is a diagram for explaining the contents related to the processing performed by the fourth acquisition unit 14. As shown in FIG.

[0044] The fourth acquisition unit 14 acquires a distance related to the imaging unit 21 in the image information generated by the imaging unit 21 based on the image information with the latitude and longitude information acquired by the second acquisition unit 12 and the imaging latitude and longitude information acquired by the third acquisition unit 13. That is, the fourth acquisition unit 14 calculates the distance (dx, dy) from the imaging unit 21 for all coordinates, for example, for each pixel of the image information generated by the imaging unit 21, based on the image information with the latitude and longitude information and the information (imaging latitude and longitude information) of the longitude and latitude (C_Lon, C_Lat) of the position where the imaging unit 21 is arranged. Here, dx is, for example, the distance from the position P0 where the imaging unit 21 is arranged on the vertical line segment C passing through the center of the image based on the image information (see FIG. 5). dy is, for example, the distance from the center line (line segment C) in the horizontal direction of the image based on the image information (see FIG. 5).

[0045] As a specific example, the fourth acquisition unit 14 calculates the distances dx, dy at an arbitrary position P5 (longitude and latitude (Lon, Lat)) in the image information using the following equations (4) and (5). dx = (Lat - C_Lat) / m1_Lat ... (4) dy = (Lon - C_Lon) / m1_Lon ... (5) Here, m1_Lat is the latitude per meter and m1_Lon is the longitude per meter.

[0046] The fourth acquisition unit 14 also combines the image information with the latitude and longitude information and the distances dx and dy calculated as described above, and assigns the latitude and longitude information and the distances dx and dy to each pixel of the image information. As a specific example, the fourth acquisition unit 14 assigns [dx, dy, Lon, Lat] to the position P5 described above. The fourth acquisition unit 14 may store in the storage unit 22, for example, information (distance and latitude / longitude information) [dx, dy, Lon, Lat] assigned to each pixel of the image information. This is the pre-processing.

[0047] The process of calculating the distance and latitude and longitude to the subject imaged by the imaging unit 21 is as follows. The fifth acquisition unit 15 acquires the distance from the imaging unit 21 to the subject based on the image information with the longitude and latitude information acquired by the second acquisition unit 12 and the distance related to the imaging unit 21 acquired by the fourth acquisition unit 14. That is, first, when the imaging unit 21 performs new imaging, the fifth acquisition unit 15 performs subject recognition based on the image captured by the imaging unit 21, for example. When recognizing a person, for example, the fifth acquisition unit 15 recognizes the person recorded in the image information using a known person recognition process and generates a rectangular frame (ROI) surrounding the person. In addition, the fifth acquisition unit 15 is not limited to the example of recognizing a person, and may recognize a moving object such as an automobile, a motorcycle, a two-wheeled vehicle, a wheelchair, and a train. As a result of performing subject recognition, the fifth acquisition unit 15 identifies the position where the subject is recognized (for example, the center position where the subject contacts the ground (the center position on the lower side of the ROI)) and identifies a pixel in the image information at that position. The fifth acquisition unit 15 acquires information on the longitude and latitude of the identified pixel (position of the subject) and information on the distance from the imaging unit 21, based on the longitude and latitude information acquired by the fourth acquisition unit . For example, when the subject moves, the fifth acquisition section 15 may acquire information on the longitude and latitude at each position according to the movement and information on the distance from the imaging section 21.

[0048] Fig. 6 is a diagram for explaining the contents related to the processing performed by the generation unit 16. Fig. 6(A) is a diagram for explaining the contents related to the image captured by the imaging unit 21, and Fig. 6(B) is a diagram for explaining the contents related to the map information.

[0049] When the subject captured by the imaging unit 21 is a moving body, the generation unit 16 may generate a movement path of the moving body based on the image information accompanied by longitude and latitude information acquired by the second acquisition unit 12 and the distance from the imaging unit 21 to the subject acquired by the fifth acquisition unit 15. In other words, when the generation unit 16 acquires information on longitude and latitude at each position according to the movement of the subject and information on the distance from the imaging unit 21 using the fifth acquisition unit 15, it becomes possible to generate a trajectory (movement route) of the movement of the subject by correlating the longitude and latitude information with map information stored in the memory unit 22.

[0050] As a specific example, when the imaging unit 21 captures an image of the vehicle E as a subject (see FIG. 6(A)), and the fifth acquisition unit 15 acquires position information of the vehicle E, the generation unit 16 projects a position based on the position information onto the map. When the vehicle E moves, the generation unit 16 projects a position based on the position information of the vehicle E sequentially acquired by the fifth acquisition unit 15 onto the map, and projects a trajectory (movement path) F of the movement of the vehicle E onto the map (see FIG. 6(B)).

[0051] Alternatively, when the generation unit 16 can obtain position information of the subject (moving object) by using, for example, a Global Navigation Satellite System (GNSS) (for example, by using a navigation device, etc.), it becomes possible to generate a trajectory (movement path) of the subject within the image captured by the imaging unit 21 by projecting and transforming the position information onto the image captured by the imaging unit 21 based on the longitude and latitude information.

[0052] When the subject (moving object) captured by the imaging unit 21 is temporarily blocked by an obstruction, the determination unit 17 may predict the destination of the moving object based on the moving path generated by the generation unit 16, and determine whether the moving object before being temporarily blocked by the obstruction is the same as the moving object after being temporarily blocked by the obstruction. The determination unit 17 may use, for example, the trajectory (moving path) of the subject (moving object) generated by the generation unit 16 to estimate the moving direction of the subject and the result of recognizing the subject (result of inferring the identity of the subject), thereby estimating the identity of the subject (for example, a vehicle, etc.) even when the subject captured by the imaging unit 21 is temporarily blocked, for example, when a vehicle as the subject is temporarily blocked by an oncoming vehicle. Here, the determination unit 17 may use, for example, known techniques for estimating the moving direction of the subject and inferring the identity of the subject.

[0053] The sixth acquisition unit 18 may acquire the longitude and latitude at the position of the subject imaged by the imaging unit 21, based on the image information with the longitude and latitude information acquired by the second acquisition unit 12 and the distance from the imaging unit 21 to the subject acquired by the fifth acquisition unit 15. That is, for example, when the subject (moving body) acquires position information using GNSS, the sixth acquisition unit 18 may estimate the continuity of the movement trajectory (movement path) of the subject based on the movement trajectory (movement path) of the subject using GNSS and the position (longitude and latitude) of the subject imaged by the imaging unit 21 and acquired by the fifth acquisition unit 15 (or the movement trajectory (movement path) generated by the generation unit 16). As a result, by disposing the imaging unit 21 in a tunnel, garage, indoors, or the like where GNSS radio waves cannot be received, the sixth acquisition unit 18 becomes able to acquire the continuity of the trajectory of the movement of the subject (for example, the position information of the subject when the subject enters a tunnel, etc.) even when the subject, whose position information is acquired using GNSS, enters a tunnel, etc.

[0054] The output control unit 19 may control the output unit to output information on longitude and latitude and distance from the imaging unit 21 acquired by the fifth acquisition unit 15, the subject's movement path generated by the generation unit 16, the determination result of the subject's identity by the determination unit 17, and the acquisition result of the subject's continuity acquired by the sixth acquisition unit 18 (collectively referred to as "output information"). The output unit may be, for example, a storage unit 22, a communication unit 23, and a display unit 24. That is, the output control unit 19 may control the storage unit 22 to store the output information, for example. The output control unit 19 may, for example, control the communication unit 23 to transmit the output information to an outside of the information processing device 1 (for example, a server (not shown)). The output control unit 19 may control the display unit 24 to display characters, symbols, and images based on the output information, for example.

[0055] Next, an information processing method according to an embodiment will be described.

[0056] First, a pre-processing performed by the information processing device 1 in an information processing method according to an embodiment will be described. FIG. 7 is a first flowchart illustrating an information processing method according to an embodiment.

[0057] In step ST101, the imaging unit 21 captures an image of a subject and generates image information. The subject in this case may be the background or the like.

[0058] In step ST102, the first acquisition unit 11 acquires map information in which longitude and latitude information is recorded. The first acquisition unit 11 may acquire map information stored in the storage unit 22, for example, or may acquire map information stored in an external server or the like outside the information processing device 1 via the communication unit 23.

[0059] In step ST103, the second acquisition unit 12 acquires image information with latitude and longitude information by associating the longitude and latitude within the imaging range in the map information with a predetermined position in the image information based on the image information captured in step ST101 and the latitude and longitude information in the map information acquired in step ST102. For example, the second acquisition unit 12 may associate the longitude and latitude based on the latitude and longitude information with the predetermined position of the image information based on associating a plurality of characteristic positions in the image information with a plurality of positions corresponding to the characteristics in the map information. In this case, the second acquisition unit 12 may associate the longitude and latitude with each of a plurality of pixels constituting the image information.

[0060] In step ST104, the third acquisition unit 13 acquires imaging latitude and longitude information relating to the longitude and latitude of the imaging unit 21 based on at least four points in the image information and at least four points in the map information corresponding to the at least four points. In this case, when the image information records a rectangular imaging range having vertical and horizontal directions, the third acquisition unit 13 may set at least four points in the image information by setting at least two points on each of at least two lines in the vertical direction of the imaging range.

[0061] In step ST105, the fourth acquisition unit 14 acquires distances dx, dy related to the imaging unit 21 in the image information generated by the imaging unit 21 based on the image information with latitude and longitude information acquired in step ST102 and the imaging latitude and longitude information acquired in step ST103. The fourth acquisition unit 14 may store in the storage unit 22, for example, information (distance and latitude / longitude information) [dx, dy, Lon, Lat] assigned to each pixel of the image information. This is the pre-processing.

[0062] Next, post-processing such as acquisition of subject position information performed by the information processing device 1, which is an information processing method according to an embodiment, will be described. FIG. 8 is a second flowchart illustrating the information processing method according to an embodiment.

[0063] In step ST201, the imaging unit 21 captures an image of a subject. The subject in this case may be, for example, a person, or a moving object such as an automobile, a motorcycle, a two-wheeled vehicle, a wheelchair, or a train.

[0064] In step ST202, the fifth acquisition unit 15 recognizes the subject captured in step ST201, identifies the position of the subject on the image, and acquires position information of the identified position (subject) and the distance from the imaging unit 21 to the identified position (subject) based on the longitude and latitude distance information acquired in the preprocessing described with reference to Fig. 7. When the subject captured in step ST101 moves, the fifth acquisition unit 15 acquires the position information of the subject and the distance from the imaging unit 21 to the subject, for example, continuously or intermittently (at every predetermined time) in accordance with the movement of the subject.

[0065] In step ST203, when the subject captured in step ST101 moves, the generation unit 16 generates a moving route of the subject based on the distance and latitude / longitude information. That is, the generation unit 16 projects a trajectory (moving route) of the subject's movement on a map based on, for example, the position information of the subject acquired in step ST202 and the distance from the imaging unit 21 to the subject.

[0066] In step ST204, when the subject imaged in step ST101 is temporarily blocked by an obstruction, the judgment unit 17 predicts the destination of the subject based on the movement path generated in step ST203, and judges whether the subject before being temporarily blocked by the obstruction and the subject after being temporarily blocked by the obstruction are the same.

[0067] In step ST205, the sixth acquisition unit 18 acquires the longitude and latitude of the position of the subject imaged in step ST205 based on the longitude and latitude information. For example, when the subject acquires position information using GNSS, the sixth acquisition unit 18 may estimate the continuity of the movement path of the subject based on the trajectory (movement path) of the movement of the subject using GNSS and the position (longitude and latitude) of the subject acquired in step ST202 (or the movement path generated in step ST203). This makes it possible for the sixth acquisition unit 18 to acquire the continuity of the movement path of the subject (for example, position information of the subject when the subject enters a tunnel, etc.) even when, for example, the imaging unit 21 is disposed in a place where it cannot receive radio waves of GNSS.

[0068] Next, the effects of this embodiment will be described. The information processing device 1 includes a first acquisition unit 11 that acquires map information in which latitude and longitude information is recorded, including an imaging range in which a subject is imaged by the imaging unit 21; a second acquisition unit 12 that acquires image information with latitude and longitude information by corresponding longitude and latitude within the imaging range in the map information to a predetermined position in the image information based on the latitude and longitude information and image information captured by the imaging unit 21; a third acquisition unit 13 that acquires the imaging latitude and longitude information; a fourth acquisition unit 14 that acquires a distance related to the imaging unit 21 in the image information generated by the imaging unit 21 based on the image information with the latitude and longitude information and the imaging latitude and longitude information; and a fifth acquisition unit 15 that acquires the distance from the imaging unit 21 to the subject based on the image information with the latitude and longitude information and the distance related to the imaging unit 21. This enables the information processing device 1 to identify the position (for example, longitude and latitude) of the subject imaged by the imaging unit 21.

[0069] In the information processing device 1, the second acquisition unit 12 may correspond the longitude and latitude based on the longitude and latitude information to a specified position of the image information based on corresponding multiple positions that are characteristic of the image information with multiple positions that correspond to features in the map information. The information processing device 1 can associate the position information of the map information with the image captured by the imaging unit 21. Thus, the information processing device 1 can identify the position (for example, longitude and latitude) of the subject captured by the imaging unit 21.

[0070] In the information processing device 1, the second acquisition unit 12 may associate longitude and latitude with each of a plurality of pixels that constitute the image information. This allows the information processing device 1 to associate position information with each pixel, thereby enabling it to identify more detailed position information of the subject.

[0071] In the information processing device 1, when the image information records a rectangular imaging range having a vertical and horizontal direction, the third acquisition unit 13 may set at least four points in the image information by setting at least two points on each of at least two vertical lines in the imaging range. This enables the information processing device 1 to obtain position information on the position where the imaging unit 21 is located.

[0072] When the subject captured by the imaging unit 21 is a moving body, the information processing device 1 may be provided with a generation unit 16 that generates a movement path of the moving body based on image information with longitude and latitude information and the distance from the imaging unit 21 to the subject. This enables the information processing device 1 to plot the trajectory (movement route) of the subject's movement on a map, or to plot the trajectory (movement route) of the subject's movement on an image captured by the imaging unit 21.

[0073] The information processing device 1 may be provided with a judgment unit 17 that, when a subject (moving object) imaged by the imaging unit 21 is temporarily blocked by an obstruction, predicts the destination of the moving object based on the movement path generated by the generation unit 16 and judges whether the moving object before it is temporarily blocked by the obstruction and the moving object after it is temporarily blocked by the obstruction are the same. Since the information processing device 1 can determine the identity of the subject (moving object), for example, when the subject is a vehicle, even if the subject is blocked by an oncoming vehicle, the information processing device 1 can generate the trajectory (moving path) of the vehicle's movement.

[0074] The information processing device 1 may be provided with a sixth acquisition unit 18 that acquires the longitude and latitude of the position of the subject imaged by the imaging unit 21 based on the image information with latitude and longitude information and the distance from the imaging unit 21 to the subject. As a result, when the subject can obtain location information using GNSS, even if the subject moves to a location where location information using GNSS cannot be obtained, the information processing device 1 can determine the identity of the subject and estimate the subject's destination by placing the imaging unit 21 at the location.

[0075] In the information processing method, a computer executes a first acquisition step of acquiring map information in which latitude and longitude information is recorded, including an imaging range in which the subject is imaged in the imaging step; a second acquisition step of acquiring image information with latitude and longitude information by corresponding the longitude and latitude within the imaging range in the map information to a predetermined position in the image information based on the latitude and longitude information and the image information captured by the imaging step; a third acquisition step of acquiring the imaging latitude and longitude information; a fourth acquisition step of acquiring a distance related to the imaging unit 21 in the image information generated by the imaging step based on the image information with the latitude and longitude information and the imaging latitude and longitude information; and a fifth acquisition step of acquiring the distance from the imaging unit 21 to the subject based on the image information with the latitude and longitude information and the distance related to the imaging unit 21. The information processing method can associate position information of the map information with an image captured by the imaging unit 21. Thus, the information processing method can specify the position (e.g., longitude and latitude) of the subject captured by the imaging unit 21.

[0076] The information processing program causes the computer to realize a first acquisition function that acquires map information in which latitude and longitude information is recorded, including an imaging range in which a subject is imaged by the imaging function; a second acquisition function that acquires image information with latitude and longitude information by corresponding longitude and latitude within the imaging range in the map information to a predetermined position in the image information based on the latitude and longitude information and image information captured by the imaging function; a third acquisition function that acquires the imaging latitude and longitude information; a fourth acquisition function that acquires a distance related to the imaging function in the image information generated by the imaging function based on the image information with latitude and longitude information and the imaging latitude and longitude information; and a fifth acquisition function that acquires the distance from the imaging function to the subject based on the image information with latitude and longitude information and the distance related to the imaging function. The information processing program can associate the position information of the map information with the image captured by the imaging unit 21. Thus, the information processing program can identify the position (e.g., longitude and latitude) of the subject captured by the imaging unit 21.

[0077] Each unit of the information processing device 1 described above may be realized as a function of a processing unit of a computer or the like. That is, the first acquisition unit 11, the second acquisition unit 12, the third acquisition unit 13, the fourth acquisition unit 14, the fifth acquisition unit 15, the generation unit 16, the determination unit 17, the sixth acquisition unit 18, and the output control unit 19 (control unit 10) of the information processing device 1 may be realized as a first acquisition function, a second acquisition function, a third acquisition function, a fourth acquisition function, a fifth acquisition function, a generation function, a determination function, a sixth acquisition function, and an output control function (control function), respectively, by the processing unit of a computer or the like. The information processing program can cause a computer to realize each of the above-mentioned functions. The information processing program may be recorded in a non-transitory computer-readable recording medium, such as an external memory or an optical disk. As described above, each unit of the information processing device 1 may be realized by an arithmetic processing device of a computer or the like. The arithmetic processing device or the like is configured by, for example, an integrated circuit or the like. Therefore, each unit of the information processing device 1 may be realized as a circuit constituting the arithmetic processing device or the like. That is, the first acquisition unit 11, the second acquisition unit 12, the third acquisition unit 13, the fourth acquisition unit 14, the fifth acquisition unit 15, the generation unit 16, the judgment unit 17, the sixth acquisition unit 18, and the output control unit 19 (control unit 10) of the information processing device 1 may be realized as a first acquisition circuit, a second acquisition circuit, a third acquisition circuit, a fourth acquisition circuit, a fifth acquisition circuit, a generation circuit, a judgment circuit, a sixth acquisition circuit, and an output control circuit (control circuit) constituting the arithmetic processing device of a computer or the like. The imaging unit 21, the storage unit 22, the communication unit 23, and the display unit 24 of the information processing device 1 may be realized as an imaging function, a storage function, a communication function, and a display function including the functions of an arithmetic processing device, etc. The imaging unit 21, the storage unit 22, the communication unit 23, and the display unit 24 of the information processing device 1 may be realized as an imaging circuit, a storage circuit, a communication circuit, and a display circuit by being configured, for example, by an integrated circuit, etc. The imaging unit 21, the storage unit 22, the communication unit 23, and the display unit 24 of the information processing device 1 may be realized as an imaging device, a storage device, a communication device, and a display device by being configured, for example, by being configured by a plurality of devices. [Explanation of symbols]

[0078] 1. Information processing device 10 Control section 11 First acquisition part 12 Second acquisition part 13 Third acquisition part 14 4th acquisition part 15 5th acquisition part 16 Generation part 17 Judgment section 18 6th acquisition part 19 Output control section 21 Imaging unit 22 Memory section 23 Communications Department 24 Display section

Claims

1. an imaging unit that captures an image of a subject and generates image information; a first acquisition unit that acquires map information in which latitude and longitude information relating to longitude and latitude is recorded, the map information including an imaging range in which an object is imaged by the imaging unit; a second acquisition unit that acquires image information to which latitude and longitude information is added by associating longitude and latitude within an imaging range in the map information with a predetermined position in the image information, based on latitude and longitude information in the map information acquired by the first acquisition unit and image information captured by the imaging unit; a third acquisition unit that acquires imaging latitude and longitude information relating to the longitude and latitude of the imaging unit based on at least four points in the image information and at least four points in the map information corresponding to the at least four points; a fourth acquisition unit that acquires a distance related to the imaging unit in image information generated by the imaging unit based on image information with latitude and longitude information acquired by the second acquisition unit and imaging latitude and longitude information acquired by the third acquisition unit; a fifth acquisition unit that acquires a distance from the imaging unit to a subject based on image information to which latitude and longitude information is added and acquired by the second acquisition unit, and a distance related to the imaging unit and acquired by the fourth acquisition unit; An information processing device comprising:

2. The second acquisition unit associates a longitude and a latitude based on the longitude and latitude information with a predetermined position of the image information based on association between a plurality of positions that are characteristic of the image information and a plurality of positions that correspond to the characteristics of the image information in the map information. The information processing device according to claim 1 .

3. The second acquisition unit associates a longitude and a latitude with each of a plurality of pixels constituting the image information.

3. The information processing device according to claim 1 or 2.

4. When the image information records a rectangular imaging range having a vertical and horizontal direction, the third acquisition unit sets at least two points on at least two lines in the vertical direction of the imaging range, thereby setting at least four points in the image information.

4. The information processing device according to claim 1.

5. When the subject captured by the imaging unit is a moving body, a generation unit is provided that generates a moving path of the moving body based on image information to which latitude and longitude information is added, which is acquired by a second acquisition unit, and a distance from the imaging unit to the subject, which is acquired by the fifth acquisition unit. The information processing device according to any one of claims 1 to 4.

6. and a determination unit that, when the moving object imaged by the imaging unit is temporarily blocked by an obstruction, predicts a destination of the moving object based on the movement path generated by the generation unit and determines whether the moving object before being temporarily blocked by the obstruction is the same as the moving object after being temporarily blocked by the obstruction. The information processing device according to claim 5 .

7. and a sixth acquisition unit that acquires the longitude and latitude of a position of the subject captured by the imaging unit based on image information to which latitude and longitude information is added acquired by the second acquisition unit and a distance from the imaging unit to the subject acquired by the fifth acquisition unit. The information processing device according to any one of claims 1 to 6.

8. The computer an imaging step of imaging an object and generating image information by an imaging unit; a first acquisition step of acquiring map information in which latitude and longitude information relating to longitude and latitude is recorded, the map information including an imaging range in which an object is imaged in the imaging step; a second acquisition step of acquiring image information to which latitude and longitude information is added by associating longitude and latitude within an imaging range in the map information with a predetermined position in the image information based on latitude and longitude information in the map information acquired in the first acquisition step and image information captured in the imaging step; a third acquisition step of acquiring imaging latitude and longitude information relating to the longitude and latitude of the imaging unit based on at least four points in the image information and at least four points in the map information corresponding to the at least four points; a fourth acquisition step of acquiring a distance related to the imaging unit in the image information generated by the imaging step, based on the image information to which the latitude and longitude information is added acquired by the second acquisition step and the imaging latitude and longitude information acquired by the third acquisition step; a fifth acquisition step of acquiring a distance from the imaging unit to a subject based on image information to which latitude and longitude information is added acquired in the second acquisition step and a distance related to the imaging unit acquired in the fourth acquisition step; An information processing method for performing the above.

9. On the computer, An imaging function for capturing an image of a subject and generating image information; a first acquisition function that acquires map information in which latitude and longitude information is recorded, the latitude and longitude information including an imaging range in which an object is imaged by the imaging function; a second acquisition function that acquires image information with latitude and longitude information by associating longitude and latitude within an imaging range in the map information with a predetermined position in the image information based on latitude and longitude information in the map information acquired by the first acquisition function and image information captured by the imaging function; a third acquisition function that acquires imaging latitude and longitude information related to the longitude and latitude of the imaging function based on at least four points in the image information and at least four points in the map information corresponding to the at least four points; a fourth acquisition function that acquires a distance related to the imaging function in image information generated by the imaging function based on image information with latitude and longitude information acquired by the second acquisition function and imaging latitude and longitude information acquired by the third acquisition function; a fifth acquisition function that acquires a distance from the imaging function to a subject based on image information to which latitude and longitude information is added and acquired by the second acquisition function, and a distance related to the imaging function and acquired by the fourth acquisition function; An information processing program that realizes this.

Images