Information processing method, program, and information processing device

Abstract

To associate at least information on three-dimensional point cloud data with two-dimensional image information, to increase the possibility of analysis on the two-dimensional image information.SOLUTION: An information processing method is executed by an information processing device including a processor. The method includes: acquiring, by the processor, a captured image pertaining to a predetermined space; acquiring three-dimensional point cloud data pertaining to the space; setting a condition pertaining to an angle-of-view of the three-dimensional point cloud data; adjusting the angle-of-view of the three-dimensional point cloud data on the basis of the condition; generating a two-dimensional point cloud image from the adjusted three-dimensional point cloud data; and comparing the captured image with the two-dimensional point cloud image to determine the similarity.SELECTED DRAWING: Figure 6

Description

[Technical Field]

[0001] The present invention relates to an information processing method, a program, and an information processing device. [Background technology]

[0002] Conventionally, there are known techniques for generating highly accurate three-dimensional point cloud data relating to spaces and objects. For example, Patent Document 1 discloses a technique for generating highly reliable three-dimensional point cloud data relating to roads. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2020-17200 A Summary of the Invention [Problem to be solved by the invention]

[0004] However, even if highly reliable 3D point cloud data could be generated in the past, its scope of use was limited to map generation, etc. Furthermore, images captured by existing general 2D imaging devices (e.g., 2D cameras) are 2D image information, and in order to obtain information in the depth direction, for example, it was necessary to install a separate depth sensor or change to a 3D imaging device (e.g., 3D camera).

[0005] Therefore, the disclosed technology has been developed in consideration of these circumstances, and aims to provide a technology that associates at least information regarding three-dimensional point cloud data with two-dimensional image information, thereby increasing the analytical feasibility of two-dimensional image information. [Means for solving the problem]

[0006] An information processing method that is one aspect of the disclosed technology is an information processing method executed by an information processing device that includes a processor, in which the processor performs the following operations: acquiring an image of a predetermined space; acquiring three-dimensional point cloud data of the space; setting conditions related to the angle of view of the three-dimensional point cloud data; adjusting the angle of view of the three-dimensional point cloud data based on the conditions; generating a two-dimensional point cloud image from the adjusted three-dimensional point cloud data; and comparing the image of the captured image with the two-dimensional point cloud image to determine similarity. [Effects of the Invention]

[0007] According to the present invention, at least information relating to three-dimensional point cloud data can be associated with two-dimensional image information, thereby increasing the analytical potential of the two-dimensional image information. [Brief explanation of the drawings]

[0008] [Figure 1] 1 is a diagram showing an example of the configuration of an information processing system according to an embodiment of the present invention. [Figure 2] FIG. 1 is a diagram showing an example of the configuration of an information processing apparatus according to an embodiment of the present invention. [Figure 3] FIG. 4 is a diagram showing an example of a captured image. [Figure 4] FIG. 1 is a diagram showing an example of three-dimensional point cloud data. [Figure 5] FIG. 1 is a diagram showing an example of a two-dimensional point cloud image. [Figure 6] 10 is a flowchart showing an example of processing of the information processing apparatus according to the present embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0009] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the accompanying drawings.

[0010] <System Overview> Fig. 1 is a diagram showing an example of the configuration of an information processing system 1 according to an embodiment of the present invention. In the example shown in Fig. 1, the information processing system 1 illustratively includes an information processing device 10, an information processing device 20, a database 30, and a network N. Furthermore, the number of the information processing device 10, the information processing device 20, and the database 30 may be one or more.

[0011] The information processing device 10 is, for example, a server, and acquires captured images of a predetermined space from an imaging device (hereinafter, a camera will be described as an example) (not shown) connected to the information processing device 10 directly or via a network N. The information processing device 10 also transmits the captured images to the information processing device 20 via the network N. As a non-limiting example, the camera may be a surveillance camera installed in a predetermined space, a camera provided in a communication terminal such as a smartphone, or a camera mounted on a mobility such as an in-vehicle camera. The information processing device 10 may also be a camera itself that can communicate with the information processing device 20.

[0012] The information processing device 20 is, for example, a server, and receives captured images relating to a predetermined space from the information processing device 10. The information processing device 20 also acquires three-dimensional point cloud data relating to the predetermined space from a database 30 that stores three-dimensional point cloud data. The information processing device 20 identifies three-dimensional point cloud data corresponding to the captured images based on the acquired captured images and three-dimensional point cloud data. Detailed processing by the information processing device 20 will be described later.

[0013] The network N is realized by, for example, a network such as the Internet or a mobile phone network, a LAN (Local Area Network), or a network that combines these.

[0014] <Configuration of information processing device> 2 is a diagram showing an example of the configuration of an information processing device 20 according to an embodiment of the present invention. The information processing device 20 includes one or more processors (CPU: Central Processing Unit) 210, one or more network communication interfaces 220, a storage device 230, a user interface 240, and one or more communication buses 250 for interconnecting these components. The user interface 240 may be connected via a network.

[0015] Storage device 230 may be, for example, a high-speed random-access memory such as a DRAM, an SRAM, or other random-access solid-state storage device. Storage device 230 may also be a non-volatile memory such as one or more magnetic disk storage devices, optical disk storage devices, flash memory devices, or other non-volatile solid-state storage devices. Storage device 230 may also be a non-transitory computer-readable recording medium.

[0016] Another example of storage device 230 may be one or more storage devices located remotely from processor 210. In some embodiments, storage device 230 stores programs, modules, and data structures, or a subset thereof, that are executed by processor 210.

[0017] The storage device 230 stores data used by the information processing system 1. For example, the storage device 230 stores three-dimensional point cloud data of a predetermined region (e.g., Japan), three-dimensional high-precision map data (also called "HD map") generated using the three-dimensional point cloud data, and may also store captured image data transmitted by the information processing device 10. The storage device 230 may be either a storage device built into the information processing device 20 or an external storage device.

[0018] The processor 210 that executes the processing according to this embodiment will be described. The processor 210 executes a program stored in the storage device 230, thereby controlling the processing executed by the information processing unit 212.

[0019] The information processing unit 212 includes, for example, an acquisition unit 213, a setting unit 214, an adjustment unit 215, a generation unit 216, a determination unit 217, a superimposition unit 218, and an output unit 219.

[0020] The acquiring unit 213 acquires a captured image CI relating to a predetermined space. For example, the acquiring unit 213 acquires a captured image CI transmitted via the network N in response to an instruction from the information processing device 10, such as a captured image CI of a predetermined space captured by a camera, or a captured image CI captured by a camera built into the information processing device 10. Alternatively, the captured images captured by the camera may be sequentially stored in a database, and some of the captured images CI may be transmitted to the information processing device 20 in response to an instruction from the information processing device 10. The acquiring unit 213 may acquire the captured image CI by reading the captured image CI from the storage device 230. Alternatively, the acquiring unit 213 may acquire a captured image CI posted on a social networking service (SNS) or the like via the network N.

[0021] As described above, the captured image CI is an image of a predetermined space captured by, for example, a camera built into the information processing device 10 shown in FIG. 1 or a camera (not shown) connected to the information processing device 10. The predetermined space includes, but is not limited to, highways, public roads, racing circuits, parking lots, runways, tunnels, parks, stadiums, event venues, animal farms, and other building facilities. In the following, in this embodiment, a highway will be described as an example of the predetermined space.

[0022] Fig. 3 is a diagram showing an example of a captured image CI. In the example shown in Fig. 3, the acquisition unit 213 acquires an image of an expressway captured by a surveillance camera installed on the expressway as the captured image CI. The acquisition unit 213 acquires the captured image CI from the information processing device 10, the surveillance camera, or a database that stores the captured image CI, for example, via the network communication interface 220 shown in Fig. 2.

[0023] Furthermore, the acquisition unit 213 acquires three-dimensional point cloud data 3D relating to a space. The acquisition unit 213 acquires the three-dimensional point cloud data 3D relating to a predetermined space, for example, from the database 30 shown in FIG. 1. As a specific example, the acquisition unit 213 may acquire the installation position (latitude, longitude, height, direction, etc.) of the camera that captured the captured image CI, and, assuming that a virtual camera is installed at this installation position, acquire the three-dimensional point cloud data 3D acquired by this virtual camera. The acquisition unit 213 may acquire installation position information set by an external device regarding the installation position of the camera, or may acquire position information acquired by a GPS (Global Positioning System) function provided in a device having the camera.

[0024] Fig. 4 is a diagram showing an example of three-dimensional point cloud data 3D. In the example shown in Fig. 4, the acquisition unit 213 acquires the three-dimensional point cloud data 3D related to the expressway. The three-dimensional point cloud data 3D is generated, for example, by a vehicle equipped with an MMS (Mobile Mapping System) measurement system running on the expressway and performing three-dimensional measurement of the surrounding terrain, and is stored in the database 30. Note that the three-dimensional point cloud data may be generated by any of drone measurement, fixed-point measurement, measurement by aerial LiDAR, and measurement using satellite / SAR images, in addition to measurement by MMS.

[0025] The setting unit 214 sets conditions related to the 3D angle of view of the 3D point cloud data. The setting unit 214 sets the conditions using, for example, one or more angle of view parameters of the 3D point cloud data 3D included in the conditions related to the angle of view. As a non-limiting example, the angle of view parameters include the latitude (unit: degrees) of the camera installation position, the longitude (unit: degrees) of the camera installation position, the height (unit: meters) of the camera installation from the road surface, the horizontal FOV (unit: degrees), the vertical FOV (unit: degrees), and the depression angle θ (unit: degrees) to the road surface.

[0026] The setting unit 214 may set an arbitrary value as the initial value of the angle of view parameter, for example, based on the installation position of the virtual camera. The setting unit 214 may also reset the value of the angle of view parameter based on, for example, the determination result of the determination unit 217, which will be described later.

[0027] The setting unit 214 acquires the size of the captured image CI and sets the drawing size of the three-dimensional point cloud data 3D. If the size of the captured image CI is set in advance, the setting unit 214 sets that size as the drawing size, and if the size of the captured image CI is undetermined, the setting unit 214 specifies the size of the captured image CI to be acquired and sets the specified size as the drawing size of the three-dimensional point cloud data 3D.

[0028] The adjustment unit 215 adjusts the angle of view of the three-dimensional point cloud data 3D based on the conditions (e.g., angle of view parameters) set by the setting unit 214. The adjustment unit 215 adjusts the angle of view of the three-dimensional point cloud data 3D based on, for example, the one or more angle of view parameters set by the setting unit 214.

[0029] More specifically, when the setting unit 214 changes the value of the angle-of-view parameter, the adjustment unit 215 may adjust the angle of view of the three-dimensional point cloud data 3D based on the changed value of the angle-of-view parameter.

[0030] The generation unit 216 generates a two-dimensional point cloud image 2D from the three-dimensional point cloud data 3D adjusted by the adjustment unit 215. The adjusted three-dimensional point cloud data 3D includes three-dimensional point cloud data 3D with a predetermined angle of view. The generation unit 216, for example, uses the adjusted angle of view and rendering size to render the three-dimensional point cloud data 3D, captures the rendered three-dimensional point cloud data 3D, and generates a two-dimensional point cloud image 2D as two-dimensional image data. The captured two-dimensional point cloud image 2D is, for example, image data of the same size as the captured image CI. In the present invention, "capturing" is used to mean, for example, "generating three-dimensional point cloud data with a rendering size rendered at the adjusted angle of view from the installation position of the virtual camera as two-dimensional image data," regardless of whether it is actually displayed on the display unit. The generated two-dimensional point cloud image 2D may be stored in the storage device 230.

[0031] The determination unit 217 compares the captured image CI with the two-dimensional point cloud image 2D to determine the similarity. For example, when the captured image CI and the two-dimensional point cloud image 2D are superimposed, the determination unit 217 determines the similarity based on the degree of agreement between feature points in the captured image CI (for example, any points on the edges of captured objects such as roads and roadside trees) and points in the 2D point cloud image 2D corresponding to the feature points of the captured image CI (for example, the similarity of the feature amounts of the feature points). Furthermore, the determination unit 217 converts the color of a specific captured object such as a road or roadside trees in the captured image CI to the color of a point in the two-dimensional point cloud data (for example, white), and converts the color of a part other than the specific captured object to the color of a part without a point in the two-dimensional point cloud data (for example, black). The determination unit 217 may calculate the square error of the color components (for example, RGB values) between the converted captured image CI and the two-dimensional point cloud image 2D, and determine the similarity based on the sum of the square errors of each pixel. The determination unit 217 may use one or more feature points extracted by a known feature point extraction process for the feature points in the captured image CI, or may use feature points of an object recognized by performing object recognition on the captured image CI. As described above, comparing the captured image CI with the two-dimensional point cloud image 2D includes comparing the feature points of both images.

[0032] The above processing makes it possible to determine the similarity between the captured image CI and three-dimensional point cloud data of different dimensions. As a result of this similarity determination, it becomes possible to identify the three-dimensional point cloud data corresponding to the captured image CI, and it becomes possible to associate at least information related to the three-dimensional point cloud data with the two-dimensional image information. As a result, it becomes possible to increase the analytical potential of the two-dimensional image information.

[0033] Furthermore, the determination unit 217 may determine the similarity by comparing a part of the captured image CI with a part of the two-dimensional point cloud image 2D. For example, the determination unit 217 may determine the similarity by comparing only feature points of objects (e.g., roads in the captured image CI) whose size is larger than a predetermined value among the objects recognized in the captured image CI. This reduces the processing load on the determination unit 217.

[0034] The setting unit 214 may set a condition related to the angle of view of the 3D point cloud data 3D based on the similarity. For example, when the determination unit 217 determines that the similarity between the captured image CI and the 2D point cloud image 2D is low, the setting unit 214 resets the condition related to the angle of view of the 3D point cloud data 3D used to generate the 2D point cloud image 2D (e.g., angle of view parameters). More specifically, the setting unit 214 changes the value of one or more angle of view parameters to reset the angle of view of the 3D point cloud data 3D. The setting unit 214 may also reset the angle of view of the 3D point cloud data 3D until the determination unit 217 determines that the similarity is high. This enables the information processing device 20 to appropriately adjust the angle of view of the 3D point cloud data 3D, and the superimposition unit 218 (described later) to appropriately superimpose specific 3D point cloud data 3D corresponding to the specific 2D point cloud image 2D on the captured image CI. As a result, it becomes possible to perform a precise analysis of the 2D image information.

[0035] 3 and 4, for example, the determination unit 217 compares a two-dimensional point cloud image generated from the three-dimensional point cloud data 3D drawn from the viewpoint (also referred to as a "virtual viewpoint") and angle of view of a virtual camera shown in FIG. 4 with the captured image CI shown in FIG. 3, and determines that the two have low similarity. That is, in the captured image CI shown in FIG. 3, roadside trees are located at the left edge of the image. In contrast, in the three-dimensional point cloud data 3D viewed from the viewpoint (virtual viewpoint) and angle of view of the virtual camera shown in FIG. 4, a road that is not shown in the captured image CI is displayed to the left of the roadside trees. Therefore, for example, the determination unit 217 calculates the squared error between the RGB values of the roads in the captured image CI and the RGB values of the roads in the two-dimensional point cloud image 2D by the above process, and determines that the similarity is low if the sum of the squared errors of each pixel is equal to or greater than a predetermined value.

[0036] Fig. 5 is a diagram showing an example of a two-dimensional point cloud image 2D. The example shown in Fig. 5 is obtained by adjusting the angle of view of the three-dimensional point cloud data 3D shown in Fig. 4 by the adjustment unit 215 and then capturing it by the generation unit 216. In the example shown in Fig. 5, the angle of view of the three-dimensional point cloud data 3D shown in Fig. 4 has been adjusted, and as with the captured image CI shown in Fig. 3, a roadside tree is positioned at the left edge of the image. Therefore, for example, the determination unit 217 calculates the squared error between the RGB values of the roadside tree in the captured image CI and the RGB values of the roadside tree in the two-dimensional point cloud image 2D by the above process, and determines that the similarity is high if the sum of the squared errors of each pixel is less than a predetermined value.

[0037] The adjustment unit 215 may adjust the angle of view parameters by sequentially changing the angle of view parameters one by one. Alternatively, the adjustment unit 215 may calculate a movement vector between one or more feature points of the captured image CI and a feature point of the 2D point cloud image 2D corresponding to the feature points, and adjust the angle of view parameters based on the movement vector. The movement vector is a vector that indicates the distance and direction from a first feature point of the reference image to a second feature point of the 2D point cloud image corresponding to the first feature point, with the captured image CI as the reference image. The latter method makes it possible to quickly search for the 2D point cloud data that is most similar to the captured image CI.

[0038] The superimposing unit 218 superimposes specific three-dimensional point cloud data 3D corresponding to a specific two-dimensional point cloud image 2D with the captured image CI when the similarity satisfies a predetermined condition. The predetermined condition includes, for example, that the two images are most similar to each other or that the sum of the squared errors is less than a threshold value.

[0039] The superimposing unit 218 superimposes, for example, three-dimensional point cloud data 3D corresponding to the two-dimensional point cloud image 2D determined by the determining unit 217 to have high similarity, and the three-dimensional point cloud data 3D of the above-mentioned drawing size, on the captured image CI.

[0040] As a result, the information processing device 20 can superimpose the captured image CI and the three-dimensional point cloud data 3D at an appropriate angle of view, and can appropriately associate at least information related to the three-dimensional point cloud data with the two-dimensional image information. As a result, more precise analysis of the two-dimensional image information becomes possible. For example, since the three-dimensional point cloud data 3D is a collection of points having three-dimensional coordinate values, by superimposing the captured image CI with the three-dimensional point cloud data 3D with the adjusted angle of view, the information processing device 20 can calculate the position, number, size, etc. of vehicles using the distance in the depth direction on the expressway in the captured image CI.

[0041] Furthermore, for example, when a traffic jam occurs on a highway in the captured image CI, the information processing device 20 can calculate the vehicle density within a predetermined distance based on three-dimensional coordinate values (particularly values in the depth direction).

[0042] Furthermore, for example, if there is a fallen object on a highway in the captured image CI, the information processing device 20 can calculate the position of the fallen object based on the three-dimensional coordinate values.

[0043] The similarity includes a similarity between the captured image CI and the two-dimensional point cloud image 2D, and when the determination unit 217 determines that the similarity is equal to or greater than a predetermined threshold, the superimposition unit 218 may superimpose the captured image CI on specific three-dimensional point cloud data 3D. For example, the superimposition unit 218 superimposes the captured image CI on the three-dimensional point cloud data 3D of the drawing size corresponding to the two-dimensional point cloud image 2D whose similarity is determined by the determination unit 217 to be equal to or greater than the predetermined threshold. The similarity may be calculated using a general similarity calculation method such as cosine similarity or Euclidean distance.

[0044] Furthermore, when the determination unit 217 determines that the similarity is less than a predetermined threshold, the setting unit 214 may reset a condition related to the angle of view of the three-dimensional point cloud data 3D using one or more angle of view parameters. For example, when the determination unit 217 determines that the similarity between the captured image CI and the two-dimensional point cloud image 2D is less than a predetermined threshold, the setting unit 214 resets a condition related to the angle of view of the three-dimensional point cloud data 3D used when generating the two-dimensional point cloud image 2D (for example, angle of view parameters). More specifically, the setting unit 214 changes the values of one or more angle of view parameters to reset the angle of view of the three-dimensional point cloud data 3D. Furthermore, the setting unit 214 may reset the angle of view of the three-dimensional point cloud data 3D until the determination unit 217 determines that the similarity is equal to or greater than a predetermined threshold.

[0045] As a result, the information processing device 20 can automatically and appropriately adjust the angle of view of the three-dimensional point cloud data 3D.

[0046] The superimposing unit 218 may acquire specific 3D map data based on specific 3D point cloud data, and superimpose the captured image CI on the specific 3D map data. For example, the acquiring unit 213 may acquire 3D map data of the drawing size corresponding to the 3D point cloud data with an adjusted angle of view from the database 30, and the superimposing unit 218 may superimpose the captured image CI on the 3D map data.

[0047] Furthermore, the 3D map data used in this embodiment may be, for example, high-precision 3D map data used for autonomous driving, etc. As a specific example, this map data is map data called a dynamic map that is provided in real time and to which more dynamic information such as information on surrounding vehicles and traffic information is added.

[0048] The three-dimensional map data used in this embodiment is classified into, for example, static information SI1, semi-static information SI2, semi-dynamic information MI1, and dynamic information MI2.

[0049] The static information SI1 is high-precision three-dimensional basic map data (high-precision three-dimensional map data) and includes road surface information, lane information, three-dimensional structures, etc., and is composed of three-dimensional position coordinates and linear vector data indicating features. The quasi-static information SI2, quasi-dynamic information MI1, and dynamic information MI2 are dynamic data that change from moment to moment, and are data that are superimposed on the static information based on position information. The information includes static information SI1, quasi-static information SI2, quasi-dynamic information MI1, and dynamic information MI2, and each piece of information is associated with each other.

[0050] The quasi-static information SI2 includes traffic regulation information, road construction information, wide-area weather information, etc. The quasi-dynamic information MI1 includes accident information, congestion information, narrow-area weather information, etc. The dynamic information MI2 includes ITS (Intelligent Transport System) information, including information on nearby vehicles, pedestrians, traffic lights, etc.

[0051] The 3D map data in this embodiment may also include 3D map data generated from satellite images. For example, high-precision map data is generated by correcting satellite images, and this embodiment can also be applied to this 3D map data.

[0052] As a result, the information processing device 20 can superimpose the captured image CI on the 3D map data at an appropriate angle of view, and can analyze the objects shown in the captured image CI based on the information contained in the 3D map data. This further enhances the analyzability of the 2D image information.

[0053] For example, when a traffic jam occurs on a highway in the captured image CI, the information processing device 20 can analyze whether the traffic jam is heading in an uphill direction or an downhill direction based on the static information SI1. Also, for example, the information processing device 20 can analyze the cause of the traffic jam based on the semi-dynamic information MI1.

[0054] Furthermore, for example, when construction work is being carried out on an expressway in the captured image CI, the information processing device 20 can analyze the location where the construction work is being carried out based on the quasi-static information SI2.

[0055] The output unit 219 outputs data in which the captured image CI and the three-dimensional point cloud data 3D with the adjusted angle of view are superimposed. For example, the output unit 219 displays the data in which the captured image CI and the three-dimensional point cloud data 3D with the adjusted angle of view superimposed on a screen provided in the information processing device 20. For example, the output unit 219 may transmit the data in which the captured image CI and the three-dimensional point cloud data 3D with the adjusted angle of view superimposed on an external information processing device.

[0056] Furthermore, the output unit 219 may transmit, for example, only the three-dimensional point cloud data 3D whose angle of view has been adjusted to the information processing device 10. In this case, the information processing device 10 may superimpose the captured image CI transmitted to the information processing device 20 and the three-dimensional point cloud data 3D whose angle of view has been adjusted.

[0057] Furthermore, the information processing device 10, which has received the three-dimensional point cloud data 3D with the adjusted angle of view, may superimpose the three-dimensional point cloud data 3D with image data captured at a different time and with the same angle of view and at the same position as the captured image CI shown in Fig. 3. This allows the information processing device 10 to calculate, for example, the positions, number, and sizes of vehicles in the captured image captured at a different time from the captured image CI. Furthermore, the output unit 219 may output three-dimensional map data, particularly high-precision three-dimensional map data (HD map), instead of the three-dimensional point cloud data.

[0058] <Operation> Next, the operation according to this embodiment will be described. Fig. 6 is a flowchart showing an example of the processing of the information processing device 20 according to this embodiment.

[0059] In step S11, the acquisition unit 213 of the information processing device 20 acquires a captured image CI relating to a predetermined space.

[0060] In step S12, the acquisition unit 213 of the information processing device 20 acquires three-dimensional point cloud data 3D relating to space. For example, three-dimensional point cloud data including a scene similar to the captured image CI is acquired.

[0061] In step S13, the setting unit 214 of the information processing device 20 sets a condition related to the angle of view of the three-dimensional point cloud data 3D. The condition related to the angle of view includes one or more angle of view parameters.

[0062] In step S14, the adjustment unit 215 of the information processing device 20 adjusts the angle of view of the three-dimensional point cloud data 3D based on the conditions set by the setting unit 214.

[0063] In step S15, the generation unit 216 of the information processing device 20 generates a two-dimensional point cloud image 2D from the adjusted three-dimensional point cloud data 3D.

[0064] In step S16, the determination unit 217 of the information processing device 20 compares the captured image CI with the two-dimensional point cloud image 2D to determine the similarity.

[0065] In step S16, if the determination unit 217 determines that the similarity does not satisfy the condition (for example, the similarity is equal to or greater than a threshold), the process returns to step S13, and the setting unit 214 changes the condition regarding the angle of view based on the determined similarity. On the other hand, if the determination unit 217 determines that the similarity satisfies the condition, the process proceeds to step S17.

[0066] In step S17, the superimposing unit 218 of the information processing device 20 superimposes the captured image CI with specific three-dimensional point cloud data 3D corresponding to the specific two-dimensional point cloud image 2D when the similarity satisfies a predetermined condition.

[0067] In step S18, the output unit 219 of the information processing device 20 outputs data in which the captured image CI and the adjusted three-dimensional point cloud data 3D are superimposed.

[0068] <Other embodiments> The above-described embodiments are provided to facilitate understanding of the present invention and are not to be construed as limiting the present invention. The present invention may be modified or improved without departing from the spirit thereof, and equivalents thereof are also included in the present invention. Furthermore, the present invention can be formed into various disclosures by appropriately combining multiple components disclosed in the above-described embodiments. For example, some components may be deleted from all the components shown in the embodiments. Furthermore, components may be appropriately combined in different embodiments.

[0069] In the above embodiment, the "predetermined space" has been described using an expressway as an example, but is not limited to this. Examples of the "predetermined space" include, but are not limited to, general roads, circuit roads, parking lots, runways, tunnels, parks, stadiums, event venues, animal farms, and other building facilities.

[0070] For example, if the "predetermined space" is a parking lot, the information processing device 10 or the information processing device 20 may output data in which a captured image CI of the parking lot is superimposed on 3D point cloud data with an adjusted angle of view, through the above-described processes. The captured image CI may also include a parked vehicle and an empty parking space, and the information processing device 10 or the information processing device 20 may calculate the size of the parking space. For example, the information processing device 10 or the information processing device 20 may transmit the size and location of an empty parking space to a communication terminal or the like carried by a driver who is about to park a vehicle. This allows the driver to easily grasp the location of a parking space large enough for parking.

[0071] Furthermore, when the driver's vehicle is traveling in a place where GPS cannot be used, such as a parking lot, the information processing device 10 or 20 may calculate the position of the driver's vehicle based on data obtained by superimposing an image of the parking lot taken by the driver's communication terminal or the like on 3D point cloud data with an adjusted angle of view. This allows the driver to know the position of his or her vehicle even in a parking lot where GPS cannot be used, and makes it possible to provide parking assistance.

[0072] Furthermore, the information processing unit 212 may perform processing to identify the vehicle position using the above-described processing. For example, when the acquisition unit 213 acquires vehicle position information from a Global Navigation Satellite System (GNSS) or a Global Positioning System (GPS), the information processing unit 212 identifies three-dimensional point cloud data 3D corresponding to the vehicle position information based on the vehicle position information. For example, the information processing unit 212 identifies three-dimensional point cloud data 3D of a predetermined range that includes the GPS or GNSS position information.

[0073] The acquisition unit 213 also acquires the captured image CI from an imaging device of the vehicle (such as a drive recorder or a mobile terminal of the driver). At this time, the information processing unit 212 can accurately identify the vehicle's position information using the captured image CI and the three-dimensional point cloud data 3D identified based on the position information of the GPS or GNSS.

[0074] For example, the setting unit 214 can set the three-dimensional point cloud data 3D of the periphery of the vehicle by using the position information of the GPS or GNSS as the installation position of the virtual camera described above.

[0075] By performing the above-described processing, the adjustment unit 215, the generation unit 216, and the determination unit 217 can determine the similarity between the captured image CI and the two-dimensional point cloud image 2D and identify, for example, the two-dimensional point cloud image 2D with the highest similarity. The information processing unit 212 may identify the position where the captured image CI was captured, i.e., the position of the vehicle, using the two-dimensional point cloud image 2D with the highest similarity or the three-dimensional point cloud image 3D corresponding to this two-dimensional point cloud image 2D.

[0076] By the above processing, in cases where the accuracy of GPS or GNSS position information is not high, it becomes possible to appropriately identify the imaging position of the captured image CI (e.g., the position of the vehicle) by using a 3D point cloud image 3D based on GPS or GNSS position information and the captured image CI to align the 2D point cloud image generated by the 3D point cloud image 3D with the captured image CI.

[0077] Furthermore, if the driver's vehicle and another vehicle (hereinafter referred to as the "other vehicle") are involved in an accident in a location where GPS cannot be used, the position of the other vehicle may be calculated based on data (hereinafter referred to as "superimposed data") obtained by superimposing an image of the driving location captured by the driver's communication terminal or the like (e.g., a drive recorder) on a two-dimensional point cloud image generated based on three-dimensional point cloud data with an adjusted angle of view. The information processing device 20 may include a calculation unit that performs this calculation process. The calculation unit may calculate the position of the other vehicle included in the image of the driving location captured by the driver's communication terminal or the like (e.g., a drive recorder) using, for example, coordinate values of points included in the two-dimensional point cloud image. This allows the driver to determine the position of the other vehicle even if the driver is involved in an accident with the other vehicle in a location where GPS cannot be used, for example.

[0078] The calculation unit may also calculate the position of the other vehicle based on the superimposed data for each frame, and calculate the speed of the other vehicle at the time of the accident (which may include immediately before or immediately after the accident) based on the calculated position. The calculation unit may calculate the speed of the other vehicle based on, for example, the difference between the position of the other vehicle in a predetermined frame and the position of the other vehicle in the immediately preceding (or immediately following) frame.

[0079] For example, if the "predetermined space" is a stadium, the information processing device 10 or the information processing device 20 may output data in which a captured image CI of the stadium is superimposed on 3D point cloud data through the above processes. The captured image CI may also include players in the stadium and equipment used in the sport. This allows the information processing device 10 or the information processing device 20 to calculate the positional relationship between the players and the ball in the captured image CI, for example, and facilitates analysis of the strategy of the match.

[0080] For example, if the "predetermined space" is a livestock farm, the information processing device 10 or the information processing device 20 may output data in which a captured image CI of the livestock farm is superimposed on 3D point cloud data through the above-described processes. The captured image CI may also include one or more animals in the livestock farm. This allows the information processing device 10 or the information processing device 20 to calculate, for example, the distance traveled by a livestock animal and its positional relationship with other animals, thereby making it possible to easily analyze the behavior of the animals. [Explanation of symbols]

[0081] 1...information processing system, 10...information processing device, 20...information processing device, 30...database, 210...processor, 212...information processing unit, 213...acquisition unit, 214...setting unit, 215...adjustment unit, 216...generation unit, 217...determination unit, 218...superposition unit, 219...output unit, 220...network communication interface, 230...storage device, 240...user interface, 250...communication bus, CI...captured image, 2D...two-dimensional point cloud image, 3D...three-dimensional point cloud data

Claims

1. An information processing method executed by an information processing device including a processor, the processor: Acquiring a captured image of a predetermined space; acquiring three-dimensional point cloud data relating to the space; setting a condition regarding the angle of view of the three-dimensional point cloud data; adjusting the angle of view of the three-dimensional point cloud data based on the conditions; generating a two-dimensional point cloud image from the adjusted three-dimensional point cloud data; and comparing the captured image with the two-dimensional point cloud image to determine similarity.

2. The information processing method according to claim 1 , wherein the setting includes changing the condition based on the similarity.

3. The information processing method according to claim 1 , wherein the processor further executes superimposing specific three-dimensional point cloud data corresponding to a specific two-dimensional point cloud image when the similarity satisfies a predetermined condition on the captured image.

4. the similarity includes a degree of similarity between the captured image and the two-dimensional point cloud image, The information processing method according to claim 3 , wherein, when the similarity is equal to or greater than a predetermined threshold, the superimposing includes superimposing the captured image and the specific three-dimensional point cloud data.

5. The superimposing step comprises: acquiring specific three-dimensional map data based on the specific three-dimensional point cloud data; The information processing method according to claim 3 , further comprising superimposing the captured image and the specific three-dimensional map data.

6. The information processing method according to claim 1 , wherein the processor further executes specifying an imaging position of the captured image based on the two-dimensional point cloud image for which the similarity is determined to satisfy a predetermined condition.

7. A program to be executed by an information processing device including a processor, the processor: Acquiring a captured image of a predetermined space; acquiring three-dimensional point cloud data relating to the space; setting a condition regarding the angle of view of the three-dimensional point cloud data; adjusting the angle of view of the three-dimensional point cloud data based on the conditions; generating a two-dimensional point cloud image from the adjusted three-dimensional point cloud data; A program that executes the process of comparing the captured image with the two-dimensional point cloud image to determine similarity.

8. An information processing device including a processor, the processor: Acquiring a captured image of a predetermined space; acquiring three-dimensional point cloud data relating to the space; setting a condition regarding the angle of view of the three-dimensional point cloud data; adjusting the angle of view of the three-dimensional point cloud data based on the conditions; generating a two-dimensional point cloud image from the adjusted three-dimensional point cloud data; An information processing device that executes the step of comparing the captured image with the two-dimensional point cloud image to determine similarity.

Images