Information processing method, program, and information processing device

Abstract

Provided is an information processing method executed by an information processing device including a processor, wherein the processor executes: acquiring a captured image pertaining to a prescribed space; acquiring three-dimensional point group data pertaining to the space; setting a condition pertaining to an angle-of-view of the three-dimensional point group data; adjusting the angle-of-view of the three-dimensional point group data on the basis of the condition; generating a two-dimensional point group image from the adjusted three-dimensional point group data; and comparing the captured image and the two-dimensional point group image to determine the similarity.

Description

Information processing method, program, and information processing device

[0001] The present invention relates to an information processing method, a program, and an information processing device.

[0002] Conventionally, there are known techniques for generating highly accurate three-dimensional point cloud data relating to spaces or objects. For example, Patent Literature 1 discloses a technique for generating highly reliable three-dimensional point cloud data relating to roads.

[0003] Japanese Patent Application Laid-Open No. 2020-17200

[0004] However, in the past, even if highly reliable 3D point cloud data could be generated, its scope of use was limited to map generation, etc. Furthermore, images captured by existing general 2D imaging devices (e.g., 2D cameras) are 2D image information, and in order to obtain information in the depth direction, for example, it was necessary to provide a separate depth sensor or change to a 3D imaging device (e.g., 3D camera).

[0005] Therefore, the disclosed technology has been developed in consideration of these circumstances, and aims to provide a technology that associates at least information regarding three-dimensional point cloud data with two-dimensional image information, thereby increasing the analytical feasibility of the two-dimensional image information.

[0006] An information processing method that is one aspect of the disclosed technology is an information processing method executed by an information processing device that includes a processor, in which the processor acquires an image of a predetermined space, acquires three-dimensional point cloud data of the space, sets conditions related to the angle of view of the three-dimensional point cloud data, adjusts the angle of view of the three-dimensional point cloud data based on the conditions, generates a two-dimensional point cloud image from the adjusted three-dimensional point cloud data, and compares the image and the two-dimensional point cloud image to determine similarity.

[0007] According to the present invention, at least information relating to three-dimensional point cloud data can be associated with two-dimensional image information, thereby increasing the analytical potential of the two-dimensional image information.

[0008] A diagram showing an example of the configuration of an information processing system according to an embodiment of the present invention. A diagram showing an example of the configuration of an information processing device according to an embodiment of the present invention. A diagram showing an example of a captured image. A diagram showing an example of three-dimensional point cloud data. A diagram showing an example of a two-dimensional point cloud image. A flowchart showing an example of processing by the information processing device according to the present embodiment.

[0009] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the accompanying drawings.

[0010] <System Overview> Fig. 1 is a diagram showing an example of the configuration of an information processing system 1 according to an embodiment of the present invention. In the example shown in Fig. 1, the information processing system 1 illustratively includes an information processing device 10, an information processing device 20, a database 30, and a network N. Furthermore, the number of the information processing device 10, the information processing device 20, and the database 30 may be one or more.

[0011] The information processing device 10 is, for example, a server, and acquires captured images of a predetermined space from an imaging device (hereinafter, a camera will be described as an example) (not shown) connected to the information processing device 10 directly or via a network N. The information processing device 10 also transmits the captured images to the information processing device 20 via the network N. As a non-limiting example, the camera may be a surveillance camera installed in a predetermined space, a camera provided on a communication terminal such as a smartphone, or a camera mounted on a mobility device such as an in-vehicle camera. The information processing device 10 may also be a camera itself that can communicate with the information processing device 20.

[0012] The information processing device 20 is, for example, a server, and receives captured images of a predetermined space from the information processing device 10. The information processing device 20 also acquires three-dimensional point cloud data of the predetermined space from a database 30 that stores three-dimensional point cloud data. The information processing device 20 identifies three-dimensional point cloud data corresponding to the captured images based on the acquired captured images and three-dimensional point cloud data. Detailed processing by the information processing device 20 will be described later.

[0013] The network N is realized by, for example, a network such as the Internet or a mobile phone network, a LAN (Local Area Network), or a network that combines these.

[0014] 2 is a diagram showing an example of the configuration of an information processing device 20 according to an embodiment of the present invention. The information processing device 20 includes one or more processors (CPU: Central Processing Unit) 210, one or more network communication interfaces 220, a storage device 230, a user interface 240, and one or more communication buses 250 for interconnecting these components. The user interface 240 may be connected via a network.

[0015] The storage device 230 may be, for example, a high-speed random-access memory such as a DRAM, an SRAM, or other random-access solid-state storage device. The storage device 230 may also be a non-volatile memory such as one or more magnetic disk storage devices, optical disk storage devices, flash memory devices, or other non-volatile solid-state storage devices. The storage device 230 may also be a non-transitory computer-readable recording medium.

[0016] Another example of storage device 230 may be one or more storage devices located remotely from processor 210. In some embodiments, storage device 230 stores programs, modules, and data structures, or a subset thereof, that are executed by processor 210.

[0017] The storage device 230 stores data used by the information processing system 1. For example, the storage device 230 may store three-dimensional point cloud data of a predetermined region (e.g., Japan), three-dimensional high-precision map data (also referred to as an "HD map") generated using the three-dimensional point cloud data, and may also store captured image data transmitted by the information processing device 10. The storage device 230 may be either a storage device built into the information processing device 20 or an external storage device.

[0018] The processor 210 that executes the processing according to this embodiment will be described. The processor 210 controls the processing executed by the information processing unit 212 by executing a program stored in the storage device 230.

[0019] The information processing unit 212 includes, for example, an acquisition unit 213 , a setting unit 214 , an adjustment unit 215 , a generation unit 216 , a determination unit 217 , a superposition unit 218 , and an output unit 219 .

[0020] The acquisition unit 213 acquires captured images CI relating to a predetermined space. For example, the acquisition unit 213 acquires captured images CI transmitted via the network N in response to an instruction from the information processing device 10, such as captured images CI of a predetermined space captured by a camera, or captured images CI captured by a camera built into the information processing device 10. Alternatively, the captured images captured by the camera may be sequentially stored in a database, and some of the captured images CI may be transmitted to the information processing device 20 in response to an instruction from the information processing device 10. The acquisition unit 213 may acquire the captured images CI by reading the captured images CI from the storage device 230. Alternatively, the acquisition unit 213 may acquire captured images CI posted on a social networking service (SNS) or the like via the network N.

[0021] As described above, the captured image CI is an image of a predetermined space captured by, for example, a camera built into the information processing device 10 shown in FIG. 1 or a camera (not shown) connected to the information processing device 10. The predetermined space includes, but is not limited to, highways, public roads, racing circuits, parking lots, runways, tunnels, parks, stadiums, event venues, animal farms, and other building facilities. In the following description of the present embodiment, a highway will be used as an example of the predetermined space.

[0022] Fig. 3 is a diagram showing an example of a captured image CI. In the example shown in Fig. 3, the acquisition unit 213 acquires an image of a highway captured by a surveillance camera installed on the highway as the captured image CI. The acquisition unit 213 acquires the captured image CI from the information processing device 10, the surveillance camera, or a database that stores the captured image CI, for example, via the network communication interface 220 shown in Fig. 2.

[0023] The acquisition unit 213 also acquires three-dimensional point cloud data 3D relating to a space. The acquisition unit 213 acquires the three-dimensional point cloud data 3D relating to a predetermined space, for example, from the database 30 shown in FIG. 1 . As a specific example, the acquisition unit 213 may acquire the installation position (latitude, longitude, height, direction, etc.) of the camera that captured the captured image CI, and, assuming that a virtual camera is installed at this installation position, acquire the three-dimensional point cloud data 3D acquired by this virtual camera. The acquisition unit 213 may acquire installation position information regarding the installation position of the camera set by an external device, or may acquire position information acquired by a GPS (Global Positioning System) function provided in a device having the camera.

[0024] FIG. 4 is a diagram showing an example of three-dimensional point cloud data 3D. In the example shown in FIG. 4, the acquisition unit 213 acquires the three-dimensional point cloud data 3D related to the expressway. The three-dimensional point cloud data 3D is generated, for example, by a vehicle equipped with an MMS (Mobile Mapping System) measurement system running on the expressway and performing three-dimensional measurements of the surrounding terrain, and is stored in the database 30. Note that the three-dimensional point cloud data may be generated by any of drone measurements, fixed-point measurements, aerial LiDAR measurements, and measurements using satellite / SAR images, in addition to measurements using an MMS.

[0025] The setting unit 214 sets conditions related to the 3D angle of view of the 3D point cloud data. The setting unit 214 sets the conditions using, for example, one or more angle of view parameters of the 3D point cloud data 3D included in the conditions related to the angle of view. As a non-limiting example, the angle of view parameters include the latitude (unit: degrees) of the camera installation position, the longitude (unit: degrees) of the camera installation position, the height (unit: meters) of the camera installation from the road surface, the horizontal FOV (unit: degrees), the vertical FOV (unit: degrees), and the depression angle θ (unit: degrees) to the road surface.

[0026] The setting unit 214 may set an arbitrary value as the initial value of the angle of view parameter, for example, based on the installation position of the virtual camera. The setting unit 214 may also reset the value of the angle of view parameter based on, for example, a determination result by the determination unit 217 (described later).

[0027] The setting unit 214 acquires the size of the captured image CI and sets the drawing size of the three-dimensional point cloud data 3D. If the size of the captured image CI is set in advance, the setting unit 214 sets that size as the drawing size, or if the size of the captured image CI is undetermined, the setting unit 214 specifies the size of the captured image CI to be acquired and sets the specified size as the drawing size of the three-dimensional point cloud data 3D.

[0028] The adjustment unit 215 adjusts the angle of view of the three-dimensional point cloud data 3D based on the conditions (e.g., angle of view parameters) set by the setting unit 214. The adjustment unit 215 adjusts the angle of view of the three-dimensional point cloud data 3D based on, for example, the one or more angle of view parameters set by the setting unit 214.

[0029] More specifically, when the setting unit 214 changes the value of the angle-of-view parameter, the adjustment unit 215 may adjust the angle of view of the three-dimensional point cloud data 3D based on the changed value of the angle-of-view parameter.

[0030] The generation unit 216 generates a two-dimensional point cloud image 2D from the three-dimensional point cloud data 3D adjusted by the adjustment unit 215. The adjusted three-dimensional point cloud data 3D includes three-dimensional point cloud data 3D with a predetermined angle of view. The generation unit 216, for example, uses the adjusted angle of view and drawing size to render the three-dimensional point cloud data 3D, captures the rendered three-dimensional point cloud data 3D, and generates a two-dimensional point cloud image 2D as two-dimensional image data. The captured two-dimensional point cloud image 2D is, for example, image data of the same size as the captured image CI. In the present invention, "capturing" is used to mean, for example, "generating three-dimensional point cloud data with a drawing size rendered at the adjusted angle of view from the installation position of the virtual camera as two-dimensional image data," regardless of whether it is actually displayed on the display unit. The generated two-dimensional point cloud image 2D may be stored in the storage device 230.

[0031] The determination unit 217 compares the captured image CI with the two-dimensional point cloud image 2D to determine the similarity. For example, when the captured image CI and the two-dimensional point cloud image 2D are superimposed, the determination unit 217 determines the similarity based on the degree to which feature points in the captured image CI (e.g., any points on the edges of captured objects such as roads and roadside trees) match points in the two-dimensional point cloud image 2D corresponding to the feature points in the captured image CI (e.g., the similarity of the feature amounts of the feature points). Furthermore, the determination unit 217 converts the color of a specific captured object, such as a road or roadside trees, in the captured image CI to the color of a point in the two-dimensional point cloud data (e.g., white), and converts colors other than the specific captured object to the color of a location where no point exists in the two-dimensional point cloud data (e.g., black). The determination unit 217 may calculate the squared error of the color components (e.g., RGB values) between the converted captured image CI and the two-dimensional point cloud image 2D, and determine the similarity based on the sum of the squared errors for each pixel. The determination unit 217 may use, as the feature points in the captured image CI, one or more feature points extracted by a known feature point extraction process, or may use feature points of an object recognized by performing object recognition on the captured image CI. As described above, comparing the captured image CI with the two-dimensional point cloud image 2D includes comparing the two images using their respective feature points.

[0032] The above process makes it possible to determine the similarity between the captured image CI and three-dimensional point cloud data of different dimensions. As a result of this similarity determination, it becomes possible to identify the three-dimensional point cloud data corresponding to the captured image CI, and it becomes possible to associate at least information related to the three-dimensional point cloud data with the two-dimensional image information. As a result, it becomes possible to improve the analytical feasibility of the two-dimensional image information.

[0033] The determination unit 217 may also determine the similarity by comparing a part of the captured image CI with a part of the two-dimensional point cloud image 2D. For example, the determination unit 217 may determine the similarity by comparing only feature points of objects recognized in the captured image CI whose size is larger than a predetermined value (e.g., a road in the captured image CI). This reduces the processing load on the determination unit 217.

[0034] The setting unit 214 may set a condition related to the angle of view of the 3D point cloud data 3D based on the similarity. For example, when the determination unit 217 determines that the similarity between the captured image CI and the 2D point cloud image 2D is low, the setting unit 214 resets the condition related to the angle of view of the 3D point cloud data 3D used to generate the 2D point cloud image 2D (e.g., angle of view parameters). More specifically, the setting unit 214 changes the values ​​of one or more angle of view parameters to reset the angle of view of the 3D point cloud data 3D. The setting unit 214 may also reset the angle of view of the 3D point cloud data 3D until the determination unit 217 determines that the similarity is high. This enables the information processing device 20 to appropriately adjust the angle of view of the 3D point cloud data 3D, allowing the superimposition unit 218 (described later) to appropriately superimpose specific 3D point cloud data 3D corresponding to the specific 2D point cloud image 2D on the captured image CI. As a result, it becomes possible to perform a detailed analysis of the two-dimensional image information.

[0035] 3 and 4, for example, the determination unit 217 compares a two-dimensional point cloud image generated from the three-dimensional point cloud data 3D drawn from the viewpoint (also referred to as a "virtual viewpoint") and angle of view of a virtual camera shown in FIG. 4 with the captured image CI shown in FIG. 3, and determines that the two images have low similarity. That is, in the captured image CI shown in FIG. 3, road trees are located at the left edge of the image. In contrast, in the three-dimensional point cloud data 3D viewed from the viewpoint (virtual viewpoint) and angle of view of the virtual camera shown in FIG. 4, a road not shown in the captured image CI is displayed to the left of the road trees. Therefore, for example, the determination unit 217 calculates the squared error between the RGB values ​​of the road trees in the captured image CI and the RGB values ​​of the road in the two-dimensional point cloud image 2D through the above processing, and determines that the similarity is low if the sum of the squared errors for each pixel is equal to or greater than a predetermined value.

[0036] Fig. 5 is a diagram showing an example of a two-dimensional point cloud image 2D. The example shown in Fig. 5 is obtained by adjusting the angle of view of the three-dimensional point cloud data 3D shown in Fig. 4 by the adjustment unit 215 and then capturing it by the generation unit 216. In the example shown in Fig. 5, the angle of view of the three-dimensional point cloud data 3D shown in Fig. 4 has been adjusted, and as with the captured image CI shown in Fig. 3, a roadside tree is positioned at the left edge of the image. Therefore, for example, the determination unit 217 calculates the squared error between the RGB values ​​of the roadside tree in the captured image CI and the RGB values ​​of the roadside tree in the two-dimensional point cloud image 2D through the above processing, and determines that the similarity is high if the sum of the squared errors for each pixel is less than a predetermined value.

[0037] The adjustment unit 215 may adjust the angle of view parameters one by one. Alternatively, the adjustment unit 215 may calculate a motion vector between one or more feature points of the captured image CI and a corresponding feature point of the 2D point cloud image 2D, and adjust the angle of view parameters based on the motion vector. The motion vector is a vector that indicates the distance and direction from a first feature point of the captured image CI as a reference image to a second feature point of the 2D point cloud image corresponding to the first feature point. The latter method makes it possible to quickly search for the 2D point cloud data that is most similar to the captured image CI.

[0038] The superimposing unit 218 superimposes specific three-dimensional point cloud data 3D corresponding to a specific two-dimensional point cloud image 2D with the captured image CI when the similarity satisfies a predetermined condition. The predetermined condition includes, for example, that the two images are most similar to each other or that the sum of the squared errors is less than a threshold value.

[0039] The superimposition unit 218 superimposes, for example, three-dimensional point cloud data 3D corresponding to the two-dimensional point cloud image 2D determined by the determination unit 217 to have high similarity, and the three-dimensional point cloud data 3D of the above-mentioned drawing size, on the captured image CI.

[0040] As a result, the information processing device 20 can superimpose the captured image CI and the three-dimensional point cloud data 3D at an appropriate angle of view, and can appropriately associate at least information related to the three-dimensional point cloud data with the two-dimensional image information. As a result, more precise analysis of the two-dimensional image information can be performed. For example, because the three-dimensional point cloud data 3D is a collection of points having three-dimensional coordinate values, by superimposing the captured image CI with the three-dimensional point cloud data 3D with the adjusted angle of view, the information processing device 20 can calculate the position, number, size, etc. of vehicles using the distance in the depth direction on the expressway in the captured image CI.

[0041] Furthermore, for example, when a traffic jam occurs on a highway in the captured image CI, the information processing device 20 can calculate the vehicle density within a predetermined distance based on three-dimensional coordinate values ​​(particularly values ​​in the depth direction).

[0042] Furthermore, for example, if there is a fallen object on a highway in the captured image CI, the information processing device 20 can calculate the position of the fallen object based on three-dimensional coordinate values.

[0043] The similarity includes a similarity between the captured image CI and the two-dimensional point cloud image 2D, and when the determination unit 217 determines that the similarity is equal to or greater than a predetermined threshold, the superimposition unit 218 may superimpose specific three-dimensional point cloud data 3D on the captured image CI. For example, the superimposition unit 218 superimposes the captured image CI on the three-dimensional point cloud data 3D of the drawing size corresponding to the two-dimensional point cloud image 2D whose similarity is determined by the determination unit 217 to be equal to or greater than the predetermined threshold. The similarity may be calculated using a general similarity calculation method such as cosine similarity or Euclidean distance.

[0044] Furthermore, when the determination unit 217 determines that the similarity is less than a predetermined threshold, the setting unit 214 may reset the condition related to the angle of view of the 3D point cloud data 3D using one or more angle of view parameters. For example, when the determination unit 217 determines that the similarity between the captured image CI and the 2D point cloud image 2D is less than a predetermined threshold, the setting unit 214 resets the condition related to the angle of view of the 3D point cloud data 3D used when generating the 2D point cloud image 2D (e.g., angle of view parameters). More specifically, the setting unit 214 changes the values ​​of one or more angle of view parameters to reset the angle of view of the 3D point cloud data 3D. Furthermore, the setting unit 214 may reset the angle of view of the 3D point cloud data 3D until the determination unit 217 determines that the similarity is equal to or greater than a predetermined threshold.

[0045] As a result, the information processing device 20 can automatically and appropriately adjust the angle of view of the three-dimensional point cloud data 3D.

[0046] The superimposing unit 218 may acquire specific 3D map data based on the specific 3D point cloud data, and superimpose the captured image CI on the specific 3D map data. For example, the acquiring unit 213 may acquire 3D map data of the drawing size corresponding to the 3D point cloud data with the adjusted angle of view from the database 30, and the superimposing unit 218 may superimpose the captured image CI on the 3D map data.

[0047] Furthermore, the 3D map data used in this embodiment may be, for example, high-precision 3D map data used for automated driving, etc. As a specific example, this map data is map data called a dynamic map that is provided in real time and to which more dynamic information, such as information on surrounding vehicles and traffic information, is added.

[0048] The three-dimensional map data used in this embodiment is classified into, for example, static information SI1, semi-static information SI2, semi-dynamic information MI1, and dynamic information MI2.

[0049] The static information SI1 is high-precision three-dimensional basic map data (high-precision three-dimensional map data) and includes road surface information, lane information, three-dimensional structures, etc., and is composed of three-dimensional position coordinates and linear vector data indicating features. The quasi-static information SI2, quasi-dynamic information MI1, and dynamic information MI2 are dynamic data that change from moment to moment, and are data that are superimposed on the static information based on position information. The static information SI1, quasi-static information SI2, quasi-dynamic information MI1, and dynamic information MI2 are included, and each piece of information is associated with each other.

[0050] The quasi-static information SI2 includes traffic regulation information, road construction information, wide-area weather information, etc. The quasi-dynamic information MI1 includes accident information, congestion information, narrow-area weather information, etc. The dynamic information MI2 includes ITS (Intelligent Transport System) information, including information on nearby vehicles, pedestrians, traffic lights, etc.

[0051] The three-dimensional map data in this embodiment may also include three-dimensional map data generated from satellite images. For example, high-precision map data is generated by correcting satellite images, and this embodiment can also be applied to this three-dimensional map data.

[0052] As a result, the information processing device 20 can superimpose the captured image CI on the 3D map data at an appropriate angle of view, and can analyze the objects shown in the captured image CI based on the information contained in the 3D map data, thereby further increasing the possibility of analyzing 2D image information.

[0053] For example, when a traffic jam occurs on a highway in the captured image CI, the information processing device 20 can analyze whether the traffic jam is heading in an uphill direction or an downhill direction based on the static information SI1. Also, for example, the information processing device 20 can analyze the cause of the traffic jam based on the semi-dynamic information MI1.

[0054] Furthermore, for example, when construction work is being carried out on a highway in the captured image CI, the information processing device 20 can analyze the location where the construction work is being carried out based on the quasi-static information SI2.

[0055] The output unit 219 outputs data in which the captured image CI and the three-dimensional point cloud data 3D with the adjusted angle of view are superimposed. The output unit 219 displays, for example, the data in which the captured image CI and the three-dimensional point cloud data 3D with the adjusted angle of view superimposed on a screen provided in the information processing device 20. The output unit 219 may, for example, transmit the data in which the captured image CI and the three-dimensional point cloud data 3D with the adjusted angle of view superimposed on an external information processing device.

[0056] Furthermore, the output unit 219 may transmit, for example, only the three-dimensional point cloud data 3D whose angle of view has been adjusted to the information processing device 10. In this case, the information processing device 10 may superimpose the captured image CI transmitted to the information processing device 20 and the three-dimensional point cloud data 3D whose angle of view has been adjusted.

[0057] Furthermore, the information processing device 10 that has received the three-dimensional point cloud data 3D with the adjusted angle of view may superimpose the three-dimensional point cloud data 3D with image data captured at a different time and with the same angle of view and at the same position as the captured image CI shown in Fig. 3. This allows the information processing device 10 to calculate, for example, the positions, number, and sizes of vehicles in the captured image captured at a different time from the captured image CI. Furthermore, the output unit 219 may output three-dimensional map data, particularly high-precision three-dimensional map data (HD map), instead of the three-dimensional point cloud data.

[0058] <Operation> Next, the operation according to the embodiment will be described. Fig. 6 is a flowchart showing an example of processing by the information processing device 20 according to the present embodiment.

[0059] In step S11, the acquisition unit 213 of the information processing device 20 acquires a captured image CI relating to a predetermined space.

[0060] In step S12, the acquisition unit 213 of the information processing device 20 acquires three-dimensional point cloud data 3D relating to space. For example, three-dimensional point cloud data including a scene similar to the captured image CI is acquired.

[0061] In step S13, the setting unit 214 of the information processing device 20 sets a condition related to the angle of view of the three-dimensional point cloud data 3D. The condition related to the angle of view includes one or more angle of view parameters.

[0062] In step S14 , the adjustment unit 215 of the information processing device 20 adjusts the angle of view of the three-dimensional point cloud data 3D based on the conditions set by the setting unit 214 .

[0063] In step S15, the generation unit 216 of the information processing device 20 generates a two-dimensional point cloud image 2D from the adjusted three-dimensional point cloud data 3D.

[0064] In step S16, the determination unit 217 of the information processing device 20 compares the captured image CI with the two-dimensional point cloud image 2D to determine the similarity.

[0065] In step S16, if the determination unit 217 determines that the similarity does not satisfy the condition (for example, the similarity is equal to or greater than a threshold), the process returns to step S13, and the setting unit 214 changes the condition regarding the angle of view based on the determined similarity. On the other hand, if the determination unit 217 determines that the similarity satisfies the condition, the process proceeds to step S17.

[0066] In step S17, the superimposing unit 218 of the information processing device 20 superimposes the captured image CI with specific three-dimensional point cloud data 3D corresponding to the specific two-dimensional point cloud image 2D when the similarity satisfies a predetermined condition.

[0067] In step S18, the output unit 219 of the information processing device 20 outputs data in which the captured image CI and the adjusted three-dimensional point cloud data 3D are superimposed.

[0068] <Other Embodiments> The above-described embodiments are intended to facilitate understanding of the present invention and are not to be construed as limiting the present invention. The present invention may be modified or improved without departing from the spirit thereof, and equivalents thereof are also included in the present invention. Furthermore, the present invention can be formed into various disclosures by appropriately combining multiple components disclosed in the above-described embodiments. For example, some components may be deleted from all of the components shown in the embodiments. Furthermore, components may be appropriately combined in different embodiments.

[0069] In the above embodiment, the "predetermined space" has been described using an expressway as an example, but is not limited to this. Examples of the "predetermined space" include, but are not limited to, general roads, circuit roads, parking lots, runways, tunnels, parks, stadiums, event venues, animal farms, and other building facilities.

[0070] For example, if the "predetermined space" is a parking lot, the information processing device 10 or the information processing device 20 may output data in which a captured image CI of the parking lot is superimposed on 3D point cloud data with an adjusted angle of view, through the above-described processes. The captured image CI may also include a parked vehicle and an empty parking space, and the information processing device 10 or the information processing device 20 may calculate the size of the parking space. For example, the information processing device 10 or the information processing device 20 may transmit the size and location of an empty parking space to a communication terminal or the like held by a driver who is trying to park a vehicle. This allows the driver to easily determine the location of a parking space large enough for parking.

[0071] Furthermore, when the driver's vehicle is traveling in a place where GPS cannot be used, such as a parking lot, the information processing device 10 or the information processing device 20 may calculate the position of the driver's vehicle based on data obtained by superimposing an image of the parking lot taken by the driver's communication terminal or the like on 3D point cloud data with an adjusted angle of view. This allows the driver to know the position of his or her vehicle even in a parking lot where GPS cannot be used, and makes it possible to provide parking assistance.

[0072] Furthermore, the information processing unit 212 may perform processing to identify the vehicle position using the above-described processing. For example, when the acquisition unit 213 acquires vehicle position information from a Global Navigation Satellite System (GNSS) or a Global Positioning System (GPS), the information processing unit 212 identifies three-dimensional point cloud data 3D corresponding to the vehicle position information based on the vehicle position information. For example, the information processing unit 212 identifies three-dimensional point cloud data 3D of a predetermined range that includes the GPS or GNSS position information.

[0073] The acquisition unit 213 also acquires captured images CI from an imaging device of the vehicle (such as a drive recorder or a mobile terminal of the driver). At this time, the information processing unit 212 can accurately identify the vehicle's position information using the captured images CI and the three-dimensional point cloud data 3D identified based on the position information of the GPS or GNSS.

[0074] For example, the setting unit 214 can set three-dimensional point cloud data 3D of the periphery of the vehicle by using GPS or GNSS position information as the installation position of the virtual camera described above.

[0075] By performing the above-described processing, the adjustment unit 215, the generation unit 216, and the determination unit 217 can determine the similarity between the captured image CI and the two-dimensional point cloud image 2D and identify, for example, the two-dimensional point cloud image 2D with the highest similarity. The information processing unit 212 may identify the position where the captured image CI was captured, i.e., the position of the vehicle, using the two-dimensional point cloud image 2D with the highest similarity or the three-dimensional point cloud image 3D corresponding to this two-dimensional point cloud image 2D.

[0076] Through the above processing, in cases where the accuracy of GPS or GNSS location information is not high, it becomes possible to appropriately identify the imaging position of the captured image CI (e.g., the position of the vehicle) by using a three-dimensional point cloud image 3D based on GPS or GNSS location information and the captured image CI to align the two-dimensional point cloud image generated by the three-dimensional point cloud image 3D with the captured image CI.

[0077] Furthermore, if the driver's vehicle and another vehicle (hereinafter referred to as the "other vehicle") are involved in an accident in a location where GPS cannot be used, the position of the other vehicle may be calculated based on data (hereinafter referred to as "superimposed data") obtained by superimposing an image of the driving location captured by the driver's communication terminal or the like (e.g., a dashcam) on a two-dimensional point cloud image generated based on the three-dimensional point cloud data with an adjusted angle of view. The information processing device 20 may include a calculation unit that performs this calculation process. The calculation unit may calculate the position of the other vehicle included in the image of the driving location captured by the driver's communication terminal or the like (e.g., a dashcam) using, for example, coordinate values ​​of points included in the two-dimensional point cloud image. This allows the driver to determine the position of the other vehicle even if the driver is involved in an accident with the other vehicle in a location where GPS cannot be used, for example.

[0078] The calculation unit may also calculate the position of the other vehicle based on the superimposed data for each frame, and calculate the speed of the other vehicle at the time of the accident (which may include immediately before or immediately after the accident) based on the calculated position. The calculation unit may calculate the speed of the other vehicle based on, for example, the difference between the position of the other vehicle in a given frame and the position of the other vehicle in the immediately preceding (or immediately following) frame.

[0079] For example, if the "predetermined space" is a stadium, the information processing device 10 or the information processing device 20 may output data in which a captured image CI of the stadium is superimposed on the 3D point cloud data through the above-described processes. The captured image CI may also include players in the stadium and equipment used in the sport. As a result, for example, if the sport is soccer, the information processing device 10 or the information processing device 20 can calculate the positional relationship between the players and the ball in the captured image CI, making it possible to easily analyze the strategy of the match, etc.

[0080] For example, if the "predetermined space" is a livestock farm, the information processing device 10 or the information processing device 20 may output data in which a captured image CI of the livestock farm is superimposed on 3D point cloud data through the above-described processes. The captured image CI may also include one or more animals in the livestock farm. This allows the information processing device 10 or the information processing device 20 to calculate, for example, the distance traveled by a livestock animal and its positional relationship with other animals, thereby making it easy to analyze the behavior of the animals.

[0081] 1...information processing system, 10...information processing device, 20...information processing device, 30...database, 210...processor, 212...information processing unit, 213...acquisition unit, 214...setting unit, 215...adjustment unit, 216...generation unit, 217...determination unit, 218...superposition unit, 219...output unit, 220...network communication interface, 230...storage device, 240...user interface, 250...communication bus, CI...captured image, 2D...two-dimensional point cloud image, 3D...three-dimensional point cloud data

Claims

1. An information processing method executed by an information processing device including a processor, wherein the processor executes the following operations: acquiring an image of a specified space; acquiring three-dimensional point cloud data of the space; setting conditions related to an angle of view of the three-dimensional point cloud data; adjusting the angle of view of the three-dimensional point cloud data based on the conditions; generating a two-dimensional point cloud image from the adjusted three-dimensional point cloud data; and comparing the captured image and the two-dimensional point cloud image to determine similarity.

2. The information processing method according to claim 1, wherein said setting includes changing said condition based on said similarity.

3. The information processing method according to claim 1 or 2, wherein the processor further executes superimposing the captured image with specific three-dimensional point cloud data corresponding to a specific two-dimensional point cloud image when the similarity satisfies a predetermined condition.

4. The information processing method of claim 3, wherein the similarity includes a similarity between the captured image and the two-dimensional point cloud image, and when the similarity is equal to or greater than a predetermined threshold, the superimposing includes superimposing the captured image with the specific three-dimensional point cloud data.

5. The information processing method according to claim 3, wherein the superimposing includes: acquiring specific three-dimensional map data based on the specific three-dimensional point cloud data; and superimposing the captured image and the specific three-dimensional map data.

6. The information processing method according to claim 1, wherein the processor further executes specifying an imaging position of the captured image based on a two-dimensional point cloud image for which the similarity is determined to satisfy a predetermined condition.

7. A program to be executed by an information processing device including a processor, the program causing the processor to execute the following operations: acquire an image of a specified space; acquire three-dimensional point cloud data of the space; set conditions related to an angle of view of the three-dimensional point cloud data; adjust the angle of view of the three-dimensional point cloud data based on the conditions; generate a two-dimensional point cloud image from the adjusted three-dimensional point cloud data; and determine similarity by comparing the captured image and the two-dimensional point cloud image.

8. An information processing device including a processor, wherein the processor performs the following operations: acquiring an image of a specified space; acquiring three-dimensional point cloud data of the space; setting conditions related to an angle of view of the three-dimensional point cloud data; adjusting the angle of view of the three-dimensional point cloud data based on the conditions; generating a two-dimensional point cloud image from the adjusted three-dimensional point cloud data; and comparing the captured image and the two-dimensional point cloud image to determine similarity.

Images