Information processing device, information processing method, and information processing system

Abstract

The present invention provides an information processing device, an information processing method, and an information processing system capable of increasing the density of point cloud data. This information processing device includes a point cloud data synthesis unit that performs predetermined processing on a plurality of pieces of point cloud data generated in different periods on the basis of imaging information generated by a mobile body sensor device mounted on a mobile body, and generates synthetic point cloud data by synthesizing the plurality of pieces of point cloud data.

Description

Information processing device, information processing method, and information processing system

[0001] The present technology relates to an information processing device, an information processing method, and an information processing system.

[0002] As a conventional technology, there is SfM (Structure from Motion), a technology for generating point cloud data representing terrain from multiple image data captured by a camera mounted on a mobile object such as a drone. Another conventional technology proposed is a method in which point cloud data is generated from image data captured by multiple work vehicles, transmitted to an information processing device, and the point cloud data is integrated in the information processing device, and desired point cloud data is generated by instructing the work vehicle on an imaging range (Patent Document 1).

[0003] JP 2017-186870 A

[0004] However, with such conventional technology, in order to increase the density of the generated point cloud data, it is necessary to increase the resolution and overlap rate of the original image data. However, lowering the drone's flight altitude and increasing the number of images taken poses the problem of increasing the data acquisition time (flight time), data size (storage capacity and communication volume), and data processing time (SfM processing volume) required to obtain one set of point cloud data.

[0005] The present technology has been developed in consideration of such problems, and aims to provide an information processing device, an information processing method, and an information processing system that can increase the density of point cloud data.

[0006] In order to solve the above-mentioned problems, a first technology is an information processing device including a point cloud data synthesis unit that performs predetermined processing on a plurality of point cloud data generated at different times based on photographing information generated by a mobile body sensor device mounted on a mobile body, and synthesizes the plurality of point cloud data to generate synthesized point cloud data.

[0007] The second technology is an information processing method that applies predetermined processing to multiple point cloud data generated at different times based on photographic information generated by a mobile body sensor device mounted on a mobile body, and synthesizes the multiple point cloud data to generate composite point cloud data.

[0008] The third technology is an information processing system comprising a mobile body sensor device mounted on a mobile body and generating photographic information including image data, a work sensor device generating work information indicating work performed by a work machine, a point cloud data generation device generating point cloud data based on the photographic information, and an information processing device that performs predetermined processing on multiple point cloud data generated at different times and synthesizes the multiple point cloud data to generate composite point cloud data.

[0009] 1 is a diagram illustrating a configuration of an information processing system 1000. FIG. 1 is a block diagram illustrating a configuration of an aircraft sensor device 100. FIG. 2 is a block diagram illustrating a configuration of a working sensor device 200. FIG. 3 is a block diagram illustrating a configuration of a point cloud data generation device 300. FIG. 4 is a diagram illustrating the data structures of point cloud data and attribute information. FIG. 5 is a block diagram illustrating a configuration of an information processing device 400. FIG. 6 is a sequence diagram illustrating the processing of the entire information processing system 1000. FIG. 7 is a flowchart illustrating the processing of the aircraft sensor device 100. FIG. 8 is a flowchart illustrating the processing of the working sensor device 200. FIG. 9 is a flowchart illustrating the processing of the point cloud data generation device 300. FIG. 10 is a flowchart illustrating the processing of the information processing device 400. FIG. 11 is an explanatory diagram of point cloud data synthesis. FIG. 12 is a flowchart illustrating thinning processing. FIG. 13 is a flowchart illustrating pixel value adjustment processing. FIG. 14 is a diagram illustrating the data structure of point cloud data in a second embodiment. FIG. 15 is a flowchart illustrating the processing of the point cloud data generation device 300 in the second embodiment. FIG. 16 is a flowchart illustrating the thinning processing in the second embodiment. FIG. 17 is a flowchart illustrating the pixel value adjustment processing in the second embodiment. FIG. 18 is a block diagram illustrating the configuration of a point cloud data generation device 300 in a third embodiment. FIG. 19 is a flowchart illustrating the processing of the point cloud data generation device 300 in the third embodiment. FIG. 19 is a flowchart illustrating the processing of the information processing device 400 in the third embodiment. FIG. 19 is a block diagram illustrating the configuration of an information processing device 400 in a modified example.

[0010] Hereinafter, embodiments of the present technology will be described with reference to the drawings. The description will be made in the following order. <First embodiment> [Configuration of information processing system 1000] [Configuration of aircraft sensor device 100] [Configuration of work sensor device 200] [Configuration of point cloud data generation device 300] [Configuration of information processing device 400] [Processing of entire information processing system 1000] [Processing in aircraft sensor device 100] [Processing in work sensor device 200] [Processing in point cloud data generation device 300] [Processing in information processing device 400] <Second embodiment> [Configuration of point cloud data and attribute information] [Processing in point cloud data generation device 300] [Processing in information processing device 400] <Third embodiment> [Configuration of point cloud data generation device 300] [Processing in point cloud data generation device 300] [Processing in information processing device 400] <Modifications>

[0011] 1 shows the configuration of an information processing system 1000. The information processing system 1000 includes an aeronautical sensor device 100, an operational sensor device 200, a point cloud data generation device 300, an information processing device 400, and a display terminal 500.

[0012] The aviation sensor device 100, the work sensor device 200, and the information processing device 400 are each connected to the point cloud data generation device 300. The display terminal 500 is also connected to the information processing device 400. The connection method may be a wired connection or a wireless connection. Examples of wired connection methods include HDMI (registered trademark) (High-Definition Multimedia Interface) and USB (Universal Serial Bus), and examples of wireless connection methods include Wi-Fi, networks such as 4G (fourth generation mobile communication system) and 5G (fifth generation mobile communication system), Bluetooth (registered trademark), and NFC (Near Field Communication). In addition, data and information may be supplied from the aviation sensor device 100, the work sensor device 200, and the point cloud data generation device 300 to the information processing device 400 via a recording medium such as a semiconductor memory.

[0013] The aviation sensor device 100 is mounted on an aircraft 10 (such as a drone) as a moving object. The aviation sensor device 100 corresponds to the mobile object sensor device in the claims. The aircraft 10 has a power unit and a control unit (not shown) for flight, and flies at an altitude and route instructed by a user of the information processing system 1000 or the like. The aviation sensor device 100 acquires photographic information by sensing while the aircraft 10 is flying. The aviation sensor device 100 supplies the photographic information to a point cloud data generation device 300 on the ground during or after the aircraft 10 has finished flying.

[0014] The work sensor device 200 is mounted on a work vehicle 20 serving as a work machine. The work vehicle 20 has a vehicle and work power unit and a control unit (not shown), and is one or more vehicles that travel along a route instructed by a worker (such as a civil engineering or construction contractor) to perform civil engineering work, construction work, or the like. The work sensor device 200 acquires work information while the work vehicle 20 is working. The work sensor device 200 supplies the work information to the point cloud data generation device 300 during or after work has been completed.

[0015] Work refers to civil engineering or construction work performed by the work vehicle 20 in the real world that changes the state of land. Examples of work that changes the state of land include digging holes, filling up land, cutting earth, or other actions that change the shape of the land, placing objects on the land, or erecting a building on the land. Therefore, the work vehicle 20 may be, for example, an excavator, bulldozer, wheel loader, dump truck, motor grader, road roller, crane, rock drill, or pole digger, or any other work machine used for civil engineering or construction work, also known as heavy machinery or construction machinery.

[0016] 1 shows work sensor device 200A mounted on work vehicle 20A and work sensor device 200B mounted on work vehicle 20B, but in the present technology, the number of work vehicles 20 and work sensor devices 200 is not limited to two, and may be one, or three or more. Furthermore, when there is no need to distinguish between work sensor devices 200A and 200B, they may be simply referred to as work sensor device 200.

[0017] The point cloud data generating device 300 generates point cloud data (three-dimensional data) based on the photographing information transmitted from the aeronautical sensor device 100. The point cloud data generating device 300 also adds attribute information to the point cloud data. The point cloud data generating device 300 transmits the generated point cloud data to the information processing device 400.

[0018] Point cloud data is a collection of multiple points with location and color information. It can represent terrain, objects, etc. as a collection of large numbers of points, and can be used in a variety of fields such as civil engineering, architecture, and manufacturing.

[0019] The information processing device 400 generates high-density composite point cloud data by combining the multiple point cloud data transmitted from the point cloud data generation device 300. The information processing device 400 transmits the composite point cloud data to the display terminal 500 in response to a request from the display terminal 500.

[0020] Based on a user's operation, the display terminal 500 requests point cloud data of the work area from the information processing device 400. The display terminal 500 includes a liquid crystal display, an organic EL (Electroluminescence) display, or the like, and displays the point cloud data received from the information processing device 400. Specifically, the display terminal 500 is, for example, a personal computer, a smartphone, a tablet terminal, or a wearable device.

[0021] [Configuration of the Aviation Sensor Device 100] Next, the configuration of the aircraft sensor device 100 will be described with reference to FIG.

[0022] The aircraft sensor device 100 includes a camera 101 as sensors, a satellite positioning unit 102, and an inertial measurement unit 103. The camera 101 is equipped with an imaging element, a signal processing circuit, and the like, and is capable of capturing RGB (Red, Green, Blue) or monochrome images. The satellite positioning unit 102 acquires position information (e.g., latitude, longitude, and altitude) of the aircraft 10 using, for example, a Global Navigation Satellite System (GNSS) device or a Global Positioning System (GPS) device. The inertial measurement unit 103 acquires the attitude of the camera 101 using, for example, an Inertial Measurement Unit (IMU) device. The attitude information is, for example, a three-axis rotation vector (yaw, pitch, and roll) calculated from the acceleration and angular velocity of the aircraft sensor device 100.

[0023] When the sensor control unit 104 receives an image capture instruction from the outside (such as a user of the information processing system 1000 or a remote control for operating the drone if the aircraft 10 is a drone), which instructs the generation of image capture information consisting of image data, position information, and attitude information, the sensor control unit 104 instructs the image data acquisition unit 105 to acquire image data by capturing an image with the camera 101. Furthermore, when the sensor control unit 104 receives the image capture instruction, it instructs the position information acquisition unit 106 to acquire position information at the time of capturing the image data using the satellite positioning unit 102. Furthermore, when the sensor control unit 104 receives the image capture instruction, it instructs the attitude information acquisition unit 107 to acquire the current attitude of the camera 101 using the inertial measurement unit 103.

[0024] The imaging information generating unit 108 generates imaging information by integrating the image data, position information, and orientation information together with time information indicating that they are synchronized. For example, the imaging information can be configured as one file including one frame of image data, position information, and orientation information.

[0025] When the photographing instruction is a signal repeatedly input at regular intervals, the aviation sensor device 100 continuously acquires photographing information at regular intervals. Of course, it is also possible to photograph while changing the flight path in response to instructions from the user via wireless communication during flight. Furthermore, it is also possible to add or change the timing of photographing in response to instructions from the user via wireless communication during flight.

[0026] The photographing information storage unit 109 is configured by a recording medium such as a semiconductor memory or a hard disk, and records the photographing information supplied from the photographing information generation unit 108 .

[0027] The photographing information transmission unit 110 reads out the photographing information from the photographing information storage unit 109 during flight or after landing of the aircraft 10 and transmits it to the point cloud data generation device 300 .

[0028] Furthermore, by using a removable recording medium with a general-purpose interface as the photography information storage unit 109, the photography information can be transferred by removing the recording medium from the aviation sensor device 100 after landing and connecting it to a recording medium connection slot (not shown) of the point cloud data generation device 300.

[0029] The aviation sensor device 100 may be configured by an electronic device having information processing and communication functions, such as a personal computer, a smartphone, or a tablet terminal, or the aviation sensor device 100 may be realized by such electronic device executing a program. Alternatively, the aircraft 10 itself may have the functions of the aviation sensor device 100. Alternatively, the aviation sensor device 100 may be configured as a stand-alone device.

[0030] [Configuration of Work Sensor Device 200] Next, the configuration of the work sensor device 200 mounted on the work vehicle 20 will be described with reference to FIG.

[0031] The work sensor device 200 includes, as sensors, a satellite positioning unit 201 and an operation information acquisition unit 202. The satellite positioning unit 201 acquires position information (e.g., latitude, longitude, and altitude) of the work vehicle 20 using, for example, a GNSS device or a GPS device.

[0032] The operation information acquisition unit 202 acquires operation information indicating the operation state of the work vehicle 20. The operation information may be acquired, for example, from operation information of the work vehicle 20, from a control unit of the work vehicle 20, or by capturing an image of the work vehicle 20 with a camera and recognizing the operation from the image.

[0033] The operation information indicates the operation of the work vehicle 20 to change the state of the land. As described above, the work performed by the work vehicle 20 includes, for example, digging holes, filling, cutting, filling in, placing objects on the land, and erecting a building on the land. Therefore, the operation specifically refers to the operation of the bucket, arm, etc. if the work vehicle 20 is a hydraulic excavator, or the operation of the blade, etc. if the work vehicle 20 is a bulldozer. However, these vehicle types and operations are merely examples, and any operation based on the functions of the work vehicle 20 is acceptable. The operation of the work vehicle 20 includes operations performed while the work vehicle 20 is traveling, stopped, and working. The operation of the work vehicle 20 includes all operations that cause changes in the land that result in differences between point cloud data generated based on image data photographed from the air before the operation and point cloud data generated based on image data photographed from the air after the operation.

[0034] When the sensor control unit 203 receives a generation instruction to generate work information from outside (such as a user of the information processing system 1000), it instructs the position information acquisition unit 204 to acquire current position information using the satellite positioning unit 201. Furthermore, when the sensor control unit 203 receives the generation instruction, it instructs the operation information acquisition unit 202 to acquire current operation information of the work vehicle 20.

[0035] The work information generation unit 205 generates work information by adding time information indicating that the position information supplied from the position information acquisition unit 106 and the operation information supplied from the operation information acquisition unit 202 are synchronized. As a result, the work information becomes information indicating the work being performed by the work vehicle 20 and the location where the work is being performed.

[0036] The work information transmission unit 206 transmits work information to the point cloud data generation device 300 while the work vehicle 20 is working or after the work is completed.

[0037] If the generation instruction is a signal that is repeated at regular intervals, the work sensor device 200 can generate work information at regular intervals.

[0038] The work sensor device 200 may be configured by an electronic device with information processing and communication functions, such as a personal computer, smartphone, or tablet terminal, or the work sensor device 200 may be realized by such electronic device executing a program. Furthermore, the work vehicle 20 itself may have the functions of the work sensor device 200. Furthermore, the work sensor device 200 may be configured as a stand-alone device.

[0039] [Configuration of Point Cloud Data Generating Apparatus 300] Next, the configuration of the point cloud data generating apparatus 300 will be described with reference to FIG.

[0040] The image capturing information receiving unit 301 receives image capturing information transmitted from the aircraft sensor device 100 and supplies it to the point cloud data generating unit 302 and the object identifying unit 303 .

[0041] The point cloud data generating unit 302 generates three-dimensional point cloud data using, for example, an SfM (Structure From Motion) method based on image data, position information, and orientation information as shooting information, and supplies the data to the attribute information adding unit 307 .

[0042] The object identification unit 303 identifies objects present as subjects in the image data using an object recognition method such as semantic segmentation, and notifies the attribute information addition unit 307 of the object identification results. Other object identification methods include a template matching method, a matching method based on luminance distribution information of the subject, a method based on the brightness of the subject, a method based on the color or feature amount contained in the image, a method using machine learning, and a method using artificial intelligence. These methods may also be combined to improve the identification accuracy. Furthermore, the image data may be transmitted to another device, an external server, a cloud, etc., which may perform object identification processing, and the object identification results may be received by the point cloud data generation device 300.

[0043] The work information receiving unit 304 receives the work information transmitted from the work sensor device 200 and supplies it to the work information accumulation unit 305 .

[0044] The work information accumulation unit 305 accumulates work information transmitted from multiple work sensor devices 200, and when work information transmitted from all work vehicles 20 has been collected, supplies the accumulated work information to the work area identification unit 306. If there is only one work vehicle 20, the image capture information accumulation unit 305 need only supply the work information to the work area identification unit 306 without performing any processing. Alternatively, the work information receiving unit 304 may supply the work information directly to the work area identification unit 306.

[0045] The work area identification unit 306 uses all of the work information to identify a work area that indicates an area where the land condition has changed due to work by the work vehicle 20. The work area identification unit 306 supplies the attribute information addition unit 307 with information about the identified work area, including the size of the work area and multiple pieces of position information that indicate the work area.

[0046] Based on the work area information supplied from the work area identification unit 306, the attribute information addition unit 307 adds work area inside / outside information indicating whether each point is located inside or outside the work area as attribute information to each point that constitutes the point cloud data.

[0047] Furthermore, based on the object identification results supplied from the object identification unit 303, the attribute information addition unit 307 adds object type information indicating what type of object each point constitutes to each point constituting the point cloud data as attribute information.

[0048] The point cloud data transmission unit 308 transmits the point cloud data to which the attribute information has been added to the information processing device 400 .

[0049] The point cloud data generation device 300 is equipped with a storage medium (not shown) and is capable of recording data and information at each processing stage. The point cloud data generation device 300 may also be equipped with a recording function to an external recording medium, thereby enabling data and information to be recorded at each processing stage.

[0050] An example of the data structure of point cloud data and attribute information is shown in Figure 5. Each point that makes up the point cloud data is associated with latitude, longitude, and altitude values ​​as location information. This location information is based on the location information included in the photography information that forms the basis of the point cloud data.

[0051] Each point in the point cloud data is associated with R, G, and B pixel values. Furthermore, each point is associated with attribute information, such as work area inside / outside information indicating whether the point is located inside or outside the work area, and object type information. Object types include, for example, ground (soil), buildings, people, trees, etc., but any object that can be identified by the object identification unit 303 may be used. The object type information can also be a flag indicating whether the object is unwanted or not.

[0052] The point cloud data generation device 300 is configured as described above. The point cloud data generation device 300 may be configured as an electronic device having information processing and communication functions, such as a personal computer, smartphone, or tablet terminal, or may be configured as a server or cloud. Furthermore, the point cloud data generation device 300 may be realized by an electronic device having computer functions executing a program. The program may be pre-installed on the electronic device, or may be distributed by download or storage medium, and installed by a user, etc. Furthermore, the point cloud data generation device 300 may be configured as a standalone device.

[0053] [Configuration of Information Processing Apparatus 400] Next, the configuration of the information processing apparatus 400 will be described with reference to FIG.

[0054] The point cloud data receiving unit 401 receives the point cloud data transmitted from the point cloud data generating device 300 and supplies it to the point cloud data saving unit 402 .

[0055] The point cloud data storage unit 402 stores the point cloud data supplied from the point cloud data receiving unit 401, and accumulates it together with point cloud data that was generated and supplied in the past.

[0056] The point cloud data synthesis unit 403 generates synthesized point cloud data by performing predetermined processing on multiple point cloud data generated at different times and synthesizing the multiple point cloud data read from the point cloud data storage unit 402. More specifically, the point cloud data synthesis unit 403 generates synthesized point cloud data by synthesizing new point cloud data, which is point cloud data generated most recently, with one or more old point cloud data generated earlier than the new point cloud data.

[0057] Furthermore, the point cloud data synthesis unit 403 performs processing to invalidate predetermined points that exist in the point cloud data to be synthesized.

[0058] Furthermore, the point cloud data synthesis unit 403 adjusts the synthesized point cloud data by performing either or both of thinning processing and pixel value adjustment processing on the synthesized point cloud data.

[0059] When the data request receiving unit 404 receives a request for composite point cloud data from the display terminal 500, it supplies information about the requested area to the point cloud data transmitting unit 405. The point cloud data transmitting unit 405 then reads the composite point cloud data for the requested area from the recording medium and transmits the composite point cloud data to the display terminal 500. This allows the user to view point cloud data for the entire work area or for any position on the display terminal 500. Note that the display terminal 500 can request composite point cloud data by specifying a specific area, or can request composite point cloud data without specifying an area.

[0060] The information processing device 400 is configured as described above. The information processing device 400 may be configured as an electronic device having information processing and communication functions, such as a personal computer, smartphone, or tablet terminal, or may be configured as a server or cloud. Furthermore, the information processing device 400 and the information processing method may be realized by the electronic device or server executing a program. The program may be pre-installed on the electronic device, or may be distributed by download or storage medium, etc., and installed by a user, etc. Furthermore, the information processing device 400 may be configured as a standalone device.

[0061] The information processing system 1000 is configured as described above.

[0062] [Overall Processing of Information Processing System 1000] Next, a description will be given of processing in the information processing system 1000. First, the overall processing of the information processing system 1000 will be described with reference to FIG.

[0063] First, the aircraft 10 flies over the work area, and the aircraft sensor device 100 generates initial photographic information, which is then transmitted to the point cloud data generation device 300. Upon receiving the photographic information, the point cloud data generation device 300 generates initial point cloud data based on the photographic information, and transmits the point cloud data with attribute information added to the information processing device 400. Upon receiving the initial point cloud data, the information processing device 400 stores it in the point cloud data storage unit 402.

[0064] Thereafter, the display terminal 500 requests point cloud data from the information processing device 400 at an arbitrary timing, receives the initial point cloud data transmitted from the information processing device 400, and displays it.

[0065] After the initial image capture by the aeronautical sensor device 100, the work vehicles 20A and 20B generate work information while performing their work. After completing their work, the work sensor device 200 transmits the work information to the point cloud data generation device 300.

[0066] Thereafter, the aircraft 10 flies over the work area, generates second photographic information using the aircraft sensor device 100 , and transmits the photographic information to the point cloud data generation device 300 .

[0067] When the point cloud data generating device 300 receives the photographing information and the work information, it generates point cloud data for the second time and transmits the point cloud data with the attribute information added to the information processing device 400 .

[0068] When the information processing device 400 receives the second point cloud data, it stores it in the point cloud data storage unit 402, and generates composite point cloud data by combining the initial point cloud data (old point cloud data) and the second point cloud data (new point cloud data). Thereafter, the display terminal 500 requests point cloud data from the information processing device 400 at any timing, and receives and displays the combined point cloud data transmitted from the information processing device 400.

[0069] Thereafter, the same sequence as the second time is repeated, thereby increasing the density of the new point cloud data using the old point cloud data.

[0070] [Processing in the Aircraft Sensor Device 100] Next, processing in the aircraft sensor device 100 will be described with reference to FIG.

[0071] First, in step S101, the sensor control unit 104 receives an external shooting instruction instructing generation of shooting information. In step S102, the image data acquisition unit 105 instructs the camera 101 to shoot and acquires image data. Note that the image data acquisition unit 105 may reduce the amount of image data by encoding the acquired image data.

[0072] Furthermore, when the sensor control unit 104 receives an instruction to capture an image from the outside, in step S103 the position information acquisition unit 106 instructs the satellite positioning unit 102 to acquire current position information of the aircraft 10, and acquires the position information.

[0073] Furthermore, when the sensor control unit 104 receives a shooting instruction signal from the outside, in step S104 the orientation information acquisition unit 107 instructs the inertial measurement unit 103 to acquire orientation information of the camera 101, and acquires the orientation information.

[0074] It should be noted that steps S102 to S104 do not necessarily have to be performed in this order, and they may be performed in a different order, or simultaneously or almost simultaneously.

[0075] Next, in step S105, the imaging information generation unit 108 generates imaging information by integrating the position information and posture information corresponding to the imaging time of the image data with the image data and time information. Then, in step S106, the imaging information generation unit 108 outputs the imaging information to the imaging information storage unit 109 and stores it.

[0076] Next, in step S107, the sensor control unit 104 determines whether reception of the image capture instruction has been completed. The processes of steps S102 to S106 are repeated until the image capture instruction is completed (No in step S107). For example, by periodically receiving the image capture instruction from the takeoff to the landing of the aircraft 10, it is possible to acquire multiple pieces of image capture information during flight.

[0077] If reception of the photographing instruction has ended, the process proceeds to step S108 (Yes in step S107). The photographing information transmission unit 110 transmits the photographing information to the point cloud data generation device 300. The timing for transmitting the photographing information to the point cloud data generation device 300 can be set to, for example, immediately after the photographing information is generated in step S105, and the photographing information can be transmitted to the point cloud data generation device 300 in parallel with the generation of the photographing information during flight, thereby eliminating the need to transmit the photographing information after landing.

[0078] [Processing in Working Sensor Device 200] Next, processing in working sensor device 200 will be described with reference to FIG.

[0079] In step S201, when the sensor control unit 203 receives a generation instruction from outside to generate work information, in step S202 the position information acquisition unit 204 instructs the satellite positioning unit 201 to acquire the current position information of the work vehicle 20, and acquires the position information.

[0080] Furthermore, when the sensor control unit 203 receives a generation instruction to generate work information, the operation information acquisition unit 202 acquires operation information indicating the operation state of the vehicle in step S203.

[0081] It should be noted that steps S202 and S203 do not necessarily have to be performed in this order, and they may be performed in a different order, or may be performed simultaneously or almost simultaneously.

[0082] Next, in step S204, the work information generation unit 205 generates work information indicating the area where the work vehicle 20 performed work that changed the land condition, based on the position information and operation information. For example, by excluding position information when the packet or blade of the work vehicle 20 is stopped and using only position information when the packet or blade is in operation, work information indicating the work area where the land condition has been changed can be generated.

[0083] Next, in step S205, the sensor control unit 203 determines whether reception of the generation instruction has been completed. Then, the processing of steps S202 to S204 is repeated until reception of the generation instruction has been completed (No in step S205). For example, work information can be generated by periodically repeating the process from starting to stopping the engine of the work vehicle 20.

[0084] If the generation instruction has been completed, the process proceeds to step S206 (Yes in step S205), and the work information transmission unit 206 transmits the work information to the point cloud data generation device 300. The timing for transmitting the work information to the point cloud data generation device 300 can be, for example, immediately after the work information is generated in step S204, and by transmitting the work information to the point cloud data generation device 300 in parallel with the generation of the work information during work, it is possible to eliminate the need to transmit the work information after the work is completed.

[0085] In the above explanation, work information is generated when the sensor control unit 203 receives a generation instruction in step S201, but work information may also be generated and transmitted to the point cloud data generation device 300 each time the work position of the work vehicle 20 changes by more than a predetermined distance. By generating and transmitting work information based on the work position, it is possible to reduce duplication of data included in the work information and the frequency of work information transmission.

[0086] [Processing in Point Cloud Data Generation Apparatus 300] The processing in the point cloud data generation apparatus 300 will be described with reference to FIG.

[0087] In step S301, the photographing information receiving unit 301 receives the photographing information transmitted from the aircraft sensor device 100. When the photographing information is obtained from an external recording medium, a read process is performed from the external recording medium.

[0088] In step S302, the work information receiving unit 304 receives the work information transmitted from the work sensor device 200 and supplies it to the work information accumulation unit 305. If the work information is obtained from an external recording medium, a read process from the external recording medium is performed. Then, in step S303, the work information accumulation unit 305 accumulates the work information.

[0089] Then, steps S302 to S304 are repeated until work information is received from all of the work sensor devices 200 (No in step S304).

[0090] If work information has been received from all of the work sensor devices 200, the process proceeds to step S305 (Yes in step S304). Note that if work information is not received for a predetermined period of time after work information has been received from one or more work vehicles 20, the process may proceed to step S305 assuming that work information has been received from all of the work vehicles 20.

[0091] Next, in step S305, the work area identification unit 306 reads out all work information stored in the work information storage unit 305, and identifies a work area, which is an area where the land condition has changed due to work by the work vehicle 20, based on all the work information.

[0092] For example, while work vehicle 20A is working at position a, work sensor device 200A generates work information and transmits it to point cloud data generation device 300. Also, while work vehicle 20A is working at position b, work sensor device 200A generates work information and transmits it to point cloud data generation device 300. Furthermore, while work vehicle 20A is working at position c, work sensor device 200A generates work information and transmits it to point cloud data generation device 300. In this case, work area identification unit 306 identifies the area connecting positions a, b, and c as the work area.

[0093] The work area may also be identified based on work information transmitted from multiple different work sensor devices 200. For example, work vehicle 20A works at different positions a1, a2, and a3 in sequence, and work sensor device 200A generates work information for each of positions a1, a2, and a3 and transmits it to point cloud data generation device 300. Also, work vehicle 20B works at different positions b1, b2, and b3 in sequence, and work sensor device 200B generates work information for each of positions b1, b2, and b3 and transmits it to point cloud data generation device 300. Furthermore, work vehicle 20C works at different positions c1, c2, and c3 in sequence, and work sensor device 200C generates work information for each of positions c1, c2, and c3 and transmits it to point cloud data generation device 300. In this case, the work area specifying unit 306 specifies three work areas: an area connecting a1, a2, and a3; an area connecting b1, b2, and b3; and an area connecting c1, c2, and c3.

[0094] Furthermore, if multiple work areas are close to each other or overlap, the work area identification unit 306 combines the multiple work areas into a single work area based on the location information included in the work information. On the other hand, if multiple work areas are far apart, they are treated as separate work areas. For example, if the distance between multiple work areas is within approximately 3 to 5 meters (about the length of one vehicle), the work areas are considered to be close to each other and are combined into a single work area. Furthermore, if the distance between multiple work areas is greater than approximately 5 meters, the work areas are considered to be far apart and are not combined. The distance used to determine whether work areas are close or far is merely an example, and the user of the point cloud data generation device 300 may be able to set the distance.

[0095] Next, in step S306, the point cloud data generator 302 generates point cloud data using the imaging information. The point cloud data generator 302 generates three-dimensional point cloud data using image data, position information, and orientation information included in the imaging information, for example, by the SfM technique.

[0096] Next, in step S307, the attribute information addition unit 307 determines whether each point that makes up the point cloud data is inside or outside the work area based on the work area identified in step S305, and adds the work area inside / outside information to the point cloud data as attribute information.

[0097] Next, in step S308, the object identification unit 303 identifies objects that exist as subjects in the image data included in the shooting information, and supplies the object identification results to the attribute information addition unit 307. Then, in step S309, the attribute information addition unit 307 adds object type information as attribute information to the point cloud data based on the object identification results.

[0098] Next, in step S310, the point cloud data transmission unit 308 transmits the point cloud data with the attribute information added to it to the information processing device 400. Note that the attribute information may be added to the point cloud data when the generation of point cloud data in step S306 is completed for all of the shooting information and then transmitted to the information processing device 400. It is also possible to sequentially add attribute information to the point cloud data each time the generation of point cloud data in step S306 is completed for some of the shooting information and then transmit it to the information processing device 400. Note that although the point cloud data with the attribute information added is transmitted in step S310, it is also possible to transmit the point cloud data when it is generated in step S306 and the attribute information when step S310 is completed.

[0099] It should be noted that the addition of attribute information in step S307 and the addition of attribute information in steps S308 and S309 do not necessarily have to be performed in this order, but may be performed in the reverse order, or may be performed simultaneously or almost simultaneously.

[0100] Furthermore, the processing related to the work information in steps S302 to S305 and the processing related to the shooting information in steps S306 and S308 do not have to be performed in the order shown in Figure 10; the processing related to the shooting information may be performed first, or the processing related to the work information and the processing related to the shooting information may be performed simultaneously or almost simultaneously.

[0101] [Processing in Information Processing Device 400] Next, the processing in the information processing device 400 will be described with reference to Fig. 11. Note that the processing in Fig. 11 is assumed to be processing in a state where point cloud data has already been transmitted from the point cloud data generation device 300 and one or more pieces of point cloud data have been stored in the point cloud data storage unit 402.

[0102] In step S401, the information processing device 400 initializes (n=0) an internal counter that counts the number of data sets of old point cloud data.

[0103] Next, in step S402, the point cloud data receiving unit 401 receives new point cloud data (t) generated at time t from the point cloud data generating device 300, and then in step S403, stores the point cloud data (t) in the point cloud data storage unit 402.

[0104] Next, in step S404, the point cloud data synthesis unit 403 refers to the object type information included in the attribute information of the new point cloud data (t), and identifies and invalidates points among the multiple points that make up the new point cloud data (t) that correspond to predetermined unwanted objects.

[0105] An example of an unwanted object is an object other than the ground (soil). For example, "trees" may be defined as unwanted objects in advance, and if a point whose object type is a tree is among the multiple points constituting the new point cloud data, that point is invalidated. The user of the information processing system 1000 may be able to set what constitutes unwanted objects. In this way, for example, if a tree is defined as unwanted and a point corresponding to a tree in the new point cloud data is invalidated as unwanted, the invalidated point in the new point cloud data can be supplemented with the old point cloud data in the combined point cloud data.

[0106] Next, in step S405, the information processing device 400 updates the value of the internal counter (n=n+1) to synthesize the old point cloud data and the new point cloud data.

[0107] Next, in step S406, the point cloud data synthesis unit 403 reads out the old point cloud data (t-1) generated at time t-1 from the point cloud data storage unit 402.

[0108] Next, in step S407, the point cloud data synthesis unit 403 refers to the object type information contained in the attribute information of the old point cloud data (t-1) and invalidates points among the multiple points that make up the old point cloud data (t-1) that correspond to predetermined unnecessary objects.

[0109] Next, in step S408, the point cloud data synthesis unit 403 refers to the work area inside / outside information attached to the next new point cloud data (t-n+1) after the point cloud data (t-n), and invalidates the points that are included within the work area of ​​the point cloud data (t-n+1) among the multiple points that make up the point cloud data (t-n).

[0110] When the internal counter is updated to n=1, the work area inside / outside information attached to the next most recent point cloud data after point cloud data (t-1) (which is "time t-n+1", so "time t-1+1", which is time t, i.e., new point cloud data (t) in this case) is referenced, and the points that make up the old point cloud data (t-1) that are included within the work area of ​​the new point cloud data (t) are invalidated. As a result, in the combination of the new point cloud data (t) and the old point cloud data (t-1), only the new point cloud data (t) is valid for the work area in the new point cloud data (t), i.e., the area where the land condition has changed, and the old point cloud data is not included.

[0111] Furthermore, when the internal counter is updated to n=2, the work area inside / outside information attached to the next most recent point cloud data after point cloud data (t-2) (which is "time t-n+1," so "time t-2+1," or time t-1, i.e., new point cloud data (t-1) in this case) is referenced, and the points that make up the old point cloud data (t-2) that are included within the work area of ​​the new point cloud data (t-1) are invalidated. As a result, in the combination of the new point cloud data (t-1) and the old point cloud data (t-2), only the new point cloud data (t-1) is valid for the work area in the new point cloud data (t-1), i.e., the area where the land condition has changed, and the old point cloud data is not included.

[0112] Next, in step S409, the point cloud data synthesis unit 403 synthesizes the old point cloud data (time t-n) and the new point cloud data (time t-n+1) to generate synthesized point cloud data. The generation of synthesized point cloud data does not require alignment of the point cloud data; it is sufficient to simply integrate the data. While the previous point cloud data stored in the point cloud data storage unit 402 may be used as the old point cloud data, synthesizing multiple past point cloud data as the old point cloud data can further increase the density of the points that make up the point cloud data.

[0113] The alignment of the new point cloud data and the old point cloud data is necessary when displaying the combined point cloud data, and the combined point cloud data can be visually obtained by displaying each point in a single space represented by latitude, longitude, and altitude. The display process can be performed using a known point cloud data display tool.

[0114] Next, in step S410, the number N of times for synthesizing the old point cloud data a preset number of times is compared with the counter value n. If n<N, steps S405 to S410 are repeated to repeatedly synthesize the old point cloud data N times (Yes in step S410).

[0115] Here, the synthesis of point cloud data will be explained with reference to Figure 12. The old point cloud data in Figure 12A is shown with horizontal stripes, and the white areas indicate areas where points have been invalidated as unnecessary. The new point cloud data in Figure 12B is shown with vertical stripes, and the white areas indicate areas where points have been invalidated as unnecessary. Furthermore, oval areas indicate areas where the land condition has changed in the work area in the new point cloud data. Figure 12C shows synthesized point cloud data obtained by synthesizing the old point cloud data and the new point cloud data.

[0116] In the synthesized point cloud data, the unnecessary areas of the old point cloud data are filled in with the new point cloud data. Also, in the synthesized point cloud data, the unnecessary areas of the new point cloud data are filled in with the old point cloud data. By synthesizing multiple point cloud data in this way, it is possible to fill in areas where points are lacking in each point cloud data.

[0117] In addition, in the synthesized point cloud data, the area corresponding to the working area of ​​the new point cloud data contains only the new point cloud data, and the old point cloud data is invalidated. Furthermore, in the synthesized point cloud data, the area where the old point cloud data and the new point cloud data do not overlap contains only the point clouds of the old point cloud data and the new point cloud data. In the area where points are not invalidated, the points are densely packed due to the synthesis of the old point cloud data and the new point cloud data.

[0118] The point cloud data synthesis unit 403 may also perform processing to enlarge an area where the new point cloud data and the old point cloud data do not overlap. The area to be enlarged is linked to an area where the new point cloud data and the old point cloud data overlap, thereby enlarging the point cloud data. Specifically, by merging (integrating or linking) the point cloud data of the non-overlapping area with the point cloud data of the overlapping area, the synthesis result including the non-overlapping area can be viewed when the synthesized point cloud data is displayed.

[0119] When invalidating a point, it is possible to indicate that the point is invalid by recording a value of 0 as the position information, or to delete the invalid point from the composite point cloud data.

[0120] Returning to the description of FIG. 11 , when the synthesis of N sets of old point cloud data is completed, the process proceeds to step S411 (No in step S410). Then, in step S411, the point cloud data synthesis unit 403 performs thinning processing, and further, in step S412, pixel value adjustment processing. Note that the thinning processing and pixel value adjustment processing are not essential processes, and both may be performed, or only one of them may be performed, or neither may be performed. Details of the thinning processing and pixel value adjustment processing will be described later. Then, in step S413, the synthesized point cloud data is stored on a recording medium.

[0121] Next, in step S414, the data request receiving unit 404 waits until it receives a point cloud data request from the display terminal 500. When the point cloud data request is received, the process proceeds to step S415 (Yes in step S414). Then, in step S415, the point cloud data transmitting unit 405 reads the composite point cloud data of the requested area from the recording medium and transmits it to the display terminal 500.

[0122] Note that Figure 11 describes the order in which the data request is received after the point cloud data synthesis process is completed, but it is also possible to execute the process up to saving the new point cloud data in step S403, and then after receiving the point cloud data request, execute the process from step S404 onwards to synthesize the point cloud data of the requested area and send the synthesized point cloud data to the display terminal 500.

[0123] 13, the thinning process performed by the point cloud data synthesis unit 403 will be described. Synthesizing multiple point cloud data can increase the density of the synthesized point cloud data, but performing thinning process on areas where points overlap or are concentrated can also make the points have an appropriate density.

[0124] In step S501, any valid point in the composite point cloud data is selected as a reference point for the thinning process. Next, in step S502, the distance d1 between the reference point and a neighboring point is calculated. The number of neighboring points is not limited to one, and any number of points may be selected as long as they are within a predetermined range based on the reference point.

[0125] Next, in step S503, the distance d1 is compared with a predetermined threshold th1, and nearby points closer than the threshold th1 (d1<th1) are selected as thinning-out points. The thinning-out points selected in step S503 are not limited to one, and multiple points are selected if the distance d1 is smaller than the threshold th1. Next, in step S504, the most recent point is selected from the thinning-out points as a representative point. If there are multiple most recent points, the representative point can be, for example, the first point in the point cloud data sequence, the last point in the point cloud data sequence, or the point closest to the center of gravity of the multiple points.

[0126] Next, in step S505, the average value of the pixel values ​​of the thinning-out target points is calculated. Here, the average value calculated for each R, G, and B element of the pixel value is used. Note that if there is only one thinning-out target point, there is no need to calculate the average value.

[0127] Next, in step S506, the average pixel value of the points to be thinned is applied to the representative point to determine the pixel value of the representative point. This allows the color of the points to remain in the point cloud data after thinning, rather than simply reducing the number of points by thinning.

[0128] Next, in step S507, thinning is performed by invalidating the thinning target points other than the representative points.

[0129] In step S508, it is determined whether all points constituting the composite point cloud data have been thinned out using the reference points, and steps S501 to S508 are repeated until all points have been thinned out using the reference points (No in step S508). If all points have been thinned out using the reference points, the thinning process ends (Yes in step S508).

[0130] Next, the pixel value adjustment process performed by the point cloud data synthesis unit 403 will be described with reference to Fig. 14. Although combining multiple point cloud data can increase the density of points, variations in pixel values ​​of points may occur, and pixel value adjustment process can be used to appropriately adjust the pixel values.

[0131] In step S601, any valid point in the composite point cloud data is selected as a reference point for the pixel value adjustment process, and the distance d2 between the reference point and a neighboring point is calculated. The neighboring point is not limited to one, and any number of points may be present within a predetermined range based on the reference point.

[0132] Next, in step S602, the distance d2 is compared with a predetermined lower threshold th2 and upper threshold th3, and points that lie within a certain range (th2<d2<th3) are selected as adjustment points. The adjustment point selected in step S602 is not limited to one, and multiple points can be selected as long as they lie within the above-mentioned certain range.

[0133] Next, in step S603, the average value of the pixel values ​​of the adjustment target point is calculated. The average value calculated for each of the R, G, and B elements of the pixel value is used. Note that if there is only one adjustment target point, it is not necessary to calculate the average value.

[0134] Next, in step S604, the average value is applied to all adjustment target points to determine the pixel values ​​of all adjustment target points. Pixel values ​​are adjusted by this application. Then, in step S605, it is determined whether pixel value adjustment processing has been performed using all points of the composite point cloud data as reference points, and steps S601 to S605 are executed until pixel value adjustment processing has been performed using all points as reference points (No in step S605). When pixel value adjustment processing has been performed using all points as reference points, the pixel value adjustment processing ends (Yes in step S605).

[0135] The processing according to the first embodiment of the present technology is performed in the above manner.

[0136] Second Embodiment [Configuration of Point Cloud Data and Attribute Information] A second embodiment of the present technology will be described. When acquiring photographic information using the aviation sensor device 100, the brightness of the image data and the way the light hits it may vary depending on the time of day and the weather. As a result, the brightness of point cloud data generated based on the image data may also differ significantly from that of previous point cloud data. Therefore, in the second embodiment, when multiple point cloud data with different brightnesses are combined, the brightness of the points in the combined point cloud data is prevented from varying.

[0137] 15 shows the structure of point cloud data in which exposure time is added as attribute information to each point that makes up the point cloud data. The exposure time is the exposure time when the image data corresponding to each point that makes up the point cloud data was captured. The exposure time can be saved in the image data at the time of capture as, for example, Exif information.

[0138] [Processing in Point Cloud Data Generation Apparatus 300] The processing of the point cloud data generation apparatus 300 in the second embodiment will be described with reference to Fig. 16. The steps with the same numbers as those in Fig. 10 in the first embodiment perform the same processing, and therefore their description will be omitted.

[0139] After the point cloud data generation unit 302 generates point cloud data in step S306, in step S321, the exposure time is added as attribute information to the multiple points that make up the point cloud data. If there are multiple image data sets from which the point cloud data is generated, the average value of the exposure times of those multiple image data sets is used as the attribute information. Note that when one location is photographed from multiple positions during a single shooting to generate multiple pieces of shooting information, different weather conditions (e.g., sunny and cloudy) may occur during the shooting. In such cases, the exposure time will change depending on the weather conditions, so if there are multiple image data sets from which the point cloud data is generated, it is preferable to use the average value of the exposure times of those multiple image data sets as the attribute information.

[0140] [Processing in Information Processing Device 400] In the second embodiment, similarly to the first embodiment, after the point cloud data synthesis unit 403 of the information processing device 400 synthesizes the point cloud data, the point cloud data synthesis unit 403 performs thinning processing. The thinning processing in the second embodiment will be described with reference to Fig. 17. The same processing is performed in steps assigned the same numbers as in Fig. 13 in the first embodiment, and therefore description thereof will be omitted.

[0141] In steps S501 to S504, thinning-out target points and representative points are selected as in the first embodiment, and then in step S521, points of the thinning-out target points whose exposure time is close to that of the representative points are selected as candidate points.

[0142] Next, in step S522, the pixel values ​​(RGB) of the representative point and candidate points are converted into luminance (YUV) and the maximum and minimum luminance values ​​are calculated.

[0143] Next, in step S523, the ratio of the maximum and minimum brightness values ​​is compared with a predetermined threshold value th4 to determine the degree of brightness variation. If the variation is small (No in step S523), the average values ​​of the pixel values ​​(RGB) of the representative point and candidate points are calculated in step S524.

[0144] On the other hand, if the luminance varies greatly (Yes in step S523), the average luminance (YUV) values ​​of the representative point and candidate points are calculated in step S525.

[0145] Next, in step S526, the average value calculated in step S524 or step S525 is applied to the representative point, thereby correcting the brightness to an average value when thinning out multiple points with large variations in brightness.

[0146] Next, the pixel value adjustment process in the second embodiment will be described with reference to Fig. 18. The same processes are performed in steps with the same numbers as in Fig. 14 in the first embodiment, and therefore the description will be omitted.

[0147] In step S602, points to be adjusted are selected in the same manner as in the first embodiment, and then in step S621, the latest point among the points to be adjusted is selected as a representative point.

[0148] Next, in step S622, points having exposure times close to that of the representative point are selected as candidate points from among the adjustment target points.

[0149] Next, in step S623, the pixel values ​​of the representative points and candidate points are converted from RGB to YUV, and the maximum and minimum luminance values ​​are calculated.

[0150] Next, in step S624, the ratio of the maximum value to the minimum value is compared with a predetermined threshold value th5 to determine the degree of variation in brightness. If the variation is small (No in step S623), the average values ​​of the pixel values ​​(RGB) of the representative point and candidate points are calculated in step S625.

[0151] On the other hand, if the luminance varies greatly (Yes in step S624), the average luminance (YUV) values ​​of the representative point and candidate points are calculated in step S625.

[0152] Next, in step S627, the average value calculated in step S625 or step S626 is applied to the points to be adjusted. When calculating the average value using luminance (YUV), it is possible to reflect the average value calculated for each YUV component in the point cloud, or it is possible to calculate the average value using only the Y component and reflect only the Y component in the point cloud. This allows for correction to an average brightness when combining multiple point cloud data with large variations in brightness.

[0153] Third Embodiment [Configuration of Point Cloud Data Generation Device 300] Next, a third embodiment of the present technology will be described. Since the position of point cloud data generated from imaging information may contain errors, when multiple point cloud data acquired in the same area are combined, the errors may accumulate and the positional accuracy of the combined point cloud data may deteriorate. In the third embodiment, positional errors are calculated when generating point cloud data, and the errors are removed when combining the point cloud data, thereby preventing the positional accuracy of the combined point cloud data from deteriorating.

[0154] 19, the point cloud data generating device 300 of the third embodiment includes a fixed station position acquiring unit 309. The other configurations are the same as those of the first embodiment.

[0155] The fixed station position acquisition unit 309 measures the position of the fixed station by RTK (Real Time Kinematic) positioning based on electronic reference points, with the point cloud data generation device 300 as the fixed station. The point cloud data generation unit 302 performs PPK (Post-Processing Kinematic) positioning of the camera 101 that captured the image data, based on the position information of the fixed station and the position information included in the imaging information transmitted from the aeronautical sensor device 100.

[0156] Furthermore, the point cloud data generation unit 302 generates point cloud data from the image data, position information, and orientation information, and calculates the error from a known verification point on the ground. The attribute information addition unit 307 adds the error calculated by the point cloud data generation unit 302 to the attribute information and point cloud data.

[0157] [Processing in Point Cloud Data Generation Apparatus 300] Next, the processing of the point cloud data generation apparatus 300 in the third embodiment will be described with reference to Fig. 20. Steps with the same numbers as those in Fig. 10 in the first embodiment perform the same processing, and therefore their description will be omitted.

[0158] First, in step S331, the fixed station position acquisition unit 309 measures the position of the fixed station by RTK positioning.

[0159] After step S305, in step S332, the position of the camera 101 is measured by PPK based on the photographing information and the position of the fixed station.

[0160] Next, in step S306, the point cloud data generation unit 302 generates point cloud data. Then, in step S333, the point cloud data generation unit 302 calculates the error between each point constituting the point cloud data and a verification point. A verification point is a point used to confirm whether three-dimensional data such as point cloud data is accurate, and is set in advance before processing. Then, in step S334, the error is added to the point cloud data as attribute information.

[0161] The error information can be the position information (latitude, longitude, altitude) of the verification point and the amount of error in the horizontal direction (x direction, y direction) and vertical direction (z direction) relative to that verification point, or it can be the average value of the amount of error in the x direction, y direction, and z direction for all verification points.

[0162] [Processing in Information Processing Device 400] Next, processing in the information processing device 400 in the third embodiment will be described with reference to Fig. 21. Steps with the same numbers as those in Fig. 11 in the first embodiment perform the same processing, and therefore descriptions thereof will be omitted.

[0163] As described above, error information is added as attribute information to the point cloud data transmitted from the point cloud data generation device 300. Therefore, in step S431, errors are removed from the new point cloud data (t) based on the error information.

[0164] In step S432, errors are removed from the old point cloud data (t-n) based on the error information, and in step S433, the new point cloud data from which the errors have been removed and the old point cloud data from which the errors have been removed are combined to generate combined point cloud data.

[0165] In this way, errors in each point cloud data are removed before combining multiple point cloud data, thereby preventing errors from accumulating in the combined point cloud data and deteriorating positional accuracy. When removing errors from the generated point cloud data, the amount of error at verification points near the point cloud can be subtracted from the position information of the point cloud, or, more simply, the average amount of error at all verification points can be subtracted from the position information of all point clouds. Furthermore, errors can be removed in each of the x-direction, y-direction, and z-direction, or, more simply, errors in only the z-direction can be removed.

[0166] The processing of the present technology is performed as described above. According to the present technology, by combining multiple point cloud data, the density of the point cloud data can be increased without increasing the resolution or overlap rate when capturing images by the camera 101. Increasing the density can improve the accuracy of the point cloud data. Furthermore, the point cloud data for a work area where the land condition is changing can be accurately updated using the position information of the work vehicle 20. Furthermore, missing point cloud data in an area where unwanted objects exist can be compensated for using other point cloud data. Furthermore, thinning out the point cloud data can generate point cloud data with an appropriate density. Furthermore, adjusting the pixel values ​​of the point cloud data can reduce variations in color and brightness of the combined point cloud data. Furthermore, the positional accuracy of the combined point cloud data can be improved using error information from the point cloud data.

[0167] <Modifications> Although the embodiments of the present technology have been specifically described above, the present technology is not limited to the above-described embodiments, and various modifications based on the technical ideas of the present technology are possible.

[0168] In the embodiment, the point cloud data generation device 300 generates the point cloud data and adds the attribute information, but it is also possible to generate the point cloud data and add the attribute information in the information processing device 400. The configuration of the information processing device 400 for this purpose is shown in Fig. 22. In Fig. 22, blocks with the same numbers as those in the point cloud data generation device 300 in Fig. 4 and the information processing device 400 in Fig. 6 are the same and will not be described again.

[0169] The attribute information adding unit 307 adds attribute information to the point cloud data based on information from the point cloud data generating unit 302 , the object identifying unit 303 , and the working area specifying unit 306 , and supplies the point cloud data to a point cloud data saving unit 402 .

[0170] Even with this configuration, it is possible to generate composite point cloud data and transmit it to the display terminal 500 in the same manner as in the embodiment.

[0171] Furthermore, the display terminal 500 may be configured to request only new point cloud data, not composite point cloud data, from the information processing device 400. In this case, the point cloud data synthesis unit 403 only invalidates unnecessary data, and transmits new point cloud data that has not been synthesized with the old point cloud data to the display terminal 500.

[0172] The point cloud data synthesis process in the point cloud data synthesis unit 403 may be executed in response to an instruction from the data request receiving unit 404, or the point cloud data that has been synthesised in advance may be stored, and the stored point cloud data may be supplied to the point cloud data transmission unit 405 in response to an instruction from the data request receiving unit 404.

[0173] If new point cloud data is combined with all point cloud data that was generated earlier than the new point cloud data, point cloud data that is too old may also be combined, which may reduce the accuracy of the combined point cloud data, so it is possible to limit the old point cloud data to be combined, such as data from the past few times.

[0174] In the embodiment, the aircraft sensor device 100 has been described as integrating image data, position information, and attitude information to generate the photography information. However, the aircraft sensor device 100 may transmit the image data, position information, and attitude information to the point cloud data generation device 300, which may then integrate them to generate the photography information.

[0175] In the embodiment, it has been described that the work sensor device 200 generates work information by integrating the operation information and the position information. However, the work sensor device 200 may transmit the operation information and the position information to the point cloud data generation device 300, and the point cloud data generation device 300 may integrate them to generate work information.

[0176] The aircraft 10 as a mobile body equipped with the aeronautical sensor device 100 may be a drone, which is an unmanned aerial vehicle, or a manned aircraft operated by a person, or any other aircraft that can move in the sky, such as a glider, helicopter, balloon, airship, rocket, etc. Furthermore, a mobile body that moves along a rail installed at a high position, such as the ceiling of a building, may be equipped with the mobile body sensor device.

[0177] Furthermore, the work is not limited to being performed by the work vehicle 20, but may also be performed by a non-vehicle work machine operated by a person, such as a compaction machine such as a rammer or plate. The work may also be performed by a person using a shovel or spade. In such cases, the satellite positioning unit 102 may be configured as a standalone device and attached to the worker's body or clothing, or placed near the worker. The operation information acquisition unit 202 may also be a device that captures an image of the worker with the camera 101 and acquires work information through image analysis, or a device that allows a person to input the details of the work being performed by the worker.

[0178] The present technology can also be configured as follows. (1) An information processing device including a point cloud data synthesis unit that performs predetermined processing on multiple point cloud data generated at different times based on imaging information generated by a mobile body sensor device mounted on a mobile body, and synthesizes the multiple point cloud data to generate synthesized point cloud data. (2) The information processing device described in (1) that synthesizes new point cloud data generated most recently with one or more old point cloud data generated earlier than the new point cloud data. (3) The information processing device described in (1) or (2), in which the imaging information includes image data captured by a camera provided in the mobile body sensor device, position information at the time the image data was captured, and attitude information of the camera. (4) The information processing device described in any of (1) to (3), in which a work area to be processed is identified based on work information indicating work performed by a work machine. (5) The information processing device described in (4), in which the work information includes operation information of the work machine performing work to change the state of land by work performed by the work machine, and position information indicating the location where the work is being performed. (6) The information processing device according to (4), wherein the processing invalidates points of the old point cloud data that exist within the working area in the new point cloud data. (7) The information processing device according to (4), wherein information indicating whether the points exist within or outside the working area is added to points constituting the point cloud data. (8) The information processing device according to any of (1) to (7), wherein an area in the point cloud data to be subjected to the processing is identified based on a result of object identification on image data used to generate the point cloud data. (9) The information processing device according to (8), wherein unnecessary objects are predetermined, and points corresponding to the unnecessary objects are identified based on a result of object identification on the image data. (10) The information processing device according to (9), wherein the processing invalidates points corresponding to the unnecessary objects. (11) The information processing device according to (8), wherein the result of the object identification is added to each point constituting the point cloud data. (12) The information processing device according to any of (1) to (11), wherein a thinning process is performed to thin out points constituting the combined point cloud data.(13) An information processing device according to (12), which performs the thinning processing based on an exposure time when image data for generating the point cloud data was captured. (14) An information processing device according to any of (1) to (13), which performs pixel value adjustment processing to adjust pixel values ​​of points constituting the composite point cloud data. (15) An information processing device according to (14), which performs the pixel value adjustment processing based on an exposure time when image data for generating the point cloud data was captured. (16) An information processing device according to any of (1) to (15), which generates the composite point cloud data by removing positional errors of the plurality of point cloud data and combining them. (17) An information processing method, which performs a predetermined process on a plurality of point cloud data generated at different times based on imaging information generated by a mobile body sensor device mounted on a mobile body, and combines the plurality of point cloud data to generate composite point cloud data. (18) An information processing system comprising: a sensor device for a mobile body that is mounted on a mobile body and generates photographic information including image data; a work sensor device that generates work information indicating work performed by a work machine; a point cloud data generation device that generates point cloud data based on the photographic information; and an information processing device that performs predetermined processing on multiple pieces of point cloud data generated at different times and synthesizes the multiple pieces of point cloud data to generate synthesized point cloud data.

[0179] 10. Aircraft 100. Aviation sensor device 200. Work sensor device 300. Point cloud data generating device 400. Information processing device 403. Point cloud data synthesis unit 1000. Information processing system

Claims

1. An information processing device having a point cloud data synthesis unit that performs a predetermined process on multiple point cloud data generated at different times based on shooting information generated by a mobile body sensor device mounted on a mobile body, and synthesizes the multiple point cloud data to generate synthesized point cloud data.

2. The information processing apparatus according to claim 1, wherein new point cloud data generated most recently is synthesized with one or more old point cloud data generated earlier than the new point cloud data.

3. The information processing device according to claim 1, wherein the photographing information includes image data photographed by a camera equipped in the mobile sensor device, position information at the time the image data was photographed, and attitude information of the camera.

4. The information processing device according to claim 1, which identifies a work area to be subjected to said processing based on work information indicating work performed by a work machine.

5. An information processing device according to claim 4, wherein the work information includes operation information of the work machine performing work that changes the condition of the land through work performed by the work machine, and location information indicating the location where the work is being performed.

6. The information processing apparatus according to claim 4, wherein the processing comprises invalidating points of the old point cloud data that exist within the working area in the new point cloud data.

7. The information processing device according to claim 4, wherein information indicating whether said points are present inside or outside said work area is added to the points constituting said point cloud data.

8. The information processing device according to claim 1, further comprising: a processor for processing the point cloud data based on an object identification result for image data used to generate the point cloud data; 9. The information processing device according to claim 8, wherein unwanted objects are defined in advance, and points corresponding to the unwanted objects are identified based on a result of object identification for the image data.

10. The information processing device according to claim 9, wherein the processing comprises invalidating points corresponding to the unnecessary objects.

11. The information processing device according to claim 8, wherein the result of the object identification is added to each point constituting the point cloud data.

12. The information processing device according to claim 1, further comprising a thinning process for thinning out the points constituting the composite point cloud data.

13. The information processing device according to claim 12, wherein the thinning process is performed based on an exposure time when the image data for generating the point cloud data was captured.

14. The information processing device according to claim 1, further comprising a pixel value adjustment process for adjusting pixel values ​​of points constituting the composite point cloud data.

15. The information processing device according to claim 14, wherein the pixel value adjustment process is performed based on an exposure time when the image data for generating the point cloud data was captured.

16. The information processing device according to claim 1, wherein positional errors of the plurality of point cloud data are removed and the composite point cloud data is generated by combining the plurality of point cloud data.

17. An information processing method, comprising: performing a predetermined process on multiple point cloud data generated at different times based on photographing information generated by a mobile body sensor device mounted on a mobile body; and synthesizing the multiple point cloud data to generate synthetic point cloud data.

18. An information processing system comprising: a sensor device for a moving body mounted on a moving body and generating imaging information including image data; a work sensor device generating work information indicating work performed by a work machine; a point cloud data generating device generating point cloud data based on the imaging information; and an information processing device that performs a predetermined processing on a plurality of point cloud data generated at different times, and synthesizes the plurality of point cloud data to generate synthetic point cloud data.

Images