Environmental data generation device, environmental data generation method, and program

Abstract

To ensure the accuracy of object recognition at a construction site and the accuracy of knowing a distance to a recognized object, while ensuring stability and real-time property.SOLUTION: A device has a point group data acquisition part that irradiates light to a plurality of measurement points in the space of a construction environment, receives reflected light from objects, and acquires point group data representing distances between the objects at the plurality of measurement points based on the reflected light, an image data acquisition part that acquires image data from a camera that picks up an area that includes the area to be measured by the distance measurement device in the construction environment, and an environment data generation part that generates environment data to which a label representing the object in the correspondence is added to the terrain map obtained from the point group data if the object recognized by the object recognition processing based on the point group data and the object recognized by the object recognition processing based on the image data are objects in a correspondence.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to an environmental data generating device, an environmental data generating method, and a program. [Background technology]

[0002] For construction machinery using robotic technology to perform autonomous construction work, it is necessary for the construction machinery itself to accurately recognize the construction site environment and determine its own behavior and actions. However, there are cases where the construction machinery's own actions and actions change the shape of the construction site environment, such as when a construction machinery performs civil engineering work such as piling up earth or leveling the piled up earth (embankment). For this reason, even if pre-prepared image data or other environmental data for the construction site is used to autonomously operate an autonomous construction machinery, it is unable to accurately recognize the current environment at the construction site. For example, when operating an autonomous bulldozer, it needs to recognize its positional relationship (e.g., relative distance) with other construction machinery and materials to be buried at the construction site to avoid contact with these objects. Furthermore, when a bulldozer autonomously levels an embankment, it needs to recognize not only the area to be leveled but also the current shape of the embankment and its relative distance from the embankment.

[0003] Conventional technologies for understanding terrain include those that recognize feature points in images taken by stereo cameras or unmanned aerial vehicles (e.g., drones). These technologies can recognize three-dimensional terrain by finding and analyzing feature points from high-resolution images and multiple overlapping image data. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Publication No. 2019-044480 Summary of the Invention [Problem to be solved by the invention]

[0005] However, conventional technology that uses image data captured by a camera attached to construction machinery to recognize the environment at a construction site can recognize objects such as other construction machinery and the shape of embankments, but cannot accurately recognize the distance to these objects or their size. Terrain recognition technology using stereo cameras loses accuracy the farther the object being recognized becomes. Drone terrain recognition technology includes photogrammetry using Structure from Motion (SfM), but this requires large amounts of data processing, is not real-time, and cannot be used for autonomous construction. Furthermore, when the object being imaged is flat terrain, there are fewer feature points, which reduces stability and increases the likelihood of error. Regarding object recognition technology, image-based object recognition has progressed and a certain degree of accuracy in object recognition has been achieved, but there is no technology that can simultaneously recognize relative distance with high accuracy (within a few centimeters). When performing autonomous construction work on-site, it is necessary to grasp the terrain with high accuracy and recognize objects, but conventional technology has issues with accuracy, stability, and real-time performance.

[0006] The present invention has been made in consideration of the above circumstances, and its purpose is to provide an environmental data generation device, an environmental data generation method, and a program that can ensure stability and real-time performance while ensuring the accuracy of object recognition at a construction site and the accuracy of grasping the distance to the recognized object. [Means for solving the problem]

[0007] In order to solve the above-mentioned problems, one aspect of the present invention includes a point cloud data acquisition unit that acquires point cloud data output from a distance measurement device that irradiates light onto a plurality of measurement points in a space of a construction environment, receives light reflected from an object present in the construction environment, and outputs point cloud data representing distances between the plurality of measurement points and the object based on the reflected light; The aforementionedan image data acquisition unit that acquires image data from a camera that captures an area including an area where the distance measurement device performs measurement; Using the first trained model for object recognition Recognized through object recognition processing 1st object, and based on the image data Using a second trained model for object recognition that has different functions from the first trained model Recognized through object recognition processing Second Objects and and determining that the first object and the second object are objects in a corresponding relationship when the direction and distance to the first object and the direction and distance to the second object are the same or within a certain range, and an environmental data generating unit that generates environmental data by adding labels representing the corresponding objects to a topographical map obtained from the point cloud data.

[0008] Another aspect of the present invention is an environmental data generation method executed by a computer, comprising: irradiating light onto a plurality of measurement points within a space of a construction environment; receiving light reflected from an object present in the construction environment; and acquiring point cloud data output from a distance measurement device that outputs point cloud data representing distances between the plurality of measurement points and the object based on the reflected light; and The aforementioned The distance measurement device acquires image data from a camera that captures an area including the area where the measurement is performed, and based on the point cloud data Using the first trained model for object recognition Recognized through object recognition processing 1st object, and based on the image data Using a second trained model for object recognition that has different functions from the first trained model Recognized through object recognition processing Second Objects and and determining that the first object and the second object are objects in a corresponding relationship when the direction and distance to the first object and the direction and distance to the second object are the same or within a certain range, The environmental data generating method includes generating environmental data by adding labels representing the corresponding objects to a topographical map obtained from the point cloud data.

[0009] In one aspect of the present invention, at least a distance measurement device irradiates light onto a plurality of measurement points within a space of a construction environment, receives light reflected from an object present in the construction environment, and outputs point cloud data representing distances between the plurality of measurement points and the object based on the reflected light, and acquires the point cloud data output from the distance measurement device, The aforementioned The distance measurement device acquires image data from a camera that captures an area including the area where the measurement is performed, and based on the point cloud data Using the first trained model for object recognition Recognized through object recognition processing 1st object, and based on the image data Using a second trained model for object recognition that has different functions from the first trained model Recognized through object recognition processing Second Objects and and determining that the first object and the second object are objects in a corresponding relationship when the direction and distance to the first object and the direction and distance to the second object are the same or within a certain range, The program causes a computer to generate environmental data by adding labels representing the corresponding objects to a topographical map obtained from the point cloud data. [Effects of the Invention]

[0010] As described above, according to the present invention, it is possible to ensure stability and real-time performance while ensuring the accuracy of object recognition at a construction site and the accuracy of grasping the distance to the object to be recognized. [Brief explanation of the drawings]

[0011] [Figure 1] 1 is a schematic block diagram showing the configuration of a civil engineering and construction machine 1 using an environmental data generating device. [Figure 2A] This is a conceptual diagram of the construction site as seen from the side. [Figure 2B] FIG. 10 is a diagram showing environmental data KD generated by an environmental data generating device 15. [Figure 2C] FIG. 10 is a diagram showing an example of a mesh map that can be used as environmental data. [Figure 3] 10 is a flowchart showing an example of processing in the environmental data generating device 15. DETAILED DESCRIPTION OF THE INVENTION

[0012] An environmental data generating device according to an embodiment of the present invention will now be described with reference to the drawings. Fig. 1 is a schematic block diagram showing the configuration of a civil engineering and construction machine 1 that uses an environmental data generating device according to an embodiment of the present invention. The construction machine 1 is an autonomous robot, and is heavy machinery used at construction sites, such as a bulldozer, hydraulic excavator, vibrating roller, etc. In the following description, the construction machine 1 will be described as a bulldozer that levels piled up earth (embankment) at a construction site. The construction machine 1 includes a LiDAR (Light Detection and Ranging) 11, a camera 12, a communication device 13, a positioning device 14, an environmental data generating device 15, and an operation control device 16. The configuration shown in Fig. 1 is merely an example, and some of the configuration may be omitted, or other components may be added. For example, the position measurement device 14 may be omitted from the construction machine 1 shown in Fig. 1. Alternatively, sensors such as a speed sensor that detects the speed of the construction machine 1, an acceleration sensor that detects acceleration, a yaw rate sensor that detects the angular velocity around the vertical axis, and a direction sensor that detects the orientation of the construction machine 1 may be added.

[0013] The LiDAR 11 is an example of a distance measurement device, and measures the state of a construction environment. More specifically, at a construction site where construction is performed by the civil engineering and construction machine 1, the LiDAR 11 irradiates light into the space of the construction environment, which is the measurement range of the civil engineering and construction machine 1 (for example, a certain range ahead), and receives reflected light (scattered light) of the irradiated light reflected by an object. The LiDAR 11 measures the distance to an object present in the construction environment based on the time from when the light is emitted to when the reflected light is received. The LiDAR 11 irradiates light to a plurality of measurement points within the construction environment, i.e., the measurement range for measuring the distance to the object, and measures the distance to the object for each measurement point.

[0014] The LiDAR 11 measures the distance to an object at each measurement point, for example, periodically and repeatedly. The light emitted by the LiDAR 11 is, for example, pulsed laser light. The LiDAR 11 outputs point cloud data of the construction environment, which indicates the distance to the object measured at each measurement point, to the environment data generating device 15. The LiDAR 11 is attached to any position of the civil engineering and construction machine 1. For example, the LiDAR 11 is attached to a position that is at a certain height on the civil engineering and construction machine 1 and is unlikely to change position even when a movable part such as an arm is operating.

[0015] The camera 12 is a digital camera that uses a solid-state imaging element such as a CCD (Charge Coupled Device) or a CMOS (Complementary Metal Oxide Semiconductor). The camera 12 periodically and repeatedly captures images of the range of the construction environment (or a range wider than the construction environment). The camera 12 measures the state of the construction environment by capturing images of the range of the construction environment. The camera 12 captures images of an area of ​​the construction environment that includes the area where the LiDAR 11 performs measurement.

[0016] The camera 12 may be a mono camera or a stereo camera. The camera 12 outputs captured image data to the environmental data generating device 15. The camera 12 is attached, for example, in the same location as the LiDAR 15 or at the front of the ceiling in the cockpit. The camera 12 may also be attached at any other location on the construction machine 1. Here, a position on a camera image acquired by the camera 12 is associated in advance with a position in three-dimensional space acquired by the LiDAR 11. That is, a calibration is performed in advance on the position acquired by the camera 12 and the position acquired by the LiDAR 11. Therefore, when measurements are performed on the same object by the camera 12 and the LiDAR 11, the coordinate systems are associated so that the direction and distance to the object identified based on the image data acquired from the camera 12 can be the same as the direction and distance to the object identified based on the point cloud data measured by the LiDAR 11.

[0017] The communication device 13 wirelessly communicates with a management device used by an administrator to determine a construction plan, a server device that manages the construction plan, etc., using, for example, a cellular network or a Wi-Fi network. The communication device 13 outputs information received from outside to the environmental data generating device 15, and transmits various information output from the environmental data generating device 15 to external devices. The communication device 13 may be provided within the environmental data generating device 15.

[0018] The positioning device 14 measures the position of the construction machine 1 based on, for example, a signal received from a GPS (Global Positioning System) satellite. The positioning device 14 outputs information indicating the measured position of the construction machine 1 to the environmental data generating device 15. The environmental data generating device 15 references the information on the position of the construction machine 1 output by the positioning device 14, generates control values ​​for autonomously operating the construction machine 1, and outputs the control values ​​to the construction machine 1. The positioning device 14 may measure the position of the construction machine 1 when the control value generated by the environmental data generating device 15 is a request to measure the position. The positioning device 14 may also be a gyro sensor.

[0019] The environmental data generating device 15 is mounted on the construction machine 1, recognizes the surrounding environment, and generates environmental data according to the recognition results. The objects recognized as the surrounding environment are, for example, objects present in the surrounding area. Examples of objects include embankments, other construction machines, people, materials, etc. FIG. 2 is a diagram showing an example in which the environment data generating device 15 recognizes the surrounding environment. FIG. 2A is a conceptual diagram of a construction site viewed from the side, and FIG. 2B is a diagram showing an example of environmental data KD generated by the environmental data generating device 15. As shown in FIG. The environmental data generating device 15 acquires point cloud data output from the LiDAR 11 and image data output from the camera 12. The environmental data generating device 15 generates environmental data KD by assigning labels LB based on attributes obtained from the image data or point cloud data to a three-dimensional topographical map MP based on the acquired point cloud data. 2A, there is a raised place MB in front of the construction machine 1, and a worker SG is standing on the place MB. The LiDAR 11 measures the area including the place MB and the worker SG to generate point cloud data, and the camera 12 captures an image of the area including the place MB and the worker SG to generate image data. The environmental data KD in Fig. 2B is environmental data obtained by measuring the construction site shown in Fig. 2A, and is a diagram showing an example of a topographical map MP and labels LB assigned to objects (here, workers LB) present at the construction site. This topographical map MP shows the position and shape of a location MB, as well as the position and shape of a worker SG. The topographical map MP is also expressed as a contour diagram according to the shape and height of the location MB and worker SG. The topographical map may also be a mesh map, as shown in Fig. 2C. The mesh map shown in Fig. 2C is a map in which the construction site is viewed from the information perspective, and is divided into meshes of a predetermined size, with a height value assigned to each mesh. The environmental data may be a mesh map in which labels are added to positions on the map corresponding to the positions where objects are measured.

[0020] Returning to FIG. 1, the environmental data generating device 15 includes an acquisition unit 150, a recognition unit 151, an environmental data generating unit 152, and a storage unit 153. The acquisition unit 150, the recognition unit 151, and the environmental data generating unit 152 are realized by, for example, a hardware processor such as a CPU (Central Processing Unit) executing a program (software). Furthermore, some or all of these components may be realized by hardware (including circuitry) such as an LSI (Large Scale Integration), an ASIC (Application Specific Integrated Circuit), an FPGA (Field-Programmable Gate Array), or a GPU (Graphics Processing Unit), or may be realized by a combination of software and hardware. The program may be stored in advance in a storage device (a storage device with a non-transitory storage medium) such as the HDD or flash memory of the environmental data generating device 15, or may be stored in a removable storage medium such as a DVD or CD-ROM, and installed in the HDD or flash memory of the environmental data generating device 15 by inserting the storage medium (non-transitory storage medium) into a drive device, or may be stored in the storage unit 153.

[0021] The acquisition unit 150 acquires various types of information obtained from the outside. The acquisition unit 150 can acquire the results measured by the LiDAR 11, the image data captured by the camera 12, the data received by the communication device 13, and the results of positioning by the positioning device 14. The acquisition unit 150 includes a point cloud data acquisition unit 1501 and an image data acquisition unit 1502 . The point cloud data acquisition unit 1501 acquires the point cloud data output from the LiDAR 11. The point cloud data acquisition unit 1501 outputs the acquired point cloud data to the recognition unit 151. The image data acquisition unit 1502 acquires image data from the camera 12 that captures an area in the construction environment (construction site) including an area where the distance measurement device performs measurement. The image data acquisition unit 1502 outputs the acquired image data to the recognition unit 151.

[0022] The point cloud data acquisition unit 1501 and the image data acquisition unit 1502 sequentially acquire point cloud data periodically output by the LiDAR 11 and image data periodically output by the camera 12, and output them to the recognition unit 151. The point cloud data acquisition unit 1501 and the image data acquisition unit 1502 may store the acquired point cloud data and image data in a memory (not shown). In this case, the recognition unit 151 executes processing related to object recognition based on the point cloud data and image data stored in this memory.

[0023] The recognition unit 151 recognizes an object based on the data acquired by the acquisition unit 150. More specifically, the recognition unit 151 has a first function of recognizing an object based on the point cloud data acquired by the point cloud data acquisition unit 1501, and a second function of recognizing an object based on the image data acquired by the image data acquisition unit 1502.

[0024] When performing object recognition based on point cloud data (first function), the recognition unit 151 may perform the recognition using a model for object recognition. The model for object recognition is a trained model. The trained model is, for example, a model for object recognition by AI (Artificial Intelligence) generated by learning such as deep learning or pattern matching, and is, for example, a first trained model. This first trained model is a model that uses annotated point cloud data as training data and learns the relationship between the point cloud data and objects. This training is performed by a learning device such as a computer. The trained model may be generated by training using a support vector machine (SVM), a decision tree, a k-nn (k-nearest neighbor) classifier, or the like. The training may be a trained model obtained by rule-based learning. When the first trained model is used, the recognition unit 151 inputs the point cloud data acquired by the point cloud data acquisition unit 1501 as input data to the first trained model and performs object recognition based on the output results from the first trained model. The recognition results obtained by the first trained model include, for example, information including at least one of the names and attributes of the construction target object to be constructed, surrounding construction machinery, materials, people, and various other objects existing in the construction environment. These recognition results are obtained, for example, as object attributes. The object to be constructed is, for example, an embankment. The construction machinery present in the surrounding area is, for example, a dump truck, a backhoe, etc. The materials are, for example, pipes, etc. The people are, for example, workers, etc. Others are, for example, steel beams, reinforcing bars, etc. being constructed in the construction environment. In this way, the recognition unit 151 can obtain attributes of the object based on the point cloud data, as well as information such as the distance to the object, height, size, and shape. The first trained model may be a trained model obtained by prior training in a training device and stored in a predetermined storage area so that it can be used by the recognition unit 151.

[0025] Furthermore, when performing object recognition based on image data acquired by the image data acquisition unit 1502 (second function), the recognition unit 151 may perform recognition using an object recognition model. The object recognition model used to realize the second function is, for example, a trained model, but a second trained model different from the first trained model used to realize the first function is used. The recognition unit 151 may also perform object recognition based on image data by other means, such as image analysis using a template. The recognition results obtained by the second trained model include, for example, information including at least one of the names and attributes of the construction target object, surrounding construction machinery, materials, people, and various other objects present in the construction environment. These recognition results are obtained, for example, as object attributes. In this way, the recognition unit 151 can obtain attributes of the object based on the image data, as well as information such as the distance to the object, height, size, and shape. The second trained model may be a trained model obtained by prior training in a training device and stored in a predetermined storage area so that it can be used by the recognition unit 151.

[0026] When an object recognized by performing object recognition processing based on point cloud data corresponds to an object recognized by performing object recognition processing based on image data, the environmental data generation unit 152 generates environmental data by adding labels representing the corresponding objects to a topographical map obtained from the point cloud data.

[0027] When determining whether there is a correspondence, the environmental data generation unit 152 may assign attributes based on image data as labels to a terrain map based on point cloud data if the direction and distance to an object recognized based on the first function and the direction and distance to an object recognized based on the second function are the same or within a certain range. In this case, the distance to an object and its shape can be determined using the LiDAR 11, and the recognition result of the object can be determined using image data, so that attributes recognized with high accuracy can be integrated as labels into a terrain map measured with high accuracy.

[0028] Furthermore, when determining whether or not there is a correspondence, the environmental data generation unit 152 may determine that an object recognized by performing object recognition processing based on point cloud data and an object recognized by performing object recognition processing based on image data are in the same direction relative to the civil engineering and construction machine 1 and the attributes obtained by the object recognition processing match. As a result, even if an object is recognized based on image data and the distance to the object is determined, if the object is located at a certain distance from the camera 12, the direction can be determined even if the distance determined based on the image data cannot be accurately determined. On the other hand, if the distance to an object is determined using point cloud data and the object is recognized, even if the object recognition accuracy is insufficient, the distance to the object can be measured with high accuracy by integrating attributes obtained from the image data into a topographical map based on the point cloud data, thereby improving the accuracy of object recognition.

[0029] In addition, when the environmental data generation unit 152 determines whether an object recognized by performing object recognition processing based on point cloud data and an object recognized by performing object recognition processing based on image data are in the same direction, it may determine that the directions in which the objects exist are the same if the directions in which the objects exist are within a certain range.

[0030] Furthermore, the environment data generation unit 152 may determine that there is a correspondence relationship when the direction and shape recognized based on the first function and the direction and shape recognized based on the second function are the same, respectively. When determining whether the shapes are the same, the environment data generation unit 152 may determine that there is a correspondence relationship when the outline of an object measured based on point cloud data and the outline of an object recognized based on image data are similar to each other, and generate environment data. Regarding whether there is a similarity relationship, the degree of similarity may be calculated, and if the degree is equal to or greater than a predetermined reference value, it may be determined that there is a similarity relationship.

[0031] The environmental data generation unit 152 may generate environmental data for a terrain map generated based on point cloud data, in which attributes of objects present in the terrain map are associated with the distance and height to the objects. The environmental data may also include data representing the shape of the objects. The distance and height to the objects can be obtained from the distance and height data represented by the point cloud data corresponding to the recognized object.

[0032] The environment data generating unit 152 can sequentially generate environment data every time the object information is updated by the recognizing unit 151 . The environmental data generation unit 152 outputs the generated environmental data to the operation control device 16.

[0033] The operation control device 16 controls the operation of the civil engineering and construction machine 1. For example, based on the environmental data output from the environmental data generation unit 152, the operation control device 16 generates control values ​​for controlling the operation of the civil engineering and construction machine 1 when the civil engineering and construction machine 1 autonomously performs construction operations, and controls the civil engineering and construction machine 1 based on the control values. The control values ​​are control information that represent, for example, instructions for causing the civil engineering and construction machine 1 to travel (move) within a construction site or instructions for causing the civil engineering and construction machine 1 to perform construction work at the location of a target object.

[0034] The processing in the environmental data generating device 15 will now be described. Fig. 3 is a flowchart showing an example of the processing in the environmental data generating device 15. The processing of this flowchart is started when the civil engineering and construction machine 1 starts autonomous operation. In the following explanation, it is assumed that the civil engineering and construction machine 1 has already arrived at the position (location) of the object where construction work will be performed.

[0035] When the construction machine 1 starts autonomous operation, the environmental data generating device 15 acquires the determined construction plan via the communication device 13 (step S201). In addition, as the construction machine 1 starts operating, the LiDAR 11 starts measuring, and the camera 12 starts capturing images. Next, the acquisition unit 150 of the environmental data generating device 15 acquires point cloud data from the LiDAR 11 (step S102), and acquires image data from the camera 12 (step S103). Either the acquisition of the point cloud data or the acquisition of the image data may be performed first.

[0036] The recognition unit 151 inputs the point cloud data obtained by the acquisition unit 150 as input data to a first learned model, and performs object recognition based on the output result from the first learned model (step S103). Also, the recognition unit 151 inputs the image data obtained by the acquisition unit 150 as input data to a second learned model, and performs object recognition based on the output result from the second learned model (step S104).

[0037] After the object recognition process is performed by the recognition unit 151, the environmental data generation unit 152 identifies attributes corresponding to the objects based on the object recognition results obtained based on the point cloud data and the object recognition results obtained based on the image data, and generates environmental data by associating labels corresponding to the identified attributes with a topographical map based on the point cloud data (step S105). After the environmental data is generated, the environmental data generation unit 152 outputs the environmental data to the driving control device 16.

[0038] The operation control device 16 controls the civil engineering and construction machine 1 based on the environmental data obtained from the environmental data generation unit 152 (step S106). For example, the operation control device 16 compares the construction plan obtained in step S101 with the environmental data generated in step S105, that is, compares the state of the construction environment represented by the construction plan with the current state of the construction environment represented by the environmental data. The operation control device 16 then generates a control value based on the result of comparing the construction plan with the environmental data, and controls the civil engineering and construction machine 1 to move (for example, move to a target embankment) or perform construction work (for example, leveling an embankment) at the target location according to the generated control value. This allows the civil engineering and construction machine 1 to operate autonomously. Furthermore, since the environmental data is labeled, it is possible to grasp not only the topographical map but also the location of objects on the topography. These labels are the result of integrating the results of object recognition based on point cloud data and object recognition based on image data, thereby improving the accuracy of object recognition. Furthermore, since the environmental data is labeled for the topographical map based on point cloud data, it is possible to obtain an accurate topographical map of the construction site environment, and the results of accurate object recognition can be used. Furthermore, since it is possible to grasp the location of recognized objects on the topographical map, the civil engineering and construction machine 1 can accurately grasp the location of the construction target (e.g., embankment) and carry out construction. Furthermore, if other civil engineering and construction machines or people are detected, it is possible to calculate a movement path that avoids interference (collision) and control the attitude of the civil engineering and construction machine 1.

[0039] Thereafter, the environmental data generating device 15 determines whether or not to terminate the processing (step S107), and if the operation of the civil engineering and construction machine 1 is continuing, it determines not to terminate the processing (step S107-NO) and proceeds to step S102, and if it determines to continue the processing, it terminates the processing (step S107-YES).

[0040] According to the embodiment described above, it is possible to ensure stability and real-time performance while ensuring the accuracy of object recognition at a construction site and the accuracy of grasping the distance to the recognition target. More specifically, since a topographical map is generated from point cloud data obtained by the LiDAR 11, it is possible to accurately grasp the distance and size of an object, and attributes can be identified using at least the result of object recognition based on image data, so it is possible to ensure the accuracy of object recognition. Furthermore, since a terrain map and environmental data are generated from point cloud data obtained by the LiDAR 11, there is no need to use a large amount of photographic data as in SfM, and real-time performance is not reduced. Furthermore, even if the subject being imaged is flat, the terrain can be grasped using the LiDAR 11, so there is no need to extract feature points from photographs as in SfM, which reduces the reduction in stability and increase in error caused by the number of feature points.

[0041] In the embodiment described above, it is assumed that the civil engineering and construction machine 1 is a bulldozer that levels embankments at a construction site, but the civil engineering and construction machine 1 may be a construction machine other than a bulldozer. For example, even if the civil engineering and construction machine 1 is a vibrating roller that autonomously performs construction work of compacting earth and sand with a roller, the environmental data generating device 15 can generate environmental data (for example, a map of the construction site) that is tailored to the target construction work.

[0042] Although the above-described embodiment is directed to a construction environment for civil engineering work in which construction machinery such as bulldozers is used, the construction environment for civil engineering work may be other environments. For example, the construction environment for civil engineering work may be a tunnel construction site for reducing accidents inside the tunnel.

[0043] Furthermore, in the above-described embodiment, the case where the civil engineering and construction machine 1 operates autonomously has been described. However, when the civil engineering and construction machine 1 operates autonomously, an operator may or may not be on board the civil engineering and construction machine 1. When an operator is on board the civil engineering and construction machine 1, the operator may operate the civil engineering and construction machine 1 as needed by operating driving controls provided on the civil engineering and construction machine 1. For example, the operator may operate control objects such as a blade, arm, or baguette, and perform operations such as moving, stopping, and rotating. In this case, environmental data may be displayed on a display device provided on the civil engineering and construction machine 1. This allows the operator to understand the surrounding environment by referring to the environmental data even when visibility around the machine is poor (dark, bad weather, etc.).

[0044] The environmental data generating device 15 in the above-described embodiment may be implemented by a computer. In this case, a program for implementing the functions may be recorded on a computer-readable recording medium, and the program may be loaded into a computer system and executed. Note that the term "computer system" as used herein includes hardware such as an OS and peripheral devices. Furthermore, the term "computer-readable recording medium" refers to portable media such as flexible disks, optical magnetic disks, ROMs, and CD-ROMs, as well as storage devices such as hard disks built into a computer system. Furthermore, the term "computer-readable recording medium" may also include devices that dynamically store programs for a short period of time, such as communication lines used when transmitting programs via networks such as the Internet or telephone lines, or devices that store programs for a fixed period of time, such as volatile memory within a computer system serving as a server or client. The program may be for implementing only a portion of the functions described above, or may be capable of implementing the functions in combination with a program already stored in the computer system, or may be implemented using a programmable logic device such as an FPGA (Field Programmable Gate Array).

[0045] Although an embodiment of the present invention has been described above in detail with reference to the drawings, the specific configuration is not limited to this embodiment, and includes designs within the scope of the gist of the present invention. [Explanation of symbols]

[0046] 1... civil engineering and construction machine, 11... LiDAR, 12... camera, 13... communication device, 14... positioning device, 15... environmental data generation device, 16... operation control device, 150... acquisition unit, 151... recognition unit, 152... environmental data generation unit, 153... storage unit, 1501... point cloud data acquisition unit, 1502... image data acquisition unit

Claims

1. a point cloud data acquisition unit that acquires point cloud data output from a distance measurement device that irradiates light onto a plurality of measurement points in a space of a construction environment, receives light reflected from an object present in the construction environment, and outputs point cloud data representing distances between the plurality of measurement points and the object based on the reflected light; and an image data acquisition unit that acquires image data from a camera that captures an area in the construction environment that includes an area where the distance measurement device performs measurement; an environmental data generation unit that determines, for a first object recognized by performing object recognition processing using a first trained model for object recognition based on the point cloud data, and a second object recognized by performing object recognition processing using a second trained model for object recognition that has a function different from that of the first trained model based on the image data, that the first object and the second object are objects in a corresponding relationship if a direction and distance to the first object and a direction and distance to the second object are the same or within a certain range, and generates environmental data in which labels representing the corresponding objects are added to a topographical map obtained from the point cloud data; An environmental data generating device having:

2. The environmental data generation unit Based on the point cloud data, environmental data is generated in which the distance and height to an object are associated with a label representing the object. The environmental data generating device according to claim 1 .

3. The environmental data generating device It is mounted on civil engineering and construction machinery operating in the construction environment.

3. The environmental data generating device according to claim 1.

4. 1. A computer-implemented method for generating environmental data, comprising: At least, a distance measurement device irradiates light onto a plurality of measurement points in a space of a construction environment, receives light reflected from an object present in the construction environment, and outputs point cloud data representing distances between the plurality of measurement points and the object based on the reflected light, and acquires the point cloud data output from the distance measurement device; Acquire image data from a camera that captures an area in the construction environment that includes an area where the distance measurement device performs measurement; For a first object recognized by performing object recognition processing using a first trained model for object recognition based on the point cloud data, and a second object recognized by performing object recognition processing using a second trained model for object recognition that has a different function from the first trained model based on the image data, if the direction and distance to the first object and the direction and distance to the second object are the same or within a certain range, the first object and the second object are determined to be objects in a corresponding relationship, and environmental data is generated by adding labels representing the corresponding objects to a topographical map obtained from the point cloud data. An environmental data generation method comprising:

5. At least, a distance measurement device irradiates light onto a plurality of measurement points in a space of a construction environment, receives light reflected from an object present in the construction environment, and outputs point cloud data representing distances between the plurality of measurement points and the object based on the reflected light, and acquires the point cloud data output from the distance measurement device; Acquire image data from a camera that captures an area in the construction environment that includes an area where the distance measurement device performs measurement; For a first object recognized by performing object recognition processing using a first trained model for object recognition based on the point cloud data, and a second object recognized by performing object recognition processing using a second trained model for object recognition that has a different function from the first trained model based on the image data, if the direction and distance to the first object and the direction and distance to the second object are the same or within a certain range, the first object and the second object are determined to be objects in a corresponding relationship, and environmental data is generated by adding labels representing the corresponding objects to a topographical map obtained from the point cloud data. A program that makes a computer do something.

Images