OBJECT RECOGNITION DEVICE, ENVIRONMENTAL DATA GENERATION DEVICE, MODEL GENERATION DEVICE, OBJECT RECOGNITION METHOD, AND PROGRAM

Abstract

To provide an object recognition device capable of enhancing accuracy of object recognition at a construction site, an environment data generation device, a model generation device, an object recognition method, and a program.SOLUTION: An object recognition device includes a model acquisition unit for acquiring a model obtained by learning using a data set including point cloud data generated by a point cloud data generation device that generates point cloud data in a construction environment of civil engineering work, and a recognition unit for recognizing an object existing in the construction environment using the acquired model.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to an object recognition device, an environmental data generation device, a model generation device, an object recognition method, and a program. [Background technology]

[0002] In recent years, with the aim of reducing the number of workers on site, progress has been made in developing autonomous construction robots at construction sites. To achieve autonomous construction by robots, it is necessary for them to correctly recognize the environment of the construction site. However, at construction sites, the environmental conditions change as work progresses, such as when excavated soil is piled up, making it difficult to prepare a static map in advance and have the robot recognize the environment. In particular, in the case of autonomous construction by bulldozers, it is necessary to recognize the relative distance to objects in order to avoid contact with other construction machinery or drainage pipes. In addition, it is also necessary to recognize the shape and relative distance of the embankment in order to determine where to level the embankment and to control the leveling.

[0003] Conventionally, in order to recognize the environment of a construction site, the environment has been recognized using images captured by a camera, for example (see, for example, Patent Document 1). However, while recognition using camera images can recognize objects and embankments, it is difficult to accurately measure the distance to the target object or the size of that object. Furthermore, the recognition accuracy when using camera images is greatly affected by external factors such as weather. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Publication No. 2019-044480 Summary of the Invention [Problem to be solved by the invention]

[0005] To address these issues, one method for accurately measuring the size and relative distance of an object is to develop artificial intelligence (AI) and recognize objects using point cloud data measured by 3D-LiDAR (3D-Light Detection and Ranging, hereafter referred to as LiDAR). By using point cloud data for object recognition, it is possible to calculate the size, shape, and relative distance of a recognized object from the set of recognized points. This makes it possible to use the size, shape, and relative distance of an object to determine control values ​​for autonomous behavior of construction machinery.

[0006] To build an AI that can recognize point cloud data, it is necessary to prepare a large amount of point cloud data as training data, with each point annotated with information such as construction machinery or obstacles. However, there is no large amount of point cloud data annotated with information such as construction machinery or obstacles, and annotating large amounts of point cloud data requires a great deal of effort. This makes it difficult to build an AI and improve the accuracy of object recognition.

[0007] The present invention has been made based on the recognition of the above-mentioned problems, and aims to provide an object recognition device, an environmental data generation device, a model generation device, an object recognition method, and a program that can improve the accuracy of object recognition at construction sites. [Means for solving the problem]

[0008] The object recognition device, the environment data generation device, the model generation device, the object recognition method, and the program according to the present invention employ the following configuration. (1): An object recognition device according to one embodiment of the present invention is an object recognition device including: a model acquisition unit that acquires a model obtained by learning using a dataset including point cloud data generated by a point cloud data generation device that generates point cloud data in a construction environment of civil engineering work; and a recognition unit that recognizes objects present in the construction environment using the acquired model.

[0009] (2): In the above aspect (1), the point cloud data generating device includes a simulator that simulates the construction environment, and the simulator generates the point cloud data based on the object.

[0010] (3): In the above aspect (2), the data set includes construction environment data regarding the object tagged with tag data in the simulation of the simulator as input data of the model, and includes annotation data assigned to each point of the point cloud data based on the construction environment data as output data of the model.

[0011] (4): Furthermore, an environmental data generation device according to one aspect of the present invention is an environmental data generation device that includes an object recognition device according to any one of (1) to (3) above, and an environmental data generation unit that generates environmental data representing the state of the construction environment based on the object recognized by the object recognition device.

[0012] (5): In addition, a model generation device according to one aspect of the present invention is a model generation device that includes a learning unit that generates a model by learning using construction environment data of a construction environment of civil engineering work and point cloud data generated by a point cloud data generation device based on the construction environment data as a dataset, wherein the dataset includes, as input data for the model, construction environment data regarding objects tagged with tag data in a simulation of a simulator that virtualizes the construction environment, and includes, as output data for the model, annotation data assigned to each point of the point cloud data based on the construction environment data.

[0013] (6): Furthermore, an object recognition method according to one aspect of the present invention is an object recognition method in which a computer of an object recognition device acquires a model obtained by learning using a dataset including point cloud data generated by a point cloud data generation device that generates point cloud data in a construction environment of civil engineering work, and recognizes objects present in the construction environment using the acquired model.

[0014] (7): Furthermore, a program according to one aspect of the present invention is a program that causes a computer of an object recognition device to acquire a model obtained by learning using a dataset including point cloud data generated by a point cloud data generation device that generates point cloud data in a construction environment of civil engineering work, and recognizes objects present in the construction environment using the acquired model. [Effects of the Invention]

[0015] According to the above aspects (1) to (7), it is possible to improve the accuracy of object recognition at a construction site. [Brief explanation of the drawings]

[0016] [Figure 1] 1 is a configuration diagram of an environmental data generation system 1 according to an embodiment. [Figure 2] FIG. 10 is a diagram showing an image of how the environmental data generating device 100 recognizes an embankment. [Figure 3] 10 is a flowchart showing an example of processing in the model generation device 200. [Figure 4] 4 is a flowchart showing an example of processing in the environmental data generating device 100. [Figure 5] FIG. 2 is a diagram illustrating an example of a hardware configuration of an environmental data generating apparatus. DETAILED DESCRIPTION OF THE INVENTION

[0017] Hereinafter, embodiments of an object recognition device, an environment data generation device, a model generation device, an object recognition method, and a program according to the present invention will be described with reference to the drawings.

[0018] [Overall configuration] 1 is a configuration diagram of an environmental data generation system 1 according to an embodiment. The environmental data generation system 1 includes, for example, an environmental data generation device 100 and a model generation device 200. The environmental data generation device 100 is mounted on, for example, a construction machine 10. The construction machine 10 and the model generation device 200 are capable of communicating with each other via a communication network NW.

[0019] The construction machine 10 on which the environmental data generating device 100 is mounted is, for example, a bulldozer, a hydraulic excavator, a vibrating roller, etc. In the following explanation, the construction machine 10 on which the environmental data generating device 100 is mounted is assumed to be a bulldozer that levels piled up soil (embankment) at a construction site.

[0020] The construction machine 10 includes, for example, a driving operator 11, a driving control device 12, a traveling driving force output device 13, a construction driving force output device 14, a LiDAR 15, a camera 16, a communication device 17, a positioning device 18, and an environmental data generating device 100. The configuration shown in Fig. 1 is merely an example, and some of the configuration may be omitted, or other components may be added.

[0021] For example, the position measurement device 18 may be omitted from the construction machine 10 shown in Fig. 1. Alternatively, sensors such as a speed sensor that detects the speed of the construction machine 10, an acceleration sensor that detects acceleration, a yaw rate sensor that detects the angular velocity around the vertical axis, and a direction sensor that detects the orientation of the construction machine 10 may be added.

[0022] The driving operators 11 are operators that are used by workers at a construction site to manually control the operation of the civil engineering and construction machine 10. The driving operators 11 include a plurality of operators for operating the civil engineering and construction machine 10, such as operators that are operated when traveling (moving) the civil engineering and construction machine 10 and operators that are operated when performing construction work using the construction equipment provided in the civil engineering and construction machine 10.

[0023] The operator is, for example, an operating lever, a joystick, a hand grip, an operating button, etc. A sensor is attached to the driving operator 11 to detect the amount of operation of the operator or whether or not the operator is operated, and the detection results are output to the driving control device 12 or to some or all of the traveling driving force output device 13 and the construction driving force output device 14.

[0024] When the civil engineering and construction machine 10 is manually operated for construction by a worker at the construction site, the operation control device 12 controls the traveling driving force output device 13 and the construction driving force output device 14 in accordance with the detection results output by the sensors attached to the driving operator 11. On the other hand, when the civil engineering and construction machine 10 is an autonomous construction machine and performs construction operations autonomously, the operation control device 12 controls the traveling driving force output device 13 and the construction driving force output device 14 in accordance with the environmental data and control values ​​output by the environmental data generation device 100.

[0025] The traveling drive force output device 13 outputs a traveling drive force (torque) to, for example, a crawler or a wheel, for the civil engineering / construction machine 10 to travel within a construction site. The traveling drive force output device 13 includes, for example, an internal combustion engine and a control device. The control device controls the internal combustion engine according to information input from the operation control device 12 or information input from the driving operator 11.

[0026] When construction work is performed at a construction site by the civil engineering and construction machine 10, the construction driving force output device 14 outputs the driving force (torque) of the internal combustion engine as a driving force for generating hydraulic pressure in a cylinder of a hydraulic device connected to a work device such as a blade, arm, baguette, etc. The construction driving force output device 14 outputs driving force during construction in accordance with information input from the operation control device 12 or information input from the operation operator 11.

[0027] At a construction site where construction is being performed by the civil engineering and construction machine 10, the LiDAR 15 irradiates light into a space in the construction environment, which is a predetermined range in front of the civil engineering and construction machine 10, and receives reflected light (scattered light) from objects that reflect the irradiated light. The LiDAR 15 measures the distance to an object in the construction environment based on the time between irradiating the light and receiving the reflected light. The LiDAR 15 irradiates light to a plurality of measurement points within the construction environment, i.e., a measurement range for measuring the distance to an object, and measures the distance to the object for each measurement point. In this way, the LiDAR 15 measures the state of the construction environment.

[0028] The LiDAR 15 measures the distance to an object at each measurement point, for example, periodically and repeatedly. The light emitted by the LiDAR 15 is, for example, pulsed laser light. The LiDAR 15 outputs point cloud data of the construction environment, which indicates the distance to the object measured at each measurement point, to the environment data generating device 100. The LiDAR 15 is attached to any position of the civil engineering and construction machine 10. For example, the LiDAR 15 is attached to the roof of the operator's cabin where the worker sits.

[0029] The camera 16 is a digital camera that uses a solid-state imaging element such as a CCD (Charge Coupled Device) or a CMOS (Complementary Metal Oxide Semiconductor). The camera 16 captures images of the range of the construction environment (or a range wider than the construction environment) repeatedly, for example, periodically. The camera 16 measures the state of the construction environment by capturing images of the range of the construction environment.

[0030] The camera 16 may be a mono camera or a stereo camera. The camera 16 outputs captured image data to the environmental data generating device 100. The camera 16 is attached, for example, in the same location as the LiDAR 15 or at the front of the ceiling in the cockpit. The camera 16 may also be attached at any other location on the construction machine 10.

[0031] The communication device 17 communicates with a management device used by an administrator to determine a construction plan, a server device that manages the construction plan, and the like, using, for example, a cellular network or a Wi-Fi network. The communication device 17 receives, for example, information transmitted by the model generation device 200. The communication device 17 outputs the received information to the environmental data generation device 100. The communication device 17 may be provided in the environmental data generation device 100.

[0032] The positioning device 18 measures the position of the construction machine 10 based on, for example, a signal received from a GPS (Global Positioning System) satellite. The positioning device 18 outputs information indicating the measured position of the construction machine 10 to the environmental data generating device 100. The environmental data generating device 100 references the information on the position of the construction machine 10 output by the positioning device 18, generates control values ​​for autonomously operating the construction machine 10, and outputs the control values ​​to the construction machine 10. The positioning device 18 may measure the position of the construction machine 10 in response to a request from the control values ​​generated by the environmental data generating device 100.

[0033] The environmental data generating device 100 mounted on the civil engineering and construction machine 10 recognizes surrounding objects, for example, embankments. FIG. 2 is a diagram illustrating an image of the environmental data generating device 100 recognizing an embankment. The environmental data generating device 100 acquires point cloud data output by the LiDAR 15 and image data output by the camera 16. The environmental data generating device 100 inputs the acquired point cloud data into a trained model, and recognizes objects such as an embankment M based on the output results, as well as the distance to the embankment M and the height of the embankment M. The trained model is an example of a model. The environmental data generating device 100 may input image data together with the point cloud data into the trained model, and recognize the object based on the output results.

[0034] The environmental data generating device 100 includes, for example, an object recognition unit 110, an environmental data generation unit 120, and a control value generation unit 130. The object recognition unit 110, the environmental data generation unit 120, and the control value generation unit 130 are realized by, for example, a hardware processor such as a CPU (Central Processing Unit) executing a program (software). Furthermore, some or all of these components may be realized by hardware (including circuitry) such as an LSI (Large Scale Integration), an ASIC (Application Specific Integrated Circuit), an FPGA (Field-Programmable Gate Array), or a GPU (Graphics Processing Unit), or may be realized by a combination of software and hardware. The program may be stored in advance in a storage device (a storage device with a non-transitory storage medium) such as the HDD or flash memory of the environmental data generating apparatus 100, or may be stored in a removable storage medium such as a DVD or CD-ROM, and installed in the HDD or flash memory of the environmental data generating apparatus 100 by inserting the storage medium (non-transitory storage medium) into a drive device.

[0035] The object recognition unit 110 includes, for example, an acquisition unit 111 and a recognition unit 112. The acquisition unit 111 includes, for example, a model acquisition unit 111A, a point cloud data acquisition unit 111B, and an image data acquisition unit 111C. The model acquisition unit 111A acquires a trained model transmitted by the model generation device 200 and received by the communication device 17. The object recognition unit 110 is an example of an object recognition device. In the embodiment, the object recognition device is included in the environment data generation device 100 as the object recognition unit 110, but the object recognition device may be provided independently from the environment data generation device 100.

[0036] The point cloud data acquisition unit 111B acquires point cloud data output by the LiDAR 15. The point cloud data acquisition unit 111B outputs the acquired point cloud data to the recognition unit 112. The image data acquisition unit 111C acquires image data output by the camera 16. The image data acquisition unit 111C outputs the acquired image data to the recognition unit 112. The point cloud data acquisition unit 111B and the image data acquisition unit 111C sequentially acquire point cloud data periodically output by the LiDAR 15 and image data periodically output by the camera 16, and output them to the recognition unit 112. The point cloud data acquisition unit 111B and the image data acquisition unit 111C may store the acquired point cloud data and image data in a memory (not shown). In this case, the recognition unit 112 executes the following functions based on the point cloud data and image data stored in the memory.

[0037] The recognition unit 112 inputs the point cloud data acquired by the point cloud data acquisition unit 111B as input data into the trained model acquired by the model acquisition unit 111A. The recognition unit 112 performs object recognition based on the output result of the trained model to which the point cloud data has been input. Through object recognition, the recognition unit 112 recognizes objects present in the construction environment, such as construction targets for embankments and other construction work, other construction machinery, and materials.

[0038] The recognition unit 112 performs object recognition based on the image data acquired by the image data acquisition unit 111C. Object recognition based on the image data is performed, for example, by inputting the image data into a second trained model. The recognition unit 112 may also perform object recognition based on the image data by other means, such as image analysis using a template. The recognition unit 112 performs object recognition based on the image data based on the output result of the second trained model. The recognition unit 112 corrects the result of object recognition based on the output result of the trained model using the result of object recognition based on the image data. The recognition unit 112 outputs object information, in which at least information about the object recognized in the corrected recognition result is linked to point cloud data, to the environment data generation unit 120. The object information includes, for example, information about the relative distance from the object measured by the LiDAR 15, as well as information about the attributes, size, height, shape, and so on of the object recognized by the recognition unit 112. The attribute of the object is information indicating, for example, whether the object is an embankment or a change in soil other than an embankment, that is, a change in the topography of the construction site.

[0039] The object information may include information indicating the attributes of the object, such as the type of other construction machinery, materials, etc., recognized by the recognition unit 112, as well as the relative distance to the other construction machinery, materials, etc., recognized by the recognition unit 112. The recognition unit 112 executes the above-mentioned object recognition function every time point cloud data is output from the point cloud data acquisition unit 111B, sequentially updates the object information, and outputs it to the environment data generation unit 120. The recognition unit 112 may recognize objects in the construction environment using the point cloud data acquired by the point cloud data acquisition unit 111B and the image data acquired by the image data acquisition unit 111C as input data.

[0040] The environmental data generation unit 120 generates environmental data representing the state of the construction environment based on the object information output by the recognition unit 112 included in the object recognition unit 110. The environmental data is, for example, a map of the construction site that includes the distance between the civil engineering construction machine 10 and the object to be worked on, i.e., information on the position within the construction site where the object exists, and information on the positions of other objects that are not the target of the construction work, such as other construction machines and materials, that exist within the construction site. The environmental data generation unit 120 outputs the generated environmental data to the control value generation unit 130. Every time the object information is updated by the recognition unit 112, the environmental data generation unit 120 sequentially updates the environmental data and outputs it to the control value generation unit 130.

[0041] The control value generation unit 130 generates control values ​​for controlling the operation of the civil engineering and construction machine 10 when the civil engineering and construction machine 10 autonomously performs construction operations, based on the environmental data output by the environmental data generation unit 120. The control values ​​are control information that represent, for example, instructions for causing the civil engineering and construction machine 10 to travel (move) within a construction site or instructions for causing the civil engineering and construction machine 10 to perform construction work at the location of a target object. The control value generation unit 130 outputs the generated control values ​​to the operation control device 12. The operation control device 12 controls the traveling driving force output device 13 and the construction driving force output device 14 in accordance with the control values ​​input from the control value generation unit 130.

[0042] The model generating device 200 includes, for example, a point cloud data generating unit 210 and a learning unit 220. The point cloud data generating unit 210 includes, for example, an input interface 211 and a simulation unit 212. The point cloud data generating unit 210 generates a dataset including point cloud data used for learning in the learning unit 220. The point cloud data generating unit 210 is an example of a point cloud data generating device.

[0043] The input interface 211 outputs data provided based on input operations by a worker or the like to the simulation unit. As the input interface 211, for example, a mouse, a keyboard, a touch panel, a drag ball, a switch, a button, a joystick, a camera, an infrared sensor, a microphone, etc. may be used.

[0044] The simulation unit 212 executes a simulation based on the reproduction data of the construction site provided based on the input operation of the input interface 211. By executing the simulation, the simulation unit 212 generates construction environment data that reproduces a virtual construction site based on the reproduction data of the construction site.

[0045] Examples of the reproduction data of the construction site include 3D data created by BIM (Building Information Modeling) / CIM (Construction Information Modeling) and 3D point cloud map data measured by a drone, etc. The construction environment data may be a construction site based on the construction environment data generated by the environment data generating device 100 instead of a virtual construction site.

[0046] In the construction site simulation, there are multiple objects, such as embankments and civil engineering machinery. Each object in the simulation is tagged with attribute information. The construction environment data includes the attribute information tagged to the object.

[0047] The simulation unit 212 executes a simulation of measuring the reproduced construction site using 3DLiDAR based on the generated construction environment data, and generates point cloud data as a result of the simulation. The simulation unit 212 adds annotations to the generated point cloud data based on attribute information tagged with each of the multiple objects. The attribute information is an example of tag data.

[0048] The simulation unit 212 recreates a large number of construction sites assuming various situations. The simulation unit 212 recreates a large number of construction sites, and for each of the construction sites, executes a simulation similar to the above-mentioned simulation using construction environment data, and generates point cloud data with annotations.

[0049] The simulation unit 212 generates a large amount of annotated point cloud data as a result of executing the simulation. The point cloud data generation unit 210 outputs a dataset including a large amount of construction environment data in which objects generated by the simulation unit 212 are tagged with attribute information and a large amount of annotated point cloud data to the learning unit 220. The simulation unit 212 is an example of a simulator. The simulator (simulation unit) may be provided independently of the point cloud data generation unit 210 and the model generation device 200.

[0050] The learning unit 220 performs machine learning by learning a large amount of construction environment data and point cloud data generated by the simulation unit 212 as a data set. The learning unit 220 performs machine learning by using, for example, construction environment data tagged with attribute information as input data and point cloud data to which annotations are added as output data. The learning unit 220 generates a trained model as a result of the machine learning. The learning unit 220 transmits the generated trained model to the environmental data generating device 100.

[0051] The trained model is a model used for object recognition in the environmental data generating device 100. The trained model is a model for object recognition by AI (Artificial Intelligence) generated by learning such as deep learning or pattern matching. The trained model may be generated by learning a support vector machine (SVM), a decision tree, a k-nn (k-nearest neighbor) classifier, or the like. The learning may be rule-based learning.

[0052] Next, we will explain the processing in the environmental data generation system 1. The environmental data generation system 1 generates environmental data when construction is carried out at a construction site, but before generating the environmental data, a trained model for generating environmental data is generated in the model generation device 200. Therefore, we will explain the generation of the trained model, and then explain the generation of environmental data.

[0053] FIG. 3 is a flowchart showing an example of processing in the model generation device 200. The processing shown in FIG. 3 is started when starting to generate a trained model. To generate a trained model, the model generation device 200 first generates construction environment data that reproduces a construction site in the simulation unit 212 of the point cloud data generation unit 210 based on information input to the input interface 211 (step S101). The construction site includes objects. The objects are represented by a set of points, and each point of the object is tagged with attribute information.

[0054] Next, the simulation unit 212 executes a simulation of irradiating the reproduced construction site with light from the LiDAR 15, and generates point cloud data based on the simulation. Each point in the generated point cloud data corresponds to one of the points in the construction environment data that constitutes the object.

[0055] The simulation unit 212 determines the attribute information tagged to the points of the object corresponding to each point in the point cloud data, and assigns annotations according to the determined attribute information to each point in the point cloud data (step S103). For example, if a point included in the construction environment data is tagged with attribute information of "embankment", the simulation unit 212 assigns the annotation of "embankment" to the corresponding point in the point cloud data. The simulation unit 212 similarly assigns annotations to all points included in the point cloud data, thereby generating point cloud data (step S105).

[0056] Next, the simulation unit 212 generates a large amount of data set including attribute information in the construction environment data and annotation data in the point cloud data (step S107). Next, the simulation unit 212 outputs the generated large amount of data set to the learning unit 220 (step S109).

[0057] The learning unit 220 performs machine learning using the large amount of data set output by the simulation unit 212 to generate a trained model (step S111). The learning unit 220 transmits the generated trained model to the environmental data generation device 100 (step S113). In this way, the model generation device 200 ends the processing shown in FIG.

[0058] Next, the processing in the environmental data generating device 100 will be described. Fig. 4 is a flowchart showing an example of the processing in the environmental data generating device 100. The processing in this flowchart starts when the civil engineering and construction machine 10 starts autonomous operation. In the following description, it is assumed that the civil engineering and construction machine 10 has already arrived at the position (location) of the object where construction work will be performed.

[0059] First, when the autonomous operation of the civil engineering construction machine 10 starts, the environmental data generating device 100 acquires the determined construction plan, for example, via the communication device 17 (step S201). Next, the model acquiring unit 111A acquires the trained model transmitted by the model generating device 200, for example, via the communication device 17 (step S203). The model acquiring unit 111A outputs the acquired trained model to the recognition unit 112 (step S205).

[0060] Next, the environment data generating device 100 causes the LiDAR 15 and the camera 16 to measure the state of the construction environment (step S207). The LiDAR 15 and the camera 16 output point cloud data and image data according to the measured state of the construction environment. Next, the point cloud data acquiring unit 111B acquires the point cloud data output by the LiDAR 15 (step S209). The point cloud data acquiring unit 111B outputs the acquired point cloud data to the recognition unit 112.

[0061] Furthermore, the image data acquisition unit 111C acquires image data output by the camera 16 (step S211). The image data acquisition unit 111C outputs the acquired image data to the recognition unit 112 (step S213). The processes of steps S207 and S209 and the processes of steps S211 and S213 may be performed simultaneously or interchangeably.

[0062] Next, the recognition unit 112 inputs the point cloud data output by the point cloud data acquisition unit 111B as input data to the trained model output by the model acquisition unit 111A. The recognition unit 112 recognizes the object (e.g., embankment) to be constructed this time based on the output data output by the trained model (step S215). The recognition unit 112 outputs object information in which information about the recognized object is linked to the point cloud data to the environment data generation unit 120 (step S217).

[0063] The environmental data generating unit 120 generates environmental data representing the state of the construction environment where construction will be performed this time based on the object information output by the recognizing unit 112 (step S219). Subsequently, the environmental data generating unit 120 outputs the generated environmental data to the control value generating unit 130 (step S221).

[0064] The control value generation unit 130 compares the construction plan with the environmental data, that is, compares the state of the construction environment represented by the construction plan with the current state of the construction environment represented by the environmental data (step S223).The control value generation unit 130 then generates a control value based on the result of comparing the construction plan with the environmental data (step S225).The control value generation unit 130 outputs the generated control value to the operation control device 12, and controls the traveling driving force output device 13 and the construction driving force output device 14, thereby causing the civil engineering and construction machine 10 to operate autonomously.

[0065] The environmental data generation system 1 described above generates environmental data based on the results of inputting point cloud data indicating the state of the construction environment measured by the LiDAR 15 and the camera 16 into the trained model. This allows for more accurate recognition of the construction environment at the construction site.

[0066] Furthermore, a trained model for generating environmental data is generated by the model generation device 200. In the model generation device 200, a large amount of point cloud data is generated using the point cloud data generation unit 210 including the simulation unit 212, so that a large amount of point cloud data used for generating the trained model can be easily generated. Therefore, AI can be easily constructed, and the accuracy of object recognition at the construction site can be improved.

[0067] Furthermore, since construction work and the like is carried out at construction sites, the construction environment at the construction site changes from moment to moment. Under such circumstances, it is necessary to generate environmental data in accordance with changes in the construction environment, but generating environmental data every time the construction environment changes is time-consuming. In this regard, the environmental data generation system 1 of the embodiment generates a trained model for generating environmental data by performing a simulation using the model generation device 200, so it is possible to easily provide a trained model with high accuracy. Therefore, at construction sites where the rate of change is high, it is possible to easily generate environmental data in accordance with changes in the construction environment.

[0068] In the above embodiment, it is assumed that the civil engineering and construction machine 10 is a bulldozer that levels piled up earth (embankment) at a construction site, but the same applies to construction machines other than bulldozers. For example, even if the civil engineering and construction machine 10 is a vibrating roller that autonomously performs construction work of compacting earth and sand with a roller, the environmental data generating device 100 can generate environmental data (for example, a map of the construction site) that is tailored to the target construction work.

[0069] Although the above embodiment is directed to a construction environment for civil engineering work in which construction machinery such as bulldozers is used, the construction environment for civil engineering work may be other environments. For example, the construction environment for civil engineering work may be a tunnel construction site for reducing accidents inside the tunnel. In this case, the simulation unit 212 can, for example, run a simulation in which construction machinery and people are placed inside the tunnel to generate point cloud data, and have the learning unit 220 learn the data. In this case, it becomes possible to recognize objects inside the tunnel, and the system can function as a contact prevention system that prevents contact between construction machinery and people.

[0070] [Hardware configuration] FIG. 5 is a diagram illustrating an example of the hardware configuration of an object recognition device 800 according to an embodiment. As illustrated, the object recognition device 800 includes a communication controller 800-1, a CPU 800-2, a random access memory (RAM) 800-3 used as a working memory, a read-only memory (ROM) 800-4 for storing a boot program and the like, a storage device 800-5 such as a flash memory or a hard disk drive (HDD), and a drive device 800-6, all of which are interconnected via an internal bus or a dedicated communication line. The communication controller 800-1 communicates with components other than the object recognition device 800. The storage device 800-5 stores a program 800-5a to be executed by the CPU 800-2. This program is loaded into the RAM 800-3 by a direct memory access (DMA) controller (not shown) or the like and executed by the CPU 800-2. As a result, part or all of the acquisition unit 111 (model acquisition unit 111A, point cloud data acquisition unit 111B, image data acquisition unit 111C) and recognition unit 112 in the object recognition unit 110 are realized.

[0071] The above-described embodiment can be expressed as follows. a storage device storing a program; a hardware processor; The hardware processor executes the program stored in the storage device, a model obtained by learning using a dataset including point cloud data generated by a point cloud data generating device that generates point cloud data in a construction environment of civil engineering work; Recognizing objects present in the construction environment using the acquired model. The object recognition device is configured as follows.

[0072] The above describes the form for carrying out the present invention using an embodiment, but the present invention is not limited to such an embodiment, and various modifications and substitutions can be made within the scope that does not deviate from the gist of the present invention. [Explanation of symbols]

[0073] 1 Environmental data generation system, 10 Civil engineering and construction machinery, 11 Driving operator, 12 Driving control device, 13 Traveling drive force output device, 14 Construction drive force output device, 15 LiDAR, 16 Camera, 17 Communication device, 18 Positioning device, 100 Environmental data generation device, 110 Object recognition unit, 111 Acquisition unit, 111A Model acquisition unit, 111B Point cloud data acquisition unit, 111C Image data acquisition unit, 112 Recognition unit, 120 Environmental data generation unit, 130 Control value generation unit, 200 Model generation device, 210 Point cloud data generation unit, 211 Input interface, 212 Simulation unit, 220 Learning unit

Claims

1. a model acquisition unit that acquires a model obtained by learning using a dataset including point cloud data generated by a point cloud data generation device that generates point cloud data in a construction environment of civil engineering work; a recognition unit that recognizes an object present in the construction environment using the acquired model, the point cloud data generation device includes a simulator that simulates the construction environment, The model is a model obtained by learning using a dataset including point cloud data generated by performing a simulation of irradiating light on the construction environment using the simulator. Object recognition device.

2. The simulation includes a simulation using a distance measured based on the time from irradiation to reception of reflected light of irradiated light. The object recognition device according to claim 1 .

3. The data set includes, as input data of the model, construction environment data relating to the object tagged with tag data in the simulation of the simulator, and includes, as output data of the model, data of annotations assigned to each point of the point cloud data based on the construction environment data. The object recognition device according to claim 1 or 2.

4. An object recognition device according to any one of claims 1 to 3; an environment data generation unit that generates environment data representing a state of the construction environment based on the object recognized by the object recognition device, Environmental data generator.

5. a learning unit that generates a model by learning using data sets of construction environment data of a construction environment of civil engineering work and point cloud data generated by a point cloud data generating device based on the construction environment data; The data set includes, as input data of the model, construction environment data relating to objects tagged with tag data in a simulation in which light is irradiated onto the construction environment of a simulator that virtualizes the construction environment, and includes, as output data of the model, data of annotations assigned to each point of the point cloud data based on the construction environment data. Model generation device.

6. The computer of the object recognition device a model obtained by learning using a dataset including point cloud data generated by a point cloud data generating device that generates point cloud data in a construction environment of civil engineering work; Using the acquired model, recognize objects present in the construction environment; the point cloud data generation device includes a simulator that simulates the construction environment, The model is a model obtained by learning using a dataset including point cloud data generated by performing a simulation of irradiating light on the construction environment using the simulator. Object recognition method.

7. The object recognition device's computer acquiring a model obtained by learning using a dataset including point cloud data generated by a point cloud data generating device that generates point cloud data in a construction environment of civil engineering work; Using the acquired model, objects present in the construction environment are recognized; the point cloud data generation device includes a simulator that simulates the construction environment, The model is a model obtained by learning using a dataset including point cloud data generated by performing a simulation of light irradiation using the simulator. program.

Images