Object Recognition System
The object recognition system for work machines adapts image recognition models in real-time to maintain accuracy by using a camera, control device, and learning server to handle dynamic environments with frequent topography changes.
Patent Information
- Application Number
- JP2021053330
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- Filing Date
- 2021-03-26
- Publication Date
- 2024-11-05
- Estimated Expiration
- 2041-03-26
AI Technical Summary
Existing object recognition systems for work machines struggle with maintaining high accuracy in dynamic environments where topography changes frequently, such as construction sites, as they do not account for rapid alterations in the surrounding landscape.
An object recognition system that includes a camera, a control device, and a learning server, which generates and updates image recognition models based on real-time changes in the work site environment by capturing and processing images, and transmitting them to a learning server for model adaptation.
The system ensures high-accuracy object recognition by continuously updating image recognition models to match the current environmental conditions, thereby maintaining effective object detection despite topographical changes.
Smart Images

Figure 0007577589000001 
Figure 0007577589000002 
Figure 0007577589000003
Abstract
Description
[Technical field]
[0001] The present invention relates to an object recognition system that recognizes objects present around a work machine. [Background technology]
[0002] In recent years, with the improvement of computer performance, image recognition technology based on machine learning using deep neural networks has made remarkable progress. Therefore, solutions that recognize the surrounding environment, such as recognizing objects from acquired images using machine learning, are being introduced at civil engineering and construction work sites as well, using on-site equipment and on-board equipment for work machines. Furthermore, even after the shipment of on-site equipment and on-board equipment for work machines, there are high expectations for improving the image recognition accuracy. In particular, at work sites, the environment (topography and objects around the site) is significantly different from one another, so it is thought that image recognition accuracy will deteriorate when the work site changes, and there are high expectations for image recognition technology that can be used even when the work site changes.
[0003] Patent Document 1 proposes a learning system consisting of an on-board device and a learning server. The learning system is configured so that the vehicle's driving position and the captured image during driving can be transmitted from the on-board device to the learning server, learning can be performed in the learning server, and the learning results can be reflected in the on-board device, making it possible to improve recognition accuracy even after the product is shipped. Specifically, the on-board device performs image recognition processing using parameters stored in the on-board device, and transmits the used image data as learning data to the learning server. The learning server performs learning processing for image recognition processing based on the learning data from each on-board device, updates the parameters, and provides the updated parameters to each on-board device. In addition, each on-board device sends data on the driving area of the vehicle together with image data to the server, allowing the learning server to update the parameters for each driving area, and to update the parameters to suit the driving area. [Prior art documents] [Patent documents]
[0004] [Patent Document 1] JP 2015-135552 A Summary of the Invention [Problem to be solved by the invention]
[0005] Patent Document 1 is a technology that assumes an on-board device for an automobile, and updates parameters used in image recognition processing based on the driving area. On the other hand, in places where the terrain changes from moment to moment, such as work sites, images (mainly the background) acquired at the site may change significantly before and after the terrain changes. Therefore, even in the same location, the optimal values of parameters used in image recognition processing may differ before and after the terrain changes, and image recognition accuracy may deteriorate. In this regard, the technology in Patent Document 1 assumes automobiles traveling on roads, and does not take into account the issue of surrounding terrain changes that are less relevant to use in automobiles.
[0006] The present invention has been made in consideration of the above circumstances, and has an object recognition system that can suppress a decrease in image recognition accuracy even if the environment of the work site of a work machine changes. [Means for solving the problem]
[0007] The present application includes multiple means for solving the above-mentioned problems. One example of such an object recognition system includes a learning server that generates a first image recognition model based on an image of a work site stored in a storage device, a camera that captures image of the work site, and a control device that is communicatively connected to the camera and the learning server, stores the image of the camera, and performs processing to recognize objects at the work site based on the first image recognition model generated by the learning server and the image of the camera. The object recognition system further includes a transmission command output device communicatively connected to the control device, and the transmission command output device outputs a transmission command to the control device based on changes in the topography of the work site, instructing the control device to transmit images captured by the camera for a predetermined period of time to the learning server. The learning server receives the transmission command from the control device and generates a second image recognition model based on the image for the predetermined period of time and transmits the generated second image recognition model to the control device. When the control device receives the second image recognition model from the learning server, the control device performs processing to recognize objects at the work site based on the image of the camera and the second image recognition model. Effect of the Invention
[0008] According to the present invention, even if the environment of the work site changes, an image recognition model suitable for that environment can be used, so that objects can be recognized with high accuracy. [Brief description of the drawings]
[0009] [Figure 1] 1 is a schematic diagram of a work site 1 in which an object recognition system according to a first embodiment is installed. [Diagram 2] FIG. 1 is a configuration diagram showing elements of an object recognition system according to a first embodiment. [Diagram 3] 1 is a schematic diagram of a work machine 2 in a first embodiment. [Figure 4] FIG. 2 is a system configuration diagram of an on-board device 7 of the work machine 2. [Diagram 5] 6 is a flowchart of a process performed by an image recognition processing unit 32a (first control device 21). [Figure 6]FIG. 4 is a diagram showing an example of a process of image recognition processing. [Figure 7] 11 is an example of a flowchart of processing performed by a video storage processing unit 32b (first control device 21). [Figure 8] FIG. 1 is a schematic configuration diagram of a learning server 9. [Figure 9] FIG. 2 is a schematic diagram of a transmission command output device 8. [Figure 10] 2 is an external view of a transmission command output device 8 according to the first embodiment. FIG. [Figure 11] A diagram showing example images taken by the camera at three locations (flat ground, a location filled with soil and sand, and a location with a structure). [Figure 12] 1 is a schematic diagram of a site monitoring device 22 installed at a work site 1. FIG. [Figure 13] 1 is a schematic diagram of a work site 1 in which an object recognition system according to a second embodiment is installed. [Figure 14] FIG. 11 is a configuration diagram showing elements of an object recognition system according to a second embodiment. [Figure 15] FIG. 2 is a schematic diagram of a site monitoring device 22. [Figure 16] FIG. 11 is a configuration diagram showing elements of an object recognition system according to a third embodiment. [Figure 17] FIG. 11 is a schematic configuration diagram of a site topography management system 24 according to a third embodiment. [Figure 18] FIG. 13 is a diagram showing an example of a display screen 101 of the display device 82 generated by a screen generating unit 92d (third control device 83). [Figure 19] FIG. 11 is a schematic configuration diagram of a transmission command output device 8A according to a third embodiment. [Figure 20] FIG. 11 is a configuration diagram showing elements of an object recognition system according to a fourth embodiment. [Figure 21] FIG. 11 is a system configuration diagram of an on-vehicle device 7A mounted on a work machine 2 of a fourth embodiment. [Figure 22] FIG. 11 is a side view of a working machine 2 of a fourth embodiment. [Figure 23] FIG. 13 is a diagram showing an example of a display screen of a display device 82 according to a fourth embodiment. [Figure 24] FIG. 13 is a configuration diagram showing elements of an object recognition system according to a fifth embodiment. [Diagram 25]FIG. 13 is a side view of a working machine 2 of a fifth embodiment. [Figure 26] FIG. 13 is a schematic configuration diagram of a transmission command output device 8B according to a fifth embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0010] Hereinafter, an embodiment of the present invention will be described with reference to the drawings.
[0011] (Embodiment 1) A work site object recognition system according to a first embodiment (embodiment 1) of the present invention will be described with reference to FIGS. 1 to 11. FIG.
[0012] 1 is a schematic diagram of a work site 1 in which the object recognition system of this embodiment is introduced. In the work site 1, a work machine 2, a transport machine 3, a worker 4, a site manager 5, and the like are present.
[0013] The work machine 2 is, for example, a hydraulic excavator, and performs tasks such as leveling work, excavation work, and loading work objects 6 onto the transport machine 3 at a predetermined location. The transport machine 3 is, for example, a dump truck, and transports work objects (e.g., soil) 6, etc., while entering and leaving the work site 1.
[0014] The worker 4 performs detailed work such as guiding various machines around the work machine 2, cleaning up work objects 6 that have spilled from the work machine 2, and transporting various small items. The site manager 5 is a worker who is responsible for managing the work site 1, and performs work progress management at the work site 1 or in an office set up at the work site 1.
[0015] The work machine 2 is equipped with a camera (image acquisition device) 19 capable of capturing images of the surroundings of the work machine 2, and recognizes people such as the worker 4 and the site manager 5 by using image recognition based on the images captured by the camera 19. In other words, the work machine 2 uses the images of the surroundings of the work machine 2 to recognize whether or not there is a person in the vicinity of the work machine 2.
[0016] <Object recognition system for the workplace> 2 is a configuration diagram showing elements of the object recognition system according to this embodiment. The object recognition system includes a camera 19 and a first control device 21 mounted on a work machine 2, a learning server 9, and a transmission command output device 8. The first control device 21, the learning server 9, and the transmission command output device 8 are connected to the same network, for example, via a wireless communication device, and can communicate with each other. Note that as long as the devices 21, 9, and 8 are connected to the same network, the communication format is not limited to wireless communication, and may be wired communication using, for example, copper wires or optical fibers.
[0017] <Work machinery> Fig. 3 is a schematic diagram of the work machine 2 in this embodiment. The work machine 2 is equipped with, as the on-board device 7 (see Fig. 4), a camera (image acquisition device) 19 mounted on the upper rotating body 12, a communication device 20 for communicating with the learning server 9 and the transmission command output device 8 via a network, and a first control device (controller) 21 that stores images from the camera 19 and performs processing to recognize objects in the work site 1 based on an image recognition model generated by the learning server 9 and the stored image from the camera 19.
[0018] The work machine 2 is equipped with a lower running body 11, an upper rotating body 12 provided on the lower running body 11 so as to be able to rotate left and right, a boom 15 provided rotatably on the upper rotating body 12, an arm 16 provided rotatably at the tip of the boom 15, a bucket 17 provided rotatably at the tip of the arm 16, and a driver's seat 18 where an operator gets in and operates the work machine 2. Of these, the boom 15, the arm 16, and the bucket 17 constitute a multi-joint type front working device 13.
[0019] 4 is a system configuration diagram of the on-board device 7 of the work machine 2. The on-board device 7 includes a camera 19, a communication device 20, and a first control device 21. In the figure, inside the processing device 32 of the first control device 21, the processing executed by the first control device 21 is shown divided into multiple blocks by function.
[0020] <Camera (image acquisition device) 19> For example, a digital camera equipped with a CMOS (Complementary Metal Oxide Semiconductor) image sensor is used as the camera 19. In order to capture the entire surroundings of the work machine 2 with a small number of cameras 19, it is preferable to select a wide-angle lens for the camera 19.
[0021] The camera 19 is mounted on the upper rotating body 12, and is installed so as to capture the rear of the upper rotating body 12, for example, as shown in Fig. 3. The orientation of the camera 19 is not limited to the rear of the upper rotating body 12. The number of cameras 19 mounted on the upper rotating body 12 may be one or more. Therefore, for example, the cameras 19 may be mounted on the left and right sides or in front of the upper rotating body 12, and the orientation of each camera 19 may be set so that there are no blind spots around the work machine 2.
[0022] <Communication device 20> The communication device 20 is not limited to a specific type, but may be, for example, a wireless communication device equipped with an antenna compatible with LTE-Advanced or a WiFi antenna.
[0023] <First control device 21> The first control device 21 is, for example, a controller, and includes a processing device 32 (for example, a combination of a CPU (Central Processing Unit) and a GPU (Graphics Processing Unit)), a storage device 33 (for example, a semiconductor memory (ROM, RAM), a magnetic storage device (HDD) or a solid-state drive (SSD)) in which programs executed by the processing device 32 and various data are stored, and an input / output interface 31 for communicatively connecting input / output devices such as a camera 19 and a communication device 20 to the control device 21.
[0024] The first control device 21 functions as an image recognition processing unit 32a, an image storage processing unit 32b, and a recognition model update unit 32c by the processing unit 32 executing a program.
[0025] <Image recognition processing unit 32a> The image recognition processing unit 32a performs processing to recognize objects based on the video captured by the camera 19 and stored in the storage device 33 and the image recognition model generated by the learning server 9. The type of object to be recognized is not limited, but may be, for example, a worker 4 including a site manager 5.
[0026] FIG. 5 is a flowchart of the processing performed by the image recognition processing unit 32a (first control device 21).
[0027] When the flow starts, in step S101, the image recognition processing unit 32a checks whether or not there is video input from the camera 19. If there is video input, the process proceeds to step S102, and if there is no video input, the process of step S101 is executed again.
[0028] In step S102, the image recognition processing unit 32a acquires one image from the video captured by the camera 19. For example, the latest frame of the video captured by the camera 19 can be acquired. When image acquisition is complete, the process proceeds to step S103.
[0029] In step S103, the image recognition processing unit 32a performs a recognition process to infer whether or not a recognition target exists in the acquired image, using the image acquired in step S102 and the image recognition model stored in the storage device 33. An example of the recognition process is shown in FIG. 6. For example, when an image 41 showing a worker is acquired in step S102, the recognition process is performed in step S103 to determine whether or not the recognition target (worker) exists in the image 41 and the position of the recognition target (worker) in the image 41. When the image recognition processing unit 32a detects the recognition target (worker) in the image 41 in step S103, it generates a rectangular bounding box 44 so as to surround the recognition target (worker). The image recognition processing unit 32a can calculate the position of the recognition target (worker) detected by the coordinate values (x0, y0) of the upper left point Plt and the coordinate values (x1, y1) of the lower right point Prb of the bounding box 44. The coordinate values of these two points Plt and Prb are determined with the upper left point of the image 41 set as the origin (0,0). The image recognition processing unit 32a can also generate an image (image with bounding box) 43 in which a bounding box 44 is superimposed on the image 41 showing the worker.
[0030] <Video storage processing unit 32b> When the video storage processing unit 32b receives from the transmission command output device 8 a transmission command instructing the first control device 21 to transmit the video of the camera 19 to the learning server 9, the video storage processing unit 32b performs a process of generating learning video (video of the camera 19 for a predetermined period of time) from the video of the camera 19 to be used when the learning server 9 generates a new image recognition model, and a process of transmitting the generated learning video to the learning server 9. An example of a flowchart of the process performed by the video storage processing unit 32b (first control device 21) is shown in FIG.
[0031] When the flow starts, in step S202, the video storage processing unit 32b judges whether or not a transmission command and video acquisition time data have been received from the transmission command output device 8. If the video storage processing unit 32b receives a transmission command in step S202, it judges that there has been a change in the terrain, and if it does not receive a transmission command, it judges that there has been no change in the terrain. The video acquisition time data is data that specifies the time (predetermined period) of the learning video to be transmitted to the learning server 9. However, if the time of the learning video is determined in advance on the first control device 21 side, there is no need to transmit the video acquisition time data from the transmission command output device 8 to the first control device 21. That is, in this case, in step S202, it is judged whether or not a transmission command has been received from the transmission command output device 8.
[0032] If the image storage processing unit 32b receives a transmission command in step S202 (if there has been a change in the terrain), the process proceeds to step S203; if the image storage processing unit 32b does not receive a transmission command (if there has been no change in the terrain), the process returns to step S202.
[0033] In step S203, the video storage processing unit 32b sets the video acquisition time based on the received video acquisition time data. For example, if the video acquisition time data specifies 10 minutes, the video storage processing unit 32b sets the video acquisition time to that time (10 minutes).
[0034] In step S204, the video storage processing unit 32b generates learning video for the time period set in step S203 from the video of the camera 19. The video storage processing unit 32b sets the time when the processing of step S204 starts as time 0, acquires video for the time period set in step S203 from that time, and sets the acquired video as learning video. After that, the video storage processing unit 32b stores the generated learning video in the storage device 33 in the first control device 21.
[0035] The start time of the learning video does not need to be set to the time when the processing of step S204 begins, and may be set before or after the start time of step S204. For example, the video data of camera 19 may be stored in storage device 33 or the like going back a predetermined time, and the learning video may start from a time before the start time of step S204 based on that video data. Also, the learning video may start a predetermined time after step S204.
[0036] In step S205, the video storage processing unit 32b transmits the learning video stored in the storage device 33 in step S204 to the learning server 9 via the communication device 20. After transmission, a deletion process is performed on the learning video stored in the storage device 33, and the process returns to step S202.
[0037] <Recognition model update unit 32c> When the image recognition model of the learning server 9 is updated, the recognition model update unit 32c receives the new image recognition model from the learning server 9 via the communication device 20, and performs an update process of storing the new image recognition model in the storage device 33. After this, the image recognition processing unit 32a performs an object recognition process at the work site 1 based on the new image recognition model and the image from the camera 19.
[0038] <Storage device 33> The storage device 33 can be a memory such as a random access memory (RAM) and a storage such as a hard disk drive (HDD) or a solid state drive (SSD). The storage device 33 is connected to the processing device 32 so that the processing device 32 can write and read data.
[0039] <Learning Server 9> Next, the learning server 9 will be described with reference to Fig. 8. Fig. 8 is a schematic configuration diagram of the learning server 9. In the figure, inside the processing device 52 of the learning server 9, the processes executed by the learning server 9 are shown divided into multiple blocks by function.
[0040] The learning server 9 is, for example, a server computer, and includes a processing device 52 (e.g., a combination of a CPU (Central Processing Unit) and a GPU (Graphics Processing Unit)), a storage device 53 (e.g., semiconductor memory (ROM, RAM), a magnetic storage device (HDD) or a solid-state drive (SSD)) in which programs executed by the processing device 52 and various data are stored, and an input / output interface 51 for communicatively connecting an input / output device such as a communication device 41 to the learning server 9. The location where the learning server 9 is installed is not limited. It may be installed either outside the work site 1 or inside the work site 1.
[0041] The learning server 9 functions as a learning video receiving unit 52a and a recognition model generating unit 52b by executing a program with the processing device 52.
[0042] <Communication Device 41> The communication device 41 is not limited to a specific type, but may be, for example, a wireless communication device equipped with an antenna compatible with LTE-Advanced or a WiFi antenna.
[0043] <Learning image receiving unit 52a> When learning video is transmitted from the first control device 21, the learning video receiving unit 52a performs the process of receiving the learning video via the communication device 41 and the process of storing the received learning video in the storage device 53 within the learning server 9.
[0044] <Recognition model generation unit 52b> The recognition model generating unit 52b performs a process of generating a new image recognition model by machine learning using the learning video stored in the storage device 53, and storing the generated image recognition model in the storage device 53. The type of method of generating an image recognition model using a learning video is not limited, but for example, an operator of the learning server 9 manually generates teacher data by attaching a bounding box to a recognition target (worker) in the learning video, and generates a new image recognition model based on the teacher data. The timing of learning the teacher data and generating a new image recognition model may be every time the learning video receiving unit 52a receives new learning video. However, the timing is not limited, and the update status of new learning videos may be checked at regular intervals, and learning may be performed when new learning videos are present.
[0045] <Storage device 53> The storage device 53 can be a memory such as a random access memory (RAM) and a storage such as a hard disk drive (HDD) or a solid state drive (SSD). The storage device 53 is connected to the processing device 52 so that the processing device 52 can write and read data.
[0046] <Transmission command output device 8> 9 is a schematic diagram of the transmission command output device 8. The transmission command output device 8 includes an input device 61, a communication device 62, and a second control device 63. In the figure, inside the processing device 52 of the second control device 63, the processing executed by the second control device 63 is shown divided into multiple blocks by function.
[0047] 10 is an external view of the transmission command output device 8 in this embodiment, and as shown in this figure, the transmission command output device 8 in this embodiment is a remote control, and its input device 61 is composed of a switch (topography change determination button). There is no restriction on the person who holds the transmission command output device 8, but as shown in FIG. 1, it may be held by the site manager 5, the operator of the work machine 2, or the worker 4 at the work site 1. When the holder of the transmission command output device 8 recognizes that there has been a change in the topography of the work site 1, he or she operates (presses) the input device 61. As a result, a transmission command (to which video capture time data may also be added) is output from the transmission command output device 8 to the first control device 21.
[0048] At the work site 1, for example, as work progresses, such as piling up earth and building structures on a place that was previously flat, the terrain changes significantly. The background of the image captured by the camera 19 also changes significantly due to the change in terrain. Example images captured on flat ground, at a place where earth has been piled up, and at a place where a structure has been erected are shown in (a), (b), and (c) of FIG. 11, respectively. The backgrounds in each image are significantly different. Therefore, when recognizing workers from each image, it is considered that the optimal image recognition model will be different for each image.
[0049] <Input Device 61> The input device 61 is not limited to a particular type as long as it is a device capable of outputting a transmission command in response to the owner's operation, but for example, as shown in Fig. 10, it is a push-button switch that allows the owner to press the transmission command output device 8 at any time. When the owner releases the switch 61 after pressing it, it quickly returns to the reference position before it was pressed.
[0050] <Communication Device 62> The communication device 62 is not limited to a specific type, but may be, for example, a wireless communication device equipped with an antenna compatible with LTE-Advanced or a WiFi antenna.
[0051] <Second control device 63> The second control device 63 is, for example, a microcontroller, and is equipped with a processing device 72 (for example, an MPU (Micro Processing Unit)), a storage device 73 (for example, a semiconductor memory (ROM, RAM), a magnetic storage device (HDD) or a solid state drive (SSD)) in which programs executed by the processing device 72 and various data are stored, and an input / output interface 71 for communicatively connecting input / output devices such as an input device 61 and a communication device 62 to the second control device 63.
[0052] The second control device 71 functions as an image acquisition time setting section 72a, a topography change determination section 72b, and a transmission command transmission section 72c by the processing device 72 executing a program.
[0053] <Video Acquisition Time Setting Unit 72a> The video acquisition time setting unit 72a is a part that sets the time of the learning video to be generated when the first control device 21 receives a transmission command, and stores the set time as video acquisition time data in the storage device 73. The time of the learning video (set time) can be set to a fixed value such as "10 minutes", for example.
[0054] <Topography change determination unit 72b> The terrain change determination unit 72b determines whether or not the terrain around the work machine 2 at the work site 1 has changed. In this embodiment, it determines that the terrain has changed when the push button of the input device 61 is pressed. Also, it determines that there has been no terrain change when the push button of the input device 61 is not pressed. The terrain change determination unit 72b stores the determination result (presence or absence of terrain change) in the storage device 73 as terrain change data.
[0055] The transmission command transmission unit 72c monitors whether or not there is a change in the terrain based on the terrain change data stored in the memory device 73, and if a change in the terrain is detected, it transmits a transmission command and image acquisition time data to the first control device 21 of the work machine 2.
[0056] <Storage device 73> The storage device 73 can be a memory such as a random access memory (RAM) and a storage such as a hard disk drive (HDD) or a solid state drive (SSD). The storage device 73 is connected to the processing device 72 so that the processing device 72 can write and read data.
[0057] <Operation> In the object recognition system configured as described above, when the work by the work machine 2 progresses and the holder of the transmission command output device 8 (for example, the worker 4) recognizes that there has been a change in the terrain around the work machine 2, the holder operates the input device 61 (push button) of the transmission command output device 8. This operation causes the transmission command and image capture time data to be transmitted from the transmission command output device 8 to the first control device 21 of the work machine 2.
[0058] The first control device 21 of the work machine 2, which receives the transmission command and the image acquisition time data, generates learning image from the image of the camera 19 for the set time (e.g., 10 minutes) specified in the image acquisition time data, and transmits the learning image to the learning server 9.
[0059] When the learning server 9 receives a learning video, for example, an operator of the learning server 9 generates training data by adding a bounding box to the learning video. The learning server 9 generates a new image recognition model (second image recognition model) based on the training data, and transmits the new image recognition model to the first control device 21 via the communication device 41.
[0060] The first control device 21, which has received the new image recognition model, stores the new image recognition model in the storage device 33 and recognizes the object (worker 4) based on the new image recognition model and the image captured by the camera 19. In other words, it stops using the image recognition model (first image recognition model) that was used up until then, and starts using the new image recognition model (second image recognition model). As a result, after a change in the terrain around the work machine 2, a new image recognition model is generated based on learning images of the terrain after the change, and object recognition can be performed based on the new image recognition model, thereby suppressing a decrease in object recognition accuracy due to a change in terrain.
[0061] <Effects> By using the object recognition system of the first embodiment, even if the topography of the work site 1 changes from moment to moment, a new image recognition model (second image recognition model) is generated in the learning server 9 based on the image of the camera 19 mounted on the work machine 2, based on the judgment of a person at the work site 1 (e.g., the worker 4). Then, the first control device 21 can perform object recognition based on the new image recognition model, so that a decrease in the accuracy of object recognition can be suppressed even after the topography changes. In other words, according to this embodiment, even if the environment of the work site 1 changes, an image recognition model suitable for that environment can be used, so that objects can be recognized with high accuracy.
[0062] <Other> In the above description, the transmission command output device 8 is a remote control (see FIG. 10), but other devices such as a smartphone or tablet terminal carried by the worker 4 can also be used. In addition, the cab 18 may be equipped with a switch such as that shown in FIG. 10, and the transmission command can be output by operating the switch. When the transmission command output device 8 is provided in the cab 18, the operator is usually the operator of the work machine 2. The operator of the work machine 2 actually uses the object recognition by the first control device 21 through a monitor or the like installed in the cab 18, and can actually perceive that the object recognition accuracy has decreased due to the progress of the work. Therefore, the operator can operate the transmission command output device 8 at a more appropriate timing than the worker 4.
[0063] (Embodiment 2) Next, a second embodiment (embodiment 2) of the present invention will be described with reference to Fig. 12 to Fig. 15. This embodiment is characterized in that, as shown in Fig. 12, a site monitoring device 22 is installed at a predetermined location in a work site 1, and a camera 19 and a first control device 21 are mounted thereon.
[0064] 12 is a schematic diagram of the on-site monitoring device 22 installed at the work site 1. Although the installation location of the on-site monitoring device 22 is not limited, for example, the on-site monitoring device 22 is installed at a high place such as on top of a long pole. Note that the on-site monitoring device 22 may be installed on a pole via an electric rotating table (platform head) so that the camera 19 can appropriately perform at least one of panning, tilting, and rolling.
[0065] The first control device 21 of the on-site monitoring device 22, like the first control device 21 of embodiment 1, performs processes of recognizing the recognition target object from the image acquired by the camera 19, generating the image for learning, and updating the recognition model.
[0066] Fig. 13 is a schematic diagram of a work site 1 in which the object recognition system of this embodiment is introduced. In the work site 1 of this figure, a site monitoring device 22 is added, compared to the work site shown in Fig. 1. Note that a work machine 2 and a transport machine 3 are not shown, but they may be present.
[0067] 14 is a configuration diagram showing elements of an object recognition system according to this embodiment. In the object recognition system of this embodiment, a camera 19 and a first control device 21 are mounted on a site monitoring device 22 rather than on a work machine 2.
[0068] <On-site monitoring device> 15 shows a schematic configuration diagram of the site monitoring device 22. The site monitoring device 22 has a similar configuration to the on-vehicle device 7 (see FIG. 4) of the first embodiment, and has equivalent functions.
[0069] <Effects> If the object recognition system of the second embodiment is used, when the topography of the work site 1 changes, a new image recognition model (second image recognition model) is generated in the learning server 9 based on the image of the camera 19 mounted on the site monitoring device 22 at the discretion of a person at the work site 1 (e.g., the worker 4), and the first control device 21 can perform object recognition based on the new image recognition model. In other words, according to this embodiment, even if there is no work machine 2 equipped with a camera, an image recognition model suitable for the change in the environment of the work site 1 can be used, so that objects can be recognized with high accuracy.
[0070] (Embodiment 3) Next, a third embodiment (third embodiment) of the present invention will be described with reference to drawings 16 to 19.
[0071] Fig. 16 is a configuration diagram showing elements of the object recognition system according to this embodiment. The object recognition system according to this embodiment includes a camera 19 and a first control device 21 mounted on a work machine 2, a learning server 9, a transmission command output device 8A, and a site terrain management system 24. The system according to this embodiment differs from the system according to the first embodiment in that it includes the transmission command output device 8A and the site terrain management system 24. In particular, the transmission command output device 8A is characterized in that it automatically determines whether or not there has been a change in the terrain around the work machine 2 based on the terrain information (terrain data) of the work site 1 input from the site terrain management system 24, and outputs a transmission command to the first control device 21 when it is determined that there has been a change in the terrain around the work machine 2.
[0072] <On-site terrain management system 24> 17 is a schematic diagram of the on-site topography management system 24 of this embodiment. In the figure, inside the processing device 92 of the third control device 83, the processing executed by the third control device 83 is shown divided into multiple blocks for each function.
[0073] The site terrain management system 24 is equipped with a third control device 83, an input device 81 for inputting operator operations and information external to the third control device 83 as data into the third control device 83, a display device 82 such as a monitor, and a communication device 84 for communicating with an external terminal.
[0074] The input device 81 includes at least a data input device (for example, a connection terminal for a flash memory (USB memory) in which the topographical data of the work site 1 is stored) for inputting the topographical data of the work site 1 from outside the third control device 83. In addition, a mouse, a keyboard, a touch panel that also serves as the display device 82, and the like may be included.
[0075] The topographical data of the work site 1 is not limited to being input to the third control device 83 via the input device 81, but may also be input from another terminal or the Internet via the communication device 84. The topographical data of the work site 1 may include initial topographical data obtained by surveying before construction of the work site 1, topographical data during construction obtained as appropriate during construction of the work site 1, and design drawing data showing the completed form of the topographical data. The latest topographical data of the work site 1 is preferably input to the third control device 83 periodically.
[0076] The third control device 83 is, for example, a computer, and includes a processing device 92 (for example, a combination of a CPU (Central Processing Unit) and a GPU (Graphics Processing Unit)), a storage device 93 (for example, a semiconductor memory (ROM, RAM), a magnetic storage device (HDD) or a solid state drive (SSD)) in which programs executed by the processing device 92 and various data are stored, and an input / output interface 91 for communicatively connecting input / output devices such as an input device 81, a display device 82 and a communication device 84 to the third control device 83. The installation location of the third control device 83 is not limited. It may be installed outside the work site 1 or inside the work site 1.
[0077] The third control device 83 functions as a work site information collection section 92a, a grid division section 92b, a topography change amount calculation section 92c, and a screen generation section 92d by having the processing device 92 execute a program.
[0078] The work site information collection unit 92 a is a part that performs processing to collect topographical data of a part or all of the work site 1 that is input from the input device 81 and the communication device 84 via the input / output interface 91 .
[0079] The grid division unit 92b is a part that performs processing to divide the work site 1 into grids at fixed distance intervals. The type of grid into which the work site 1 has been divided by the grid division unit 92b is stored in the storage device 93 as grid data.
[0080] The terrain change amount calculation unit 92c is a part that performs processing to calculate the amount of change in terrain height (terrain height change amount) of each grid based on a certain point in time, using the terrain data collected by the work site information collection unit 92a and the grid data generated by the grid division unit 92b. As the certain point in time, the state of the terrain of the work site 1 before construction begins can be used as the basis. As the next certain point in time, the time when the numerical value indicated by the terrain change index value data (described later) first reaches a predetermined value can be used as the basis. Thereafter, each time the numerical value indicated by the terrain change index value data (described later) reaches a predetermined value, that timing can be used as the basis.
[0081] Furthermore, the terrain change amount calculation unit 92c of this embodiment can calculate the ratio of the number of grids whose terrain height has changed by 1 m or more based on a certain point in time to the total number of grids in the work site 1, and the calculated value can be transmitted to the transmission command output device 8A as terrain change index value data. Note that, here, grids whose terrain height has changed by 1 m or more are counted as having changed, but it goes without saying that any other value can be selected instead of 1 m.
[0082] The screen generating section 92d is a section that performs processing to generate a display screen for the display device 82 based on the data of the amount of change in terrain height for each grid calculated by the terrain change amount calculating section 92c.
[0083] FIG. 18 is a diagram showing an example of the display screen 101 of the display device 82 generated by the screen generation unit 92d (third control device 83). The display screen 101 of FIG. 18 displays a terrain change map (plan view) of a certain work site A. As shown in this figure, the site terrain management system 24 is capable of managing the terrain data 102 of the work site 1 in units of grids 103 at fixed distance intervals, and the terrain change amount calculation unit 92c calculates and manages the terrain height included in each grid, thereby calculating the presence or absence of a terrain change in the work site 1. For example, the grid 104 marked with diagonal lines on the display screen 101 of FIG. 18 is a grid calculated by the terrain change amount calculation unit 92c to have changed by 1 m or more from a certain point in time. The screen generation unit 92d displays the grid with diagonal lines, so that the operator of the site terrain management system 24 (third control device 83) can easily identify the grid in which the terrain height has changed by 1 m or more on the screen of the display device 82.
[0084] <Transmission command output device 8A> FIG. 19 is a schematic diagram of the transmission command output device 8A of this embodiment. The transmission command output device 8A of this embodiment is mainly characterized in that it receives terrain change index value data indicating the ratio of grids in which the terrain height has changed by 1 m or more in the work site 1 from the site terrain management system 24, and determines that a terrain change has occurred and automatically outputs a transmission command when the ratio indicated by the terrain change index value data is a predetermined value (for example, 10%) or more. Therefore, the transmission command output device 8A of this embodiment is equipped with a second control device 63A and a communication device 62, and the input device 61 (push button) is omitted from that of the embodiment 1 shown in FIG. 9. The installation location of the transmission command output device 8A is not limited. As long as it can communicate with the work machine 2 in the work site 1, it may be installed outside the work site 1 or inside the work site 1. Note that the same parts as those in FIG. 9 are given the same reference numerals and explanations are omitted.
[0085] The second control device 63A is, for example, a microcontroller, and is equipped with a processing device 74 (e.g., an MPU (Micro Processing Unit)), a storage device 73 (e.g., a semiconductor memory (ROM, RAM), a magnetic storage device (HDD) or a solid state drive (SSD)) in which programs executed by the processing device 74 and various data are stored, and an input / output interface 71 for communicatively connecting input / output devices such as a communication device 62 to the second control device 63.
[0086] The processing device 74 of the transmission command output device 8A of the third embodiment functions as an image acquisition time setting section 74a, a topographical change determination section 74b, and a transmission command transmitting section 74c.
[0087] The terrain change determination unit 74b determines whether or not the terrain around the work machine 2 at the work site 1 has changed. In this embodiment, the terrain change determination unit 74b receives terrain change index value data from the site terrain management system 24 via the communication device 62, and determines that the terrain has changed if the ratio indicated by the data is equal to or greater than a predetermined value (10%). In other cases, it determines that there has been no terrain change. The terrain change determination unit 74b stores the determination result (presence or absence of terrain change) in the storage device 73 as terrain change data.
[0088] The video acquisition time setting unit 74a monitors whether or not there is a change in the terrain based on the terrain change data stored in the storage device 73, and if there is a change in the terrain, sets the time of the learning video to be generated when the first control device 21 receives a transmission command, and stores the set time as video acquisition time data in the storage device 73. The set time specified by the video acquisition time data may be a constant value, or may be set in proportion to the ratio indicated by the terrain change index value data. In the latter case, for example, if the received ratio is p [%] and the set time specified by the video acquisition time data is t, the video acquisition time setting unit 74a calculates the video acquisition time by the following formula (1).
[0089] t[minutes]=10+p··· Formula (1) In this example, when the ratio indicated by the terrain change index value data is 20%, the set time t is 30 minutes. Note that in this embodiment, the larger the change in the terrain, the longer the learning video time t is set, but there is no clear correlation between the change in the terrain and the time t, and even if there is a large change in the terrain, it may be possible to generate a new image recognition model with a relatively short learning video.
[0090] The transmission command transmission unit 74c monitors whether or not there is a change in the terrain based on the terrain change data stored in the memory device 73, and if a change in the terrain is detected, it transmits a transmission command and image acquisition time data to the first control device 21 of the work machine 2.
[0091] <Operation> In the object recognition system configured as described above, terrain data of the work site 1 is input each time to the third control device 83 of the site terrain management system 24 as the work (construction) by the work machine 2 progresses, and as a result, the proportion indicated by the terrain change index value data transmitted from the site terrain management system 24 to the transmission command output device 8A (the proportion of grids with a terrain height change of 1 m or more) gradually increases.
[0092] Thereafter, when the value indicated by the terrain change index value data reaches 10% or more based on the terrain condition of the work site 1 before construction by the work machine 2 begins, the transmission command output device 8A determines that a terrain change has occurred at the work site 1, and transmits a transmission command and image acquisition time data to the first control device 21 of the work machine 2.
[0093] Thereafter, similarly to the first embodiment, a new image recognition model (second image recognition model) is generated by the learning server 9 based on the learning video transmitted from the first control device 21, and the first control device 21 performs object recognition based on the new image recognition model. As a result, when the topography of the work site 1 changes, object recognition can be performed based on the new image recognition model, so that a decrease in object recognition accuracy due to the change in topography can be suppressed.
[0094] <Effects> Unlike the first and second embodiments, the object recognition system of the third embodiment allows a transmission command to be output from the transmission command output device 8A without operating the input device 61, thereby preventing the worker 4 from forgetting to operate the input device 61 and delaying the timing of updating the image recognition model. In other words, according to the present embodiment, when the environment of the work site 1 changes, an image recognition model suitable for the new environment can be quickly used, and deterioration of the object recognition accuracy can be continuously suppressed.
[0095] <Other> In the above embodiment, the on-site topography management system 24 and the transmission command output device 8A are configured as different devices, but they may be integrated into one device. That is, for example, a device (system) may be configured in which the processing function of the second control device 63A is added to the third control device 83, and an input device 81, a display device 82, and a communication device 84 are connected to the third control device 83.
[0096] In the above embodiment, the height change of the work site 1 is managed in units of grids divided into a lattice, but the work site 1 may be divided into shapes other than a grid and the height change may be managed. Also, in the above embodiment, the presence or absence of a change in the terrain is determined from the proportion of grids in which the amount of change in terrain height is 1m or more to the total number of grids in the entire work site, but the presence or absence of a change in terrain may be determined by monitoring only the amount of change in terrain height of the grid in which the work machine 2 is located and certain grids in the surrounding area.
[0097] In the above embodiment, the grids where the terrain height change amount is 1 m or more are clearly indicated on the screen of the display device 82 (see FIG. 18), but the screen display is not essential for the transmission command output device 8A to determine the presence or absence of terrain change based on the terrain change index value data. Therefore, the display device 82 can be omitted from the site terrain management system 24.
[0098] The site terrain management system 24 may be configured to manage the terrains of multiple work sites. In this case, the terrain data of each work site is input to the third control device 83 together with the identification data of each work site mainly via the communication device 84. Furthermore, the terrain change index value data of each work site calculated by the terrain change amount calculation unit 92c is transmitted to the transmission command output device 8A of the corresponding work site based on the identification data of the work site.
[0099] (Embodiment 4) Next, a fourth embodiment (fourth embodiment) of the invention will be described with reference to FIGS.
[0100] 20 is a configuration diagram showing the elements of the object recognition system according to this embodiment. The object recognition system according to this embodiment includes a camera 19 mounted on a work machine 2, a first control device 21A, an antenna 25 and a receiver 26 for satellite positioning, a learning server 9, a transmission command output device 8A, and a site topography management system 24.
[0101] The system of this embodiment measures the height of the work machine 2 by mounting a satellite positioning antenna 25 and receiver 26 on the work machine 2 and using a satellite positioning system (e.g., GNSS (Global Navigation Satellite System)) and grasps the status of changes in the terrain of the work site 1 around the work machine 2 from changes in height.The main feature of this embodiment is that it does not require preparing the latest terrain data each time construction progresses and inputting it into the site terrain management system 24, as in embodiment 3.
[0102] <In-vehicle device 7A> 21 is a system configuration diagram of an on-vehicle device 7A mounted on a work machine 2 of this embodiment. The on-vehicle device 7A of this embodiment has a configuration in which a GNSS antenna 25 and a receiver 26 are added as a position acquisition device for the work machine 2 to the system configuration of the on-vehicle device 7 of embodiment 1. The control device 21A of the on-vehicle device 7A functions as a position calculation unit 111a in addition to the units 32a, 32b, 32c equipped in the first control device 21 of embodiment 1.
[0103] The position calculation unit 111a calculates the position of the work machine 2 at the work site 1 using the positioning data of the work machine 2 output by the receiver 26, and transmits the calculation result to the site topography management system 24 as position data of the work machine 2. The position data also includes the position of the work machine 2 in the height direction.
[0104] <Work machine 2> 22 is a side view of the work machine 2 of this embodiment. In this figure, an antenna 25 for receiving satellite signals transmitted from a plurality of positioning satellites is attached to the upper surface of the upper rotating body 12, and a receiver 26 is mounted on the upper rotating body 12, which is connected to the antenna 25 and calculates the position (three-dimensional position including height) of the work machine 2 based on the satellite signals received by the antenna 25.
[0105] <On-site terrain management system 24> The hardware configuration of the site topography management system 24 of this embodiment is the same as that of the third embodiment (see FIG. 17), so illustration thereof is omitted.
[0106] The terrain change amount calculation unit 92c receives the position data of the work machine 2 transmitted from the first control device 21A, and selects one grid intersection closest to the horizontal position of the work machine 2 indicated by the position data. The terrain change amount calculation unit 92c sets the terrain height of the four grids adjacent to the selected intersection to the height of the received position data. In other words, the height data of the work machine 2 becomes the height data of the terrain of the work site. FIG. 23 is a diagram showing an example of a display screen of the display device 82 of this embodiment. In the example of this figure, the four grids 46 adjacent to the grid intersection closest to the work machine 2 are displayed with diagonal lines, and further an image 44 of the work machine 2 is drawn at the intersection. Note that these displays are performed by the screen generation unit 92d.
[0107] Next, the terrain change amount calculation unit 92c calculates the amount of change in terrain height (terrain height change amount) of each grid based on a certain point in time. As the certain point in time, the state of the terrain of the work site 1 before construction begins can be used as the reference. As the next certain point in time, the time when the numerical value indicated by the terrain change index value data first reaches a predetermined value can be used as the reference. Thereafter, each time the numerical value indicated by the terrain change index value data reaches a predetermined value, that timing can be used as the reference.
[0108] In addition, the terrain change amount calculation unit 92c of this embodiment can calculate the total value of the terrain height change amount of the four grids adjacent to the selected grid. The total value calculated here can be transmitted to the transmission command output device 8A as terrain change index data.
[0109] <Transmission command output device 8A> The transmission command output device 8A of this embodiment outputs a transmission command when it is determined that there has been a change in the terrain around the work machine 2 based on the height data of the work machine 2 calculated by the receiver 26 (specifically, terrain change index value data calculated based on the height data of the work machine 2). The configuration of the transmission command output device 8A is similar to that of embodiment 3 shown in Fig. 19. The processing device 74 of this embodiment also functions as an image acquisition time setting unit 74a, a terrain change determination unit 74b, and a transmission command transmitting unit 74c.
[0110] The terrain change determination unit 74b determines whether or not the terrain around the work machine 2 at the work site 1 has changed. In this embodiment, the terrain change determination unit 74b receives terrain change index value data from the site terrain management system 24 via the communication device 62, and determines that the terrain has changed if the total value of the terrain height change amount indicated by the data is equal to or greater than a predetermined value (e.g., 3 m). In other cases, it determines that there has been no terrain change. The terrain change determination unit 74b stores the determination result (presence or absence of terrain change) in the storage device 73 as terrain change data.
[0111] The image acquisition time setting unit 74a monitors whether or not there is a change in the terrain based on the terrain change data stored in the storage device 73, and if there is a change in the terrain, sets the time of the learning image to be generated when the first control device 21A receives a transmission command, and stores the set time as image acquisition time data in the storage device 73. The set time specified by the image acquisition time data may be a constant value, or may be set in proportion to the total value of the terrain height change amount indicated by the terrain change index value data. In the latter case, for example, if the total value of the received terrain height change amount is ΣΔh [m] and the set time specified by the image acquisition time data is t, the image acquisition time setting unit 74a calculates the image acquisition time by the following formula (2).
[0112] t[minutes]=10×ΣΔh (Formula 2) In this example, when the ratio indicated by the terrain change index value data is 5 m, the set time t is 15 minutes. Note that in this embodiment, the learning video time t is set to be longer as the terrain change becomes larger, but there is no clear correlation between the terrain change and the time t, and even if there is a large terrain change, it may be possible to generate a new image recognition model with a relatively short learning video.
[0113] The transmission command transmission unit 74c monitors whether or not there is a change in the terrain based on the terrain change data stored in the memory device 73, and if a change in the terrain is detected, it transmits a transmission command and image acquisition time data to the first control device 21A of the work machine 2.
[0114] <Effects> In the object recognition system of this embodiment, the elevation change of the work site 1 can be calculated by satellite positioning of the work machine 2, and the latest elevation change data of the terrain can be obtained without the operator having to input terrain data to the site terrain management system 24 as in embodiment 3. This makes it possible to avoid a situation in which the operator or the like forgets to input terrain data to the site terrain management system 24, causing a delay in the timing of updating the image recognition model. In other words, according to this embodiment, when the environment of the work site 1 changes, an image recognition model suitable for that environment can be quickly used, and degradation of object recognition accuracy can be continuously suppressed.
[0115] <Other> In this embodiment as well, the site topography management system 24 and the transmission command output device 8A may be integrated into one device.
[0116] In the above embodiment, the height change of the work site 1 is managed in units of grids divided into a lattice shape, but the height change may be managed by dividing the work site 1 into shapes other than a lattice shape.
[0117] (Embodiment 5) A fifth embodiment (embodiment 5) of the invention will be described below with reference to FIGS. 24 to 26. FIG.
[0118] 24 is a configuration diagram showing elements of the object recognition system according to this embodiment. The object recognition system according to this embodiment includes a camera 19 mounted on a work machine 2, a first control device 21A, two antennas 25a, 25b, a receiver 26, a topography measuring device 28, a tilt angle sensor 30, and a transmission command output device 8B, and a learning server 9. In this embodiment, the transmission command output device 8B is mounted on the work machine 2.
[0119] FIG. 25 is a side view of the work machine 2 of this embodiment. Two antennas 25a, 25b for satellite positioning are attached to the upper surface of the upper rotating body 12. By performing positioning of the two antennas 25a, 25b by a receiver 26, not only the position of the upper rotating body 12 but also the orientation (yaw angle) can be calculated. An inclination angle sensor 30 (e.g., an inertial measurement device) is attached to the upper rotating body 12 to detect its inclination angle (pitch angle and roll angle of the upper rotating body 12). In addition, a topography measurement device 28 (e.g., a stereo camera or LiDAR) is attached to four sides of the upper rotating body 12 at the front, rear, left and right to obtain topography data around the work machine 2 at the work site 1. These devices 26, 28, 30 are connected to the second control device 63B so as to be able to communicate with each other as shown in FIG. 26.
[0120] <Transmission command output device 8B> Fig. 26 is a schematic diagram of a transmission command output device 8B of this embodiment. The hardware configuration of the transmission command output device 8B is the same as that of the third embodiment shown in Fig. 19, but the functions of the processing device 76 include a position calculation unit 76a, an attitude calculation unit 76b, and a terrain calculation unit 76c in addition to an image acquisition time setting unit 76a, a terrain change determination unit 74b, and a transmission command transmission unit 74c.
[0121] The position calculation unit 76a determines the position of the upper rotating body 12 (for example, the three-dimensional position of the upper rotating body in a geographic coordinate system) based on the position data of the antennas 25a and 25b input from the receiver 26. The position of the upper rotating body 12 can be determined by using the position data of either one of the two antennas 25a and 25b.
[0122] The attitude calculation unit 76b calculates the azimuth angle of the upper rotating body 12 (e.g., the azimuth angle of the upper rotating body 12 in a geographic coordinate system) based on the position data of the antennas 25a, 25b input from the receiver 26. The attitude calculation unit 76b also calculates the inclination angle (e.g., pitch angle and roll angle) of the upper rotating body 12 based on the data input from the inclination angle sensor 30.
[0123] The terrain calculation unit 76c generates terrain data around the work machine 2 by using the terrain data around the work machine 2 acquired from the terrain measurement device 28, the position data of the upper rotating body 12 (work machine 2) calculated by the position calculation unit 76a, and the direction and attitude data of the upper rotating body (work machine 2) calculated by the attitude calculation unit 76b. Next, the terrain calculation unit 76c divides the generated terrain data into grids at regular intervals, for example, and calculates the height of each grid. If the position, direction, and attitude of the upper rotating body 12 are determined, the position, direction, and attitude of each terrain measurement device 28 attached to the upper rotating body 12 are also determined, so that the terrain data measured by each terrain measurement device 28 can be accurately positioned on a predetermined coordinate system (for example, a geographic coordinate system), and the height of each grid can be accurately calculated based on that.
[0124] Next, the terrain calculation unit 76c calculates the amount of change in terrain height (terrain height change amount) of each grid based on a certain point in time. As the certain point in time, the state of the terrain of the work site 1 before construction begins can be used as the reference. As the next certain point in time, the time when the numerical value indicated by the terrain change index value data first reaches a predetermined value can be used as the reference. Thereafter, each time the numerical value indicated by the terrain change index value data reaches a predetermined value, that timing can be used as the reference.
[0125] Furthermore, the terrain calculation unit 76c of this embodiment can calculate the total value of the terrain height change amount of grids (for example, the four grids closest to the work machine 2) included in a predetermined range based on the work machine 2. The total value calculated here can be transmitted to the terrain change determination unit 74b as terrain change index data.
[0126] The terrain change determination unit 74b determines whether or not the terrain around the work machine 2 at the work site 1 has changed. In this embodiment, the terrain change determination unit 74b receives terrain change index value data transmitted from the terrain calculation unit 76c, and determines that the terrain has changed if the total value of the terrain height change amount indicated by the data is equal to or greater than a predetermined value (e.g., 3 m). In other cases, it determines that there has been no terrain change. The terrain change determination unit 74b stores the determination result (presence or absence of terrain change) in the storage device 73 as terrain change data.
[0127] The image acquisition time setting unit 74a monitors whether or not there is a change in the terrain based on the terrain change data stored in the storage device 73, and if there is a change in the terrain, sets the time of the learning image to be generated when the first control device 21A receives a transmission command, and stores the set time as image acquisition time data in the storage device 73. The set time specified by the image acquisition time data may be a constant value, or may be set in proportion to the total value of the terrain height change amount indicated by the terrain change index value data. In the latter case, for example, if the total value of the received terrain height change amount is ΣΔh [m] and the set time specified by the image acquisition time data is t, the image acquisition time setting unit 74a calculates the image acquisition time by the above formula (2).
[0128] In this embodiment, the greater the change in the terrain, the longer the time t of the training image. However, there is no clear correlation between the change in terrain and the time t, and even if there is a major change in the terrain, it may be possible to generate a new image recognition model with a relatively short period of training image.
[0129] The transmission command transmission unit 74c monitors whether or not there is a change in the terrain based on the terrain change data stored in the memory device 73, and if a change in the terrain is detected, it transmits a transmission command and image acquisition time data to the first control device 21A of the work machine 2.
[0130] <Effects> Unlike the first and second embodiments, the object recognition system of the fifth embodiment allows a transmission command to be output from the transmission command output device 8B without operating the input device 61, and therefore it is possible to avoid a situation in which the worker 4 forgets to operate the input device 61, causing a delay in the timing of updating the image recognition model. In other words, according to this embodiment, when the environment of the work site 1 changes, an image recognition model suitable for that environment can be quickly used, and a decrease in the object recognition accuracy can be continuously suppressed. In particular, this embodiment does not require the site topography management system 24 as in the third and fourth embodiments, and therefore has the advantage of being easily applicable even to small-scale work sites.
[0131] <Other> In the above embodiment, the first control device 21A and the transmission command output device 8B are configured as separate devices, but they may be integrated into one device.
[0132] In the above embodiment, the height change of the work site 1 is managed in units of grids divided into a lattice shape, but the height change may be managed by dividing the work site 1 into shapes other than a lattice shape.
[0133] The present invention is not limited to having all the configurations described in the above embodiments, and includes those in which some of the configurations are omitted. Also, it is possible to add or replace some of the configurations of one embodiment with the configurations of another embodiment.
[0134] In addition, each configuration related to the above-mentioned control devices 21, 21A, 63, 63A, 63B, 83 and server 9 and the functions and execution processes of each configuration may be realized in part or in whole by hardware (for example, designing the logic for executing each function in an integrated circuit, etc.). In addition, the configuration related to the above-mentioned control devices and servers may be a program (software) that is read and executed by a processing device (for example, a CPU) to realize each function related to the configuration of the control device or server. Information related to the program can be stored, for example, in a semiconductor memory (flash memory, SSD, etc.), a magnetic storage device (hard disk drive, etc.), and a recording medium (magnetic disk, optical disk, etc.).
[0135] In addition, in the above description of each embodiment, the control lines and information lines are those that are considered to be necessary for the description of the embodiment, but they do not necessarily show all the control lines and information lines related to the product. In reality, it can be considered that almost all components are connected to each other. [Explanation of symbols]
[0136] 1...work site, 2...work machine, 3...transport machine, 4...worker, 5...site manager, 6...work object, 8...transmission command output device, 9...learning server, 11...lower running body, 12...upper rotating body, 13...front working device, 19...camera (image acquisition device), 21...first control device (controller), 22...site monitoring device, 24...site terrain management system, 25...antenna, 26...receiver, 28...terrain measuring device, 30...inclination angle sensor, 52...processing device, 52a...learning image receiving unit, 52b...recognition model generating unit, 53...storage device, 61...input device (switch), 62...communication device, 63...second control claim, 71...second control control device, 71...input / output interface, 72...processing device, 72a...image acquisition time setting unit, 72b...terrain change determination unit, 72c...transmission command transmission unit, 73...storage device, 74...processing device, 74a...image acquisition time setting unit, 74b...terrain change determination unit, 74c...transmission command transmission unit, 76...processing device, 76a...image acquisition time setting unit, 76a...position calculation unit, 76b...attitude calculation unit, 76c...terrain calculation unit, 83...third control device, 92...processing device, 92a...work site information collection unit, 92b...grid division unit, 92c...terrain change amount calculation unit, 92d...screen generation unit, 93...storage device, 101...display screen, 111a...position calculation unit
Claims
1. a learning server that generates a first image recognition model based on an image of a work site stored in a storage device; A camera for capturing images of the work site; a control device that is communicatively connected to the camera and the learning server, stores an image from the camera, and performs a process of recognizing objects in the work site based on the first image recognition model generated by the learning server and the image from the camera; In an object recognition system having A transmission command output device communicably connected to the control device, the transmission command output device outputs a transmission command to the control device based on a topographical change of the work site, the transmission command instructing the control device to transmit the image captured by the camera for a predetermined period of time to the learning server; the learning server receives the transmission command from the control device, generates a second image recognition model based on the video of the predetermined period, and transmits the generated second image recognition model to the control device; A work site object recognition system characterized in that, when the control device receives the second image recognition model from the learning server, it performs a process to recognize objects in the work site based on the image from the camera and the second image recognition model.
2. 2. The object recognition system of claim 1, The object recognition system is characterized in that the transmission command output device outputs the transmission command in response to an operation by a worker at the work site.
3. In the object recognition system of claim 2, The object recognition system is characterized in that the transmission command output device is either a smartphone, a tablet terminal, or a remote control carried by a worker at the work site, or a switch mounted on the driver's seat of a work machine operating at the work site.
4. In the object recognition system of claim 2, An object recognition system comprising a site monitoring device having the camera and the control device and installed at the work site.
5. 2. The object recognition system of claim 1, The camera and the control device are further provided in a work machine, The object recognition system is characterized in that the transmission command output device determines whether or not there has been a change in the terrain around the work machine based on terrain data of the work site, and outputs the transmission command if it determines that there has been a change in the terrain around the work machine.
6. The object recognition system according to claim 5, The work machine is equipped with an antenna for receiving satellite signals transmitted from a plurality of positioning satellites, a receiver for calculating height data of the work machine based on the satellite signals received by the antenna as topographical data of the work site, and the transmission command output device, An object recognition system characterized in that the transmission command output device outputs the transmission command when it is determined that there has been a change in the terrain around the work machine based on height data of the work machine.
7. The object recognition system according to claim 5, The work machine is equipped with a terrain measuring device that acquires terrain data around the work machine and the transmission command output device, An object recognition system characterized in that the transmission command output device outputs the transmission command when it is determined that there has been a change in the terrain around the work machine based on the terrain data.
8. The object recognition system according to claim 7, The object recognition system is characterized in that the topographical measurement device is a stereo camera or a LiDAR.
Citation Information
Patent Citations
Learning system, on-vehicle device, and server
JP2015135552A
Device and method of creating map for work machine
JP2017049172A
Obstacle detection system for heavy equipment
JP2019049918A
Notification device, information processing device, information processing system, information processing method, and information processing program
JP2019197373A
Excavator and excavator assisting system
WO2020091002A1
