Object recognition system

Abstract

This object recognition system comprises: a camera that captures an image of a work site; a control device that accepts input of the image acquired by the camera and recognizes a recognition target in the image using an object recognition model trained by machine learning; and a learning device that stores learning images captured at the work site and generates the object recognition model on the basis of the learning images. The control device uses the object recognition model, transmitted from the learning device, as a new object recognition model. The learning device additionally stores imaging position information of the learning images, generates imaging position distribution data indicating the distribution of the imaging positions of the learning images used to generate the object recognition model, on the basis of the imaging position information of the learning images used to generate the object recognition model, among the imaging position information, and displays the imaging position distribution data on a monitor.

Description

Object Recognition System

[0001] The present invention relates to an object recognition system that is applied to, for example, a site management system or a work machine.

[0002] In recent years, advances in computer performance have led to significant advances in machine learning image recognition technology using deep neural networks and the like. Even at worksites where work machines such as hydraulic excavators and transport vehicles such as dump trucks are in operation, machine learning-based object recognition technology is increasingly being introduced in the form of on-board equipment on the work machines and site sensors installed on-site. This type of technology involves capturing images of the target object (e.g., a person) at the worksite to create training data, generating a recognition model of the target object (e.g., a recognition model consisting of a set of parameters related to image recognition processing) based on the training data through machine learning, and then using the recognition model to recognize the target object from the image. There is great promise for maintaining and improving the recognition accuracy (detection accuracy) of the target object by updating the recognition model even after the site sensor or on-board equipment of the work machine is shipped.

[0003] Regarding updating of recognition models, Patent Document 1 discloses a learning system including multiple onboard devices mounted on vehicles traveling on roads and performing image recognition processing, and a server that updates parameters (recognition models) related to the image recognition processing based on learning data (learning images), which are captured image data used in the image recognition processing on the onboard devices. The server includes a main learning server that performs learning processing based on learning data from all driving regions, and multiple sub-learning servers that perform learning processing for each group based on learning data categorized by driving region. The main learning server classifies the learning data into groups corresponding to the driving regions and transmits the classified learning data to the sub-learning servers corresponding to each group. Each sub-learning server performs learning processing (refinement) based on the received learning data, using the learning results of the main learning server as initial values, and generates parameters for image recognition processing in the driving region corresponding to the sub-learning server. The main learning server uses the parameters corresponding to each driving region as update data and transmits the update data to each onboard device. Each onboard device updates the parameters for image recognition processing based on the received update data. As described in Patent Document 1, this allows the parameters of the on-board device to be refined to suit the region in which the vehicle is traveling, and the image recognition process is optimized for each region in which the vehicle is traveling.

[0004] Japanese Patent Application Laid-Open No. 2015-135552

[0005] Work progress at a work site typically changes locally. Furthermore, compared to the roads for automobiles that Patent Document 1 addresses, a work site is characterized by the fact that the terrain and scenery change at a much faster rate. For example, in the case of work that involves significant terrain changes, such as excavation, the scenery may change dramatically within a few days. Therefore, in order to update a recognition model after work begins and maintain recognition accuracy, it is important to determine the location of images from which the new recognition model was generated before deciding to use it. In other words, even if a new recognition model is generated based on images captured in an area of ​​the work site with little or no change, changing to that recognition model is unlikely to maintain or improve recognition accuracy.

[0006] The system in Patent Document 1 changes the recognition model (parameters) for each driving area, but it is not possible to know which image within the driving area each recognition model was generated based on. Therefore, even if the technology in Patent Document 1 is applied to object recognition at a work site, there is a high possibility that detection accuracy will decrease.

[0007] An object of the present invention is to provide an object recognition system that can determine where in a work site an image taken has been used to generate a recognition model used for object recognition based on image recognition.

[0008] The present application includes multiple means for solving the above-mentioned problems, and one example thereof is an object recognition system comprising: a camera that is installed at a work site and photographs the work site; a first control device that inputs images acquired by the camera and recognizes a recognition target on the image using an object recognition model trained by machine learning; and a second control device that stores training images photographed at the work site and generates the object recognition model based on the training images, wherein the first control device uses the object recognition model transmitted from the second control device as a new object recognition model; the second control device further stores shooting position information of the training images, and generates shooting position distribution data indicating the distribution of shooting positions of the training images used to generate the object recognition model based on the shooting position information of the training images used to generate the object recognition model, and displays the data on a monitor.

[0009] According to the present invention, it is possible to understand where within the work site the recognition model was generated based on an image taken, thereby suppressing a decrease in the recognition accuracy of the object to be recognized that occurs when the recognition model is updated.

[0010] 1 is a schematic configuration diagram of an object recognition system for a work site according to an embodiment of the present invention. A configuration diagram showing an example of a control device 22 and its peripheral devices. A configuration diagram showing an example of a learning device 23 and its peripheral devices. A schematic diagram of a work site 1 where a work machine 2 equipped with a control device 22, a camera 19, and a monitor 21 operates. A schematic configuration diagram of the work machine 2 in FIG. 4. A flowchart showing the processing of the object recognition unit 32b (control device 22). A diagram showing an example of a case where an object to be recognized (a person) is included in an image captured by the camera. A flowchart of the processing by the control device 22 in which the image data storage unit 32a stores learning images and shooting position information acquired by the camera 19 and transmits them to the learning device 23. A flowchart showing the processing of the learning device 23 (recognition model generation unit 35d). An explanatory diagram of deep learning. A flowchart showing an example of the processing of a position distribution data generation unit 35c. A flowchart showing an example of a processing by the learning device 23 to transmit position distribution data and a recognition model. An example of display of shooting position distribution data 62a on the monitor 21 of the control device 22. An example of display of shooting position distribution data 62b on the monitor 21 of the control device 22. 1 is a schematic configuration diagram of an object recognition system at a work site according to a second embodiment. FIG. 2 is a schematic configuration diagram of an example of an image acquisition device 6. FIG. 3 is a configuration diagram showing an example of a control device 122 and its peripheral devices. FIG. 4 is another example of displaying shooting position distribution data 62a on the monitor 21 of the control device 22. FIG. 5 is a schematic configuration diagram of a work progress management server 29 and its peripheral devices according to a third embodiment. FIG. 6 is a flowchart showing the processing of a learning device 23 (image data receiving unit 35a).

[0011] Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

[0012] <Object Recognition System> Fig. 1 is a schematic diagram of an object recognition system for a work site according to an embodiment of the present invention. The object recognition system in Fig. 1 includes a camera 19 that captures images of the work site, a control device (first control device) 22, a learning device (second control device) 23, and a monitor 21.

[0013] The control device 22 (first control device) is, for example, a computer including a processing device (a processor such as a CPU), a storage device (e.g., a memory), and an input / output interface (input / output unit). The control device 22 receives an image acquired by the camera 19 and recognizes a recognition target (e.g., a person) in the image using an object recognition model trained by machine learning. The control device 22 is, for example, a computer connected to a controller (on-board device) mounted on a work machine such as a hydraulic excavator at a work site and used to control the work machine, or a camera (site sensor) installed at the work site.

[0014] The learning device 23 (second control device) is, for example, a computer equipped with a processing device (a processor such as a CPU), a storage device (e.g., a memory), and an input / output interface (input / output unit). The learning device 23 stores a plurality of images (learning images) taken at the work site and generates an object recognition model based on the learning images. The object recognition model generated here can be transmitted to the control device 22 and used by the control device 22 to recognize the recognition target.

[0015] The learning device 23 stores shooting position information, which is information about the location where each training image was captured. Based on the shooting position information of the training images used to generate the object recognition model, the learning device 23 generates shooting position distribution data (details will be described later, for example, the screen display indicated by reference numeral 62 in FIG. 1 ) indicating the distribution of shooting positions of the training images used to generate the object recognition model, and displays the data on the monitor 21. The shooting position distribution data can be generated for each object recognition model. Before the control device 22 starts using a new object recognition model, it is preferable to display the shooting position distribution data of the object recognition model on the monitor 21 and confirm that training images captured in the area where object recognition will be performed are included in the training images used to generate the object recognition model.

[0016] The monitor 21 may be connected to the control device 22, the learning device 23, or a terminal different from the control device 22 and the learning device 23, for example. In this embodiment, an example will be given in which the monitor 21 is connected to the control device 22 and a monitor 21b separate from the monitor 21 is connected to the control device 23.

[0017] The learning device 23 is, for example, a server installed in a control center installed at a work site or in a remote location.

[0018] The control device 22 and the learning device 23 are configured to be able to communicate with each other. For example, the control device 22 and the learning device 23 each have a communication device such as a Wi-Fi antenna and are connected wirelessly via a network 24. The network 24 is, for example, the Internet or a local area network. Note that the form of communication between the network 24 and the control device 22 and learning device 23 is not limited to wireless communication, and wired communication may also be used.

[0019] 2 is a configuration diagram showing an example of the control device 22 and its peripheral devices. The control device 22 includes a processing device 32, a storage device 33, and an input / output unit (input / output interface) 31. Connected to the control device 22 are a camera (imaging device) 19 that captures images of the work site, a communication device 20, a monitor (display device) 21, a positioning device 26, and an input device 27.

[0020] The input / output unit 31 is an interface for connecting peripheral devices such as the camera 19 , the communication device 20 , the monitor 21 , and the input device 27 to the control device 22 .

[0021] The processing device 32 can be realized by, for example, a combination of a CPU (Central Processing Unit) and a GPU (Graphics Processing Unit).

[0022] The storage device 33 can be configured with a memory such as a RAM (Random Access Memory) and a storage such as a HDD (Hard Disk Drive) or an SSD (Solid State Drive). The storage device 33 is connected to the processing device 32 and is configured to enable writing and reading processes by the processing device 32. Source files and programs used by the processing device 32 to perform various calculations are also stored in the storage device 33.

[0023] The control device 22 can function as an image data storage unit 32a, an object recognition unit 32b, a recognition model update unit 32c, and a position calculation unit 32d by executing a program stored in the storage device 33 in the processing unit 32. For convenience, each of the units 32a to 32d is illustrated in FIG. 2 inside the processing unit 32. The processes that can be executed by each of the units 32a to 32d will be described later.

[0024] <Camera 19> The camera 19 is preferably installed at the work site and fixed at a predetermined height in a predetermined orientation. The orientation of the camera 19 is preferably determined by a pitch angle (tilt angle) and a yaw angle (pan angle). The object to which the camera is fixed may be a mobile work machine (e.g., a hydraulic excavator or a dump truck) operated within the work site, or a stand installed within the work site (however, the stand is portable). The camera 19 may be, for example, a digital camera equipped with a CMOS (Complementary Metal Oxide Semiconductor) image sensor.

[0025] <Communication Device 20> The communication device 20 is hardware for communicating with other devices, including the learning device 23, via a network 24. The communication device 20 is also called, for example, a network device, a network controller, a network card, a communication module, etc. The communication device 20 may be equipped with a connector for wired connection, or may be equipped with a wireless communication interface. For example, the communication device 20 may be equipped with an antenna compatible with LTE-Advanced, a Wi-Fi antenna, or the like.

[0026] <Monitor 21> The monitor 21 displays position distribution data that indicates the distribution of positions of the recognition target of the object recognition system on a plurality of images (for example, images taken by the camera 19) in which the recognition target is captured.

[0027] The monitor 21 may be a device that has an input function in addition to a display function. That is, for example, a capacitive touch display that allows input operations with the operator's finger. However, the monitor 21 is not limited to a touch panel display, and may be configured by combining a display with an input device such as a joystick or switch.

[0028] <Positioning Device 26> The positioning device 26 receives signals from multiple positioning satellites, calculates a position, and transmits the calculation results (position data) to the control device 22. The positioning device 26 can be, for example, a Global Navigation Satellite System (GNSS)-compatible antenna and receiver. The receiver can calculate the position of the antenna based on the positioning signal received by the antenna. For example, if an antenna is installed on the camera 19, the position of the camera 19 can be calculated. If the camera 19 and the control device 22 are mounted on a mobile object such as a work machine and it is desired to accurately calculate a desired position on the mobile object, including their positions, it is preferable to install two antennas on the mobile object and calculate the positions of the two antennas using a receiver. Note that if the camera 19 and the control device are mounted on a stationary object and the position of the stationary object is known, the positioning device 26 can be omitted from the system.

[0029] <Input Device 27> The input device 27 connected to the control device 22 is not limited to a specific type, but may be, for example, a keypad or a touch panel. It may also be a combination of input devices such as a joystick and a switch, or may be a mouse or keyboard. An input signal is sent to the control device 22 in response to an input operation on the input device 27.

[0030] 3 is a configuration diagram showing an example of the learning device 23 and its peripheral devices. The learning device 23 includes a processing device 35, a storage device 36, and an input / output unit (input / output interface) 34. The learning device 23 is connected to a communication device 20b, a monitor (display device) 21b, and an input device 25b.

[0031] The input / output unit 34 is an interface for connecting peripheral devices such as the communication device 20b, the monitor (display device) 21b, and the input device 25b to the learning device 23.

[0032] The processing device 35 can be realized by, for example, a combination of a CPU and a GPU.

[0033] The storage device 36 can be configured with a memory such as a RAM and a storage such as an HDD or SSD. The storage device 36 is connected to the processing device 35 and is configured to enable writing and reading processes by the processing device 35. Source files and programs used by the processing device 35 to perform various calculations are also stored in the storage device 36.

[0034] The learning device 23 can function as an image data receiving unit 35a, a position distribution data generating unit 35c, and a recognition model generating unit 35d by executing a program stored in the storage device 36 in the processing device 35. For convenience, each of the units 35a-35d is illustrated in FIG. 3 as being inside the processing device 35. The processes that can be executed by each of the units 35a-35d will be described later.

[0035] There are no restrictions on the location where the learning device 23 is installed, as long as it is a location that can be connected to the network 24. It may be installed either outside the work site or inside the work site.

[0036] <Communication Device 20b> The communication device 20b is hardware for communicating with other devices, including the control device 22, via the network 24. The communication device 20b is also referred to as, for example, a network device, a network controller, a network card, a communication module, etc. The communication device 20b may be equipped with a connector for wired connection, or may be equipped with a wireless communication interface. For example, the communication device 20b may be equipped with an antenna compatible with LTE-Advanced, a Wi-Fi antenna, etc.

[0037] <Monitor 21b> The monitor 21b receives a signal from the learning device 23 and displays the screen. The monitor 21b is, for example, a liquid crystal display. However, similar to the monitor 21 of the control device 22, for example, a capacitive touch display may also be used.

[0038] <Input device 25b> The input device 25b connected to the learning device 23 is not limited to a specific type, but may be, for example, a mouse or a keyboard. It may also be combined with other input devices such as a joystick or a switch. An input signal is sent to the learning device 23 in response to an input operation on the input device 25b.

[0039] In addition, if the learning device 23 is remotely connected to another device connected to the same network 24 using SSH (Secure Shell) or RDP (Remote Desktop Protocol), the monitor 21b and input device 25b can use the display device and input device of the other device.

[0040] <Worksite 1> Figure 4 is a schematic diagram of a worksite 1 where a work machine (hydraulic excavator) 2 equipped with a control device 22, a camera 19, and a monitor 21 operates. The worksite 1 is home to the work machine 2, a worker 3, a transport machine (dump truck) 4, and the like. The work machine 2 performs tasks such as leveling and excavation work, and loading work objects 5 onto the transport machine 4 at predetermined locations. The transport machine 4 is, for example, a dump truck, and transports the work objects 5 and the like while entering and leaving the worksite 1. The worker 3 also performs detailed tasks around the work machine 2, such as guiding various machines, cleaning up work objects 5 spilled from the work machine 2, and transporting various tools.

[0041] <Working machine 2> Figure 5 is a schematic configuration diagram of the working machine (hydraulic excavator) 2 in Figure 4. The working machine 2 comprises a lower traveling body 11, an upper rotating body 12 rotatably mounted on the lower traveling body, a front working mechanism 13 comprising a boom 15, an arm 16 and a bucket 17, and a control cabin 18 where an operator rides to operate the working machine 2. The front working mechanism 13 comprises a boom 15 rotatably mounted in front of the upper rotating body 12, an arm 16 rotatably mounted at the tip of the boom 15, and a bucket (working implement) 17 rotatably mounted at the tip of the arm 16. The bucket 17 is an example of a working implement (attachment) of the front working mechanism 13 and can be replaced with a grapple, breaker or the like.

[0042] A camera 19 capable of acquiring images of the surroundings of the work machine 2, a communication device 20, a monitor (display device) 21 located inside the cockpit 18, a positioning device 26, and a control device 22 are installed on the upper rotating body 12. In the present embodiment, an example has been given in which the monitor 21 is installed inside the cockpit 18, but the installation location of the monitor 21 is not limited to inside the cockpit 18. Furthermore, the monitor 21 may be connected to the control device 22 via wireless communication.

[0043] The camera 19 is installed on the upper rotating body 12, and is installed so as to capture an image of the area behind the work machine 2, for example, as shown in Figure 5. The installation position of the camera 19 does not have to be the rear of the upper rotating body 12, and multiple cameras may be installed.

[0044] 6 and 7, a process in which the control device 22 recognizes a recognition target object from an image captured by the camera 19 using the object recognition unit 32b will be described. Fig. 6 is a flowchart illustrating the process performed by the object recognition unit 32b (control device 22).

[0045] In step S201, the object recognition unit 32b (control device 22) acquires an image using the camera 19, and the process proceeds to step S202.

[0046] In step S202, the object recognition unit 32b performs a recognition process to infer whether a recognition target exists in the acquired image using the image acquired in step S202 and a recognition model (object recognition model) stored in the storage device 33, and then proceeds to step S203. As shown in FIG. 7A, when an image including a worker (person) 51, which is the recognition target, is acquired, the recognition process is performed using the recognition model stored in the storage device 33. The recognition model is defined, for example, by a neural network and parameters. The recognition result includes, for example, whether the recognition target exists, the coordinates of a rectangle (bounding box) surrounding the recognition target (i.e., the position of the recognition target in the image), the type of recognition target (e.g., whether it is a person, a transport machine, a construction machine, etc.), and the degree of recognition certainty. The coordinates of the bounding box can be any coordinate of any point related to the rectangle that contains the target, such as the two vertices located at both ends of the diagonal of the rectangle that contains the target, the midpoint of the base of the rectangle, any vertex of the rectangle, or the center of the rectangle.

[0047] In step S203, the object recognition unit 32b outputs the recognition result acquired in step S202. For example, as shown in Fig. 7B, the bounding box 52 and the type 53 of the recognized object can be displayed and output on the acquired image on the monitor 21 in a superimposed manner.

[0048] <Data storage process of control device 22> Next, using the flowchart of Figure 8, we will explain the process in which the control device 22 uses the image data storage unit 32a to store image data (learning image) captured by the camera 19 and location data (shooting location information) where the image data was captured in the memory device 33 and transmit them to the learning device 23.

[0049] In step S101, the image data storage unit 32a (control device 22) determines whether an image saving start operation (e.g., a button operation) has been operated (input) by the input device 27. When the operator operates the image saving start operation, an image saving start signal is input to the image data storage unit 32a. If the image data storage unit 32a determines that the image saving start operation has been operated, the process proceeds to step S102. If the image saving start operation has not been operated, the process of step S101 is repeated.

[0050] In step S102, the image data storage unit 32a acquires an image captured by the camera 19, and the position calculation unit 32d calculates the position data (photography position information) at which the image was captured based on the position data input from the positioning device 26. After acquiring the learning image from the camera 19 and calculating the photography position information of the learning image, the process proceeds to step S103.

[0051] Returning to FIG. 8 , in step S103, the image data storage unit 32a associates the position data (shooting position information) calculated in step S102 with the image (learning image) acquired in step S102 and stores the image in the storage device 33. This associates the shooting position information with each learning image. Note that the image storage format is not limited, and may be a still image or a video. When storing the image, compression such as AVC (Advanced Video Coding) or JPEG (Joint Photographic Experts Group) may be performed, and a resolution may be specified. Thereafter, the process proceeds to step S104.

[0052] In step S104, the image data storage unit 32a determines whether an image storage end operation (e.g., a button operation) has been input from the input device 27. If an image storage end operation has been input to the image data storage unit 32a (control device 22), the process proceeds to step S105. If an image storage end operation has not been input, the process returns to step S102, and the process of acquiring and storing learning images and shooting position information is repeated.

[0053] In step S105, the image data storage unit 32a uses the communication device 20 to transmit the learning images and shooting position information stored in the storage device 33 between the time when the image storage start operation was input and the time when the image storage end operation was input to the learning device 23. Thereafter, the process returns to step S101.

[0054] Here, images from the camera 19 mounted on the work machine 2 are used as learning images, but any images taken by a camera with the same sensor size, focal length, installation height and installation angle (pitch angle) as the camera 19 can be used as learning images for the object recognition model of the work machine 2.

[0055] In addition, although the image saving start operation and the image saving end operation are input via the input device 27 of the control device 22, they may also be input via the input device 25b of the learning device 23 or another input device. Furthermore, a signal input from the control device 22 or the learning device 23 when a predetermined condition is satisfied may be regarded as the image saving start operation or the image saving end operation.

[0056] <Data Storage Process of Learning Device 23> When the image data receiving unit 35 a of the learning device 23 receives the learning image and the shooting position information from the control device 22 , the image data receiving unit 35 a stores the received learning image and the shooting position information in the storage device 36 .

[0057] <Learning Process (Recognition Model Generation Process) of Learning Device 23> Next, the process performed by the recognition model generation unit 35d of the learning device 23 will be described with reference to FIGS. 9 and 10. FIG.

[0058] FIG. 9 is a flowchart showing the processing of the learning device 23 (recognition model generating unit 35d).

[0059] In step S801, the recognition model generation unit 35d performs processing to determine whether a learning start operation has been input from the input device 25b. If a learning start operation has been input, the processing proceeds to step S802. If a learning start operation has not been input, the processing returns to step S801.

[0060] In step S802, the recognition model generation unit 35d performs machine learning using the training images in the storage device 36 to generate a new recognition model. The training images may be used in any manner. For example, bounding boxes may be manually assigned to the recognition targets in the training images, and class information for the recognition targets may be manually assigned to generate training data, which may then be used for training. The training data is training images to which correct answer information (e.g., position information and class information for the recognition targets) has been assigned.

[0061] Here, deep learning, for example, is used to generate the recognition model. When deep learning is used, as shown in FIG. 10 , a predefined deep neural network 92 is trained by tuning parameters to minimize the difference (e.g., the deviation in the coordinate values ​​of the bounding box) between the inference result of the input image 91 and the correct answer information 93, as shown in FIG. 10 . The recognition model can be defined, for example, by the deep neural network structure and parameters used. Using the trained recognition model shown in FIG. 10 , it is possible to infer information about the bounding box 93 and class information of the object to be recognized from the input image 91, and thus the position and type of the object to be recognized can be inferred. Once a new recognition model has been generated through machine learning, the process proceeds to step S803.

[0062] In step S803, the recognition model generation unit 35d transmits a recognition model generation signal indicating that a new recognition model has been generated to the position distribution data generation unit 35c. At this time, the signal may also be transmitted to the control device 22 using the communication device 20b. When transmission of the recognition model generation signal is complete, the process returns to step S801.

[0063] In addition to the above, the timing for starting learning using teacher data may be whenever the image data receiving unit 35 a receives a new learning image. However, the timing is not limited to this, and for example, the update status of new learning images may be checked at regular intervals, and learning may be started when new learning images are present.

[0064] Furthermore, once the generation of a new recognition model is complete, identification information for the new recognition model may be assigned to each learning image used in the generation, in order to facilitate the generation process of position distribution data 62, which will be described later.

[0065] <Process for generating position distribution data of recognition target by learning device 23> The learning device 23 (position distribution data generation unit 35c) performs a process for generating shooting position distribution data indicating the distribution of shooting positions of the learning images used to generate a new recognition model, based on the shooting position information of the learning images stored in the storage device 36 that were used to generate the new recognition model.

[0066] The photographing position distribution data is data generated by the position distribution data generator 35c that compiles the photographing positions of the training images used to generate a specified recognition model and presents the distribution in the form of text (including numbers), graphs, figures, and the like. The photographing position distribution data exists for each recognition model and can be displayed in multiple formats. For example, the work site 1 is divided into multiple regions, and the regions in which the training images used to generate the specified recognition model were photographed are identified based on the photographing position information. The amount of training images containing photographing position information for each region (e.g., the number or percentage) is counted, and the counted values ​​(numerical values) are linked to the regions to form the position distribution data. Therefore, the photographing position distribution data also includes a frequency distribution map showing the distribution of the photographing positions of the training images used in the recognition model on a map of the work site 1. An example of such a frequency distribution map is the position distribution data 62b in FIG. 13, which will be described later.

[0067] Next, an example of the processing executed by the position distribution data generating unit 35c will be described with reference to Fig. 11. Fig. 11 is a flowchart showing the processing of the position distribution data generating unit 35c.

[0068] In step S301, the position distribution data generation unit 35c (learning device 23) determines whether or not it has received a recognition model generation signal from the recognition model generation unit 35d, and if it has received a recognition model generation signal, it proceeds to step S302.

[0069] In step S302, the position distribution data generation unit 35c generates shooting position distribution data 62 (see FIGS. 13 and 14, which will be described later) for the recognition model based on shooting position information of the training images used in generating the new recognition model that triggered the transmission of the recognition model generation signal. At this time, other information (e.g., shooting time information of the training images used in generating the recognition model) to be displayed on the monitor 21 together with the shooting position distribution data 62 may be calculated.

[0070] In step S303, the position distribution data generation unit 35c stores the shooting position distribution data 62 generated in step S302 in the storage device 36. If other information to be displayed on the monitor 21 together with the shooting position distribution data 62 is generated in step S302, it is preferable to store the other information in the storage device 36 as well.

[0071] <Processing for transmitting position distribution data and recognition model by learning device 23> Based on a transmission command for position distribution data, the learning device 23 (position distribution data generation unit 35c) transmits position distribution data 62 of the recognition model specified by the transmission command. Then, based on a recognition model update command input thereafter, the learning device 23 (recognition model generation unit 35d) performs processing for transmitting the recognition model specified by the update command.

[0072] 12 is a flowchart showing an example of a process for transmitting position distribution data and a recognition model by the learning device 23. The learning device 23 executes the process of FIG. 12 at a predetermined control cycle.

[0073] When the process starts, in step S401, the learning device 23 determines whether or not a position distribution data transmission command has been received. If it is determined that a position distribution data transmission command has been received, the process proceeds to step S402. The position distribution data transmission command is a command output by the control device 22 or the learning device 23 to instruct the learning device 23 to transmit position distribution data of a specified identification model, and is output, for example, by operating the respective input devices 27, 25b. This operation includes a designation operation to designate one recognition model from the recognition models stored in the learning device 23.

[0074] In step S402, the learning device 23 selects the position distribution data 62 of the recognition model specified by the position distribution data transmission command from the storage device 36 and transmits it to the sender of the position distribution data transmission command (the control device 22 or the learning device 23). As a result, the position distribution data 62 of the recognition model is displayed on the monitor of the sender of the position distribution data transmission command.

[0075] 13 and 14 show examples of how the shooting position distribution data 62 is displayed on the monitor 21 of the control device 22. The display screen 101 in Fig. 13 displays identification information 61 of the recognition model, shooting position distribution data 62a of the recognition model, a legend 63a for the shooting position distribution data, shooting time information (image acquisition date) 64 of the learning image used to generate the recognition model, and an update button 65 for the recognition model.

[0076] The identification information 61 indicates the identification information of the recognition model specified in the position distribution data transmission command.

[0077] The photographing position distribution data 62a shows the distribution of positions where the training images used to generate the recognition model to which the identification information 61 is assigned were photographed on a map of the work site 1. In the example shown in the figure, the map of the work site 1 (which may be a construction drawing, etc.) is divided into multiple cells (areas) using a grid, and the display of the cells (e.g., hatching pattern or color) is changed depending on the amount of training images photographed within the cell. The display of each cell according to the amount of training images is as shown in legend 63a. In the example of FIG. 13, the display (hatching pattern) of the cell is changed depending on the number of images photographed in that cell relative to the total number of training images used to generate the recognition model (i.e., the usage rate).

[0078] The shooting time information 64 indicates the shooting time information of the training image used to generate the recognition model to which the identification information 61 is assigned, and is generated based on the shooting time data linked to the training image in the storage device 36. In the illustrated example, the shooting date (image acquisition date) is displayed as the shooting time information. Note that if the shooting date spans multiple days, the corresponding period may be displayed on the screen.

[0079] The update button 65 is a button that is operated when updating the recognition model used by the control device 22 for object recognition to a recognition model (new recognition model) related to the shooting position distribution data 62 displayed on the screen. When the update button 65 is pressed, a recognition model update command is output to the learning device 23.

[0080] When the shooting position distribution data 62 is displayed on the monitor 21b of the learning device 23, it is preferable to display a selection request screen for the control device 22 to which the recognition model is to be sent at the timing when the update button 65 is pressed. Once the destination control device 22 has been specified on this screen, a recognition model update command is output together with the identification information of the destination control device 22.

[0081] The display screen 102 in Fig. 14 displays shooting position distribution data 62b in a different display format from that in Fig. 13. The shooting position distribution data 62b shows the number of learning images taken in each cell as a three-dimensional bar graph.

[0082] In addition, if the learning images stored in the memory device 36 of the learning device include images acquired at other work sites, the shooting location information of the learning images acquired at the other work sites will be excluded, and only the shooting location information related to work site 1 will be displayed.

[0083] Returning to the description of Figure 12, in step S403, the learning device 23 determines whether or not a recognition model update command has been received. The recognition model update command is a signal that is output when the update button 65 (see Figures 13 and 14) is operated. This signal also includes identification information for the new recognition model that was displayed in the shooting position distribution data 62. If it is determined in step S403 that a recognition model update command has been received, the process proceeds to step S404.

[0084] In step S404, the learning device 23 selects the recognition model (new recognition model) specified in the recognition model update command from the storage device 36 and transmits it to the control device 22 specified in the recognition model update command. As a result, the recognition model of the specified control device 22 is updated to the transmitted one (new recognition model).

[0085] (Effects Obtained by First Embodiment) When using the system of the first embodiment, by displaying and checking the photographing position distribution data 62 of a specified recognition model on a monitor, it is possible to confirm at which position in the work site 1 the recognition model was generated from the training images, and then update to that recognition model. In other words, if the recognition model was generated from training images taken in an area where work is planned to be performed, it is possible to prevent a decrease in the recognition accuracy of the recognition target after updating to that recognition model. In other words, according to this embodiment, it is possible to know which position in the work site the recognition model was generated based on, and therefore it is possible to prevent a decrease in object recognition accuracy associated with the use of that recognition model. In other words, according to this embodiment, it is possible to select a recognition model that can improve the recognition accuracy of the recognition target at the work site.

[0086] Note that, in the above explanation, the photographing position distribution data 62 is mainly displayed on the monitor 21 mounted on the work machine 2, and the operator considers whether or not to update the recognition model. However, the photographing position distribution data 62 may also be displayed on the monitor 21b of the learning device 23, and the operator of the learning device 23 (e.g., a service provider) may consider whether or not to update the recognition model. Alternatively, the photographing position distribution data 62 may be displayed on a monitor such as a mobile terminal, and a worker other than the operator at the work site (e.g., a site manager) may view the monitor and consider whether or not to update the recognition model. Furthermore, a computer including the control device 22 and learning device 23 may automatically decide whether or not to update the recognition model based on the distribution of photographing positions in the photographing position distribution data 62 (i.e., in this case, a decision is not made via a "screen").

[0087] 15-17, a second embodiment of the present invention will be described. The main feature of this embodiment is that a camera 19 installed at the work site 1 captures learning images, and a control device (third control device) 122 connected to the camera 19 transmits the learning images and shooting position information to a learning device 23. The learning device 23 has the same configuration as in the first embodiment, so a description thereof will be omitted.

[0088] 15 is a diagram showing the relationship between a learning device (second control device) 23, a control device (first control device) 22, and a control device (third control device) 122 in this embodiment. The object recognition system of embodiment 2 corresponds to a configuration in which a control device (third control device) 122 is added to embodiment 1. A camera 19 supported by a stand or the like and installed at the work site 1 is connected to the control device 122.

[0089] 16 is a schematic diagram of an example of an image acquisition device 6 including a camera 19 installed at a work site 1 and a control device 122 connected to the camera 19. The illustrated image acquisition device 6 is a site sensor that is a portable camera, and includes a sensor box 105 and a stand 106 that supports the sensor box 105. The sensor box 105 is equipped with the camera 19, a communication device 20, a control device 122 to which the camera 19 and the communication device 20 are connected, and a positioning device 26. The control device 122 may be capable of wireless communication with, for example, a mobile terminal (not shown) and may be controlled based on operations input to the mobile terminal. The mobile terminal may be carried by a worker at the work site 1.

[0090] Fig. 17 is a configuration diagram showing an example of the control device (third control device) 122 and its peripheral devices in Fig. 16. The control device 122 corresponds to the first control device 22 of the first embodiment shown in Fig. 2 , with the input device 27 and the monitor 21 omitted as hardware, and with the object recognition unit 32b and the recognition model update unit 32c omitted as functions (software) executed by processing of the processing device 32.

[0091] The control device 122 is configured to be able to transmit learning images and their shooting position information to the learning device 23 by executing the flowchart shown in Figure 8. Note that the image saving start operation in step S101 and the image saving end operation in step S104 may be input at the desired timing from the control device 22 or learning device 23 mounted on the work machine 2, or may be input from the mobile terminal described above.

[0092] (Effects obtained in embodiment 2) As described above, this embodiment includes a camera (installed camera) 19 that is installed at a desired position in the work site 1 and takes learning images, a positioning device 26 that receives signals from multiple positioning satellites and calculates shooting position information of the learning images taken by the camera 19, and a control device (third control device) 122 that is connected to the installed camera 19 and the positioning device 26 and can communicate with the learning device (second control device) 23, and the control device (third control device) 122 is configured to transmit the learning images taken by the installed camera 19 and the shooting position information calculated by the positioning device 26 to the learning device (second control device) 23.

[0093] Even when an object recognition system is configured using such an installed camera 19, the same effects as in embodiment 1, which uses the camera 19 mounted on the work machine 2, can be obtained. A particular advantage is that an object recognition system can be constructed by installing the installed camera 19 at a desired location within the work site 1, independent of the work machine 2. For example, one advantage is that the camera 19 can be installed depending on the situation at the site (construction progress, whether the work requires human intervention, etc.).

[0094] It is preferable that the difference between the camera specification parameters, including the sensor size, focal length, installation height, and installation pitch angle, of the camera (installed camera) 19 of the site sensor 6 and those of the other cameras 19 used at the work site 1 be within a predetermined range, and it is preferable that there is no difference between the camera specification parameters and those of the other cameras 19 (in other words, in this case, the parameters are the same). For example, if the camera specification parameters are the same as those of the camera 19 mounted on the work machine 2 in Figure 5, images taken by one camera 19 can be used as learning images for the other camera 19. It is to be noted that the "difference" in the above is calculated for each parameter, and the "predetermined range" can be set for each parameter. It is preferable that the "difference" be as small as possible.

[0095] Although the above describes a case where object recognition is not performed by the control device 122, it is also possible to enable object recognition by the control device 122 by providing an object recognition unit 32b and a recognition model update unit 32, similar to the control device 22 of embodiment 1.

[0096] 18 and 19, a third embodiment of the present invention will be described. This embodiment is a modification of the first embodiment, and relates to another display format of the shooting position distribution data 62 on the monitor 21.

[0097] 18 shows another example of the display of the photographing position distribution data 62 on the monitor 21 of the control device (first control device) 22. The same parts as in the previous figure are given the same reference numerals.

[0098] 18 shows a screen 101A in which a display switching button 66 is added to the screen 101 shown in FIG. 13, and a screen 103 to which the screen 101A is switched by inputting an operation to press the display switching button 66 on the screen 101A. The display switching button 66 is also displayed on the screen 103, and when the operation to press the display switching button 66 is input, the screen is switched to the screen 101A.

[0099] Screen 103 displays a work progress status 67 for work site 1 on the same map of work site 1 as screen 101A. The work progress status 67 can be generated based on work progress data for each area at work site 1. The work progress data is managed, for example, in cell units obtained by dividing the map of work site 1 into a grid, and the work progress status 67 can be displayed by changing the display format (for example, hatching pattern or color) of the cells according to the work progress in each cell.

[0100] (Effects obtained in embodiment 3) When the photographing position distribution data 62 and the work progress status 67 are configured to be viewable on the same map as in embodiment 3, it is possible to select the recognition model to be updated while comparing the work progress and the photographing positions.

[0101] It is preferable that the date of the work progress status 67 displayed on the screen 103 be configured to allow the user to select a desired date. For example, it is preferable to be able to select whether to match or approach the image acquisition date 64 of the shooting position distribution data displayed on the screen 101A, or to match or approach the current date. That is, the work progress status 67 displayed on the screen may be generated from work progress data with a timestamp that is the same date as or close to the image acquisition date 64, or from work progress data with a timestamp that is the same date as or close to the current date. This allows the user to check the changes in the topography of the work site 1 between when the learning video used to create the recognition model was acquired and the current time. This provides useful information for determining whether to use the generated recognition model or acquire new images and create a new recognition model. Utilizing this information allows the user to appropriately select and update the recognition model, thereby maintaining the detection accuracy of the recognition model.

[0102] The display destination of the photographing position distribution data 62 and the work progress status 67 is not limited to the monitor 21 of the control device 22, but may also be the monitor 21b of the learning device 23 or a monitor of another device including a mobile terminal.

[0103] Furthermore, in the above example, the photographing position distribution data 62 and the work progress status 67 are displayed in an alternating manner, but the two may be overlaid in a distinguishable manner on the map of the work site 1. When overlaid, it is preferable to make it easy to understand both by, for example, displaying the number of captured learning images numerically and displaying the work progress status as a heat map.

[0104] In addition to or instead of the work progress status 67, the cells in which work will be performed on a specified date (for example, the day of work) may be displayed in an identifiable manner.

[0105] Furthermore, the work progress data may be input via a network or the like from an external terminal such as a server (work progress management server) that manages the work progress at the work site 1, or may be transferred from removable media such as a flash memory, or may be stored in the control device 22 or the learning device 23. Next, an example configuration of a work progress management server will be described.

[0106] <Work progress management server 29> FIG. 19 is a schematic configuration diagram of a work progress management server 29 and its peripheral devices according to the third embodiment.

[0107] The work progress management server (fourth control device) 29 is connected to an input device 81 including an external input device such as a USB, and a communication device 84 that communicates with external terminals including the control device 22, the learning device 23, etc. The server 29 includes an input / output unit 71, a processing device (processor) 72, and a storage device (e.g., memory) 73. By executing a program stored in the storage device 73, the processing device 72 can function as a work site information collection unit 72a that collects information such as working drawings of the work site 1 and topographical information of the work site from the input device 81 and the communication device 84 via the input / output unit 71, a grid division unit 72b that divides the work site 1 into a plurality of cells using a grid, a grid topographical change calculation unit 72c that calculates the amount of change in topography of each cell using the collected topographical information and information about each cell, and a transmission information generation unit 72d that transmits the calculated information on the amount of topographical change for each cell to an external terminal (e.g., the control device 22 or the learning device 23).

[0108] The server 29 can manage the work site 1 by cells, and by managing the height of each cell, it can determine changes in the topography of the work site. For example, it can search for cells whose height has changed by 1 meter or more from a certain point in time. The processing device 72 stores such grid changes in the storage device 73, and in response to a request from the learning device 23, for example, the processing device 72 generates transmission information in the transmission information generation unit 92d and transmits it via the communication device 84.

[0109] The configuration of the learning device 23 may be the same as that shown in Fig. 3 of embodiment 1. The learning device 23 is capable of displaying information on the amount of change in topography of the work site 1 received from the work progress management server 29 on a map of the work site 1.

[0110] (Fourth embodiment) Hereinafter, a fourth embodiment of the present invention will be described with reference to Fig. 20. The hardware configuration of the fourth embodiment is the same as that of the first embodiment, and the main feature is the processing executed by the image data receiving unit 35a (processing device 35) of the learning device 23.

[0111] 20 is a flowchart showing the processing of the learning device 23 (image data receiving unit 35a). The image data receiving unit 35a starts the processing of FIG. 20 every time it receives a learning image and shooting position information from an external terminal (e.g., the control device 22).

[0112] In step S901, the image data receiving unit 35a (processing device 35) determines whether the received learning image satisfies specified conditions. Examples of the specified conditions include an image taken at a specified work site, a learning image associated with shooting location information, an image taken with a camera of specified specifications, an image taken during a specified period, an image taken under specified weather conditions, an image taken at a specified time of day, etc. Any combination of these conditions may be used as the specified conditions.

[0113] If it is determined in step S901 that the received learning image satisfies the specified conditions, the process proceeds to step S903; if it is determined that the received learning image does not satisfy the specified conditions, the process proceeds to step S902.

[0114] In step S902, the image data receiving unit 35a stores the learning image and the shooting position information in a second database (hereinafter referred to as the second DB) configured in the storage device 36 in the learning device 23.

[0115] In step S903, the image data receiving unit 35a determines whether or not the capacity of the first database (hereinafter referred to as the first DB) configured in the storage device 36 will exceed its upper limit if the received learning images are stored in the first DB. In other words, it determines whether or not the first DB has enough free space to store the learning images.

[0116] If it is determined that the capacity of the first DB will not exceed the upper limit, the process proceeds to step S904, where the learning images and the shooting location information are stored in the first DB. On the other hand, if it is determined that the capacity of the first DB will exceed the upper limit, the process proceeds to step S905, where the learning images and the shooting location information are stored in the first DB so that the data in the first DB is overwritten in order, starting with the oldest image acquisition date. This results in the first DB storing learning images that meet the specified conditions and have the most recent image acquisition date.

[0117] The flowchart may be configured so that if step S901 returns YES, step S903 is omitted and the process proceeds to step S904. Also, before executing the process of step S902, the storage format may be changed according to the free space in the second DB, as in steps S903-905.

[0118] (Effects obtained in embodiment 4) When the learning device 23 is configured as in this embodiment, learning images that meet the specified conditions are appropriately stored and updated in the first DB, so that when a recognition model is generated based on the learning images stored in the first DB, a recognition model that meets the specified conditions can be generated, and as a result, a decrease in recognition accuracy can be suppressed.

[0119] (Other) The present invention is not limited to the above-described embodiments, and includes various modifications within the scope of the gist thereof. For example, the present invention is not limited to those having all of the configurations described in the above-described embodiments, and includes those in which some of the configurations are omitted. Furthermore, it is possible to add or replace some of the configurations of one embodiment with the configurations of another embodiment.

[0120] Furthermore, the components of the control devices 22, 23, 122, and 29, as well as their functions and execution processes, may be partially or entirely implemented by hardware (e.g., by designing logic for executing each function using an integrated circuit). The components of the control devices 22, 23, 122, and 29 may be implemented as programs (software) that are read and executed by a processing device (e.g., a CPU) to implement the functions of the control devices 22, 23, 122, and 29. Information related to the programs may be stored in, for example, semiconductor memories (flash memories, SSDs, etc.), magnetic storage devices (hard disk drives, etc.), and recording media (magnetic disks, optical disks, etc.).

[0121] In addition, in the above description of each embodiment, the control lines and information lines are those that are considered necessary for the description of the embodiment, but they do not necessarily represent all the control lines and information lines related to the product. In reality, it can be considered that almost all components are interconnected.

[0122] 1...work site, 2...work machine (hydraulic excavator), 4...transport machine (dump truck), 5...work object, 6...image acquisition device, 19...camera, 20...communication device, 20b...communication device, 21...monitor (display device), 21b...monitor (display device), 22...control device (first control device), 23...learning device (second control device), 24...network, 25...input device, 25b...input device, 26...positioning device, 27...input device, 29...control device, 29...work progress management server (fourth control device), 31...input / output unit (input / output interface), 32...recognition model update unit, 32...processing device, 32a...image data storage unit, 32b...object recognition unit, 32c...recognition model update unit, 32d...position calculation unit, 33...storage device, 3 4...input / output unit (input / output interface), 35...processing device, 35a...image data receiving unit, 35c...position distribution data generating unit, 35d...recognition model generating unit, 36...storage device, 62...shooting position distribution data, 64...image acquisition date, 64...shooting time information (image acquisition date), 65...update button, 66...display switching button, 67...work progress status, 71...input / output unit, 72...processing device (processor), 72a...work site information collecting unit, 72b...grid dividing unit, 72c...grid topography change amount calculating unit, 72d...transmission information generating unit, 73...storage device, 101...display screen, 101A...display screen, 102...display screen, 103...display screen, 105...sensor box, 106...stand, 122...control device (third control device)

Claims

1. An object recognition system comprising: a camera that is installed at a work site and takes pictures of the work site; a first control device that inputs images taken by the camera and recognizes objects to be recognized in the images using an object recognition model trained by machine learning; and a second control device that stores training images taken by the camera at the work site and generates the object recognition model based on the training images, wherein the first control device uses the object recognition model transmitted from the second control device as a new object recognition model; the second control device further stores shooting position information of the training images, and generates shooting position distribution data indicating the distribution of shooting positions of the training images used to generate the object recognition model based on the shooting position information of the training images used to generate the object recognition model, and displays the data on a monitor.

2. An object recognition system according to claim 1, wherein a map of the work site is displayed on the monitor, and the photographing position distribution data is displayed on the map of the work site.

3. An object recognition system according to claim 1, wherein the monitor displays a map of the work site divided into a plurality of areas, and the amount of learning images taken in each area on the map among the learning images used to generate the object recognition model is displayed on the map.

4. An object recognition system according to claim 3, characterized in that the work progress status of the work site is further displayed on the map based on the work progress data of each area in the work site.

5. An object recognition system according to claim 1, wherein the camera, the first control device, and a positioning device that receives signals from a plurality of positioning satellites and calculates the position are mounted on a work machine operating at the work site, the learning image is taken by the camera, the shooting position information is calculated by the positioning device, the first control device and the second control device are capable of communicating with each other, and the first control device transmits the learning image and the shooting position information to the second control device.

6. An object recognition system according to claim 1, comprising: an installed camera that is installed at a desired position on the work site and takes the learning images; a positioning device that receives signals from a plurality of positioning satellites and calculates the shooting position information; and a third control device that is connected to the installed camera and the positioning device and is capable of communicating with the second control device, wherein the third control device transmits the learning images taken by the installed camera and the shooting position information calculated by the positioning device to the second control device.

7. An object recognition system according to claim 6, wherein the difference between the parameters relating to the camera specifications, including sensor size, focal length, installation height, and installation pitch angle, between the installed camera and the camera is within a predetermined range.

8. An object recognition system according to claim 1, wherein the second control device transmits the object recognition model to the first control device based on a transmission command input after the photographing position distribution data is displayed on the monitor.

9. An object recognition system according to claim 2, wherein the second control device generates the object recognition model based on a learning image of a specified work site among the learning images transmitted from the first control device.

10. An object recognition system according to claim 1, wherein the second control device further stores photographic time information of the learning images, and further displays on the monitor photographic time information of the learning images that were used to generate the object recognition model.

Images