Object recognition system

Abstract

An object recognition system has a camera that captures a work site, a control device that inputs an image acquired by the camera, recognizes a recognition object on the image using an object recognition model that has been subjected to machine learning, and a learning device that stores a learning image captured within the work site, generates the object recognition model based on the learning image. The control device utilizes the object recognition model transmitted from the learning device as a new object recognition model. The learning device also stores capture position information of the learning image, generates and displays on a monitor capture position distribution data representing a distribution of capture positions of the learning image utilized in the generation of the object recognition model based on the capture position information of the learning image utilized in the generation of the object recognition model.

Description

Technical Field

[0001] This invention relates to object recognition systems, for example, suitable for field management systems and operating machinery. Background Technology

[0002] In recent years, due to improvements in computer performance, image recognition technology using machine learning, such as deep neural networks, has seen significant development. For work sites where excavators and other machinery operate, as well as transport vehicles like dump trucks, machine learning techniques for recognizing objects from images are being implemented through onboard devices on the machinery and on-site sensors. One type of this technology involves capturing images of objects (e.g., people) at the work site to create teacher data, and then using machine learning to generate a recognition model for those objects (e.g., a model composed of a set of parameters from image recognition processing) based on this teacher data. This recognition model is then used to identify the objects from the images. Furthermore, the recognition model can be updated after the on-site sensors and onboard devices on the machinery are manufactured, thus raising hopes for maintaining and improving the accuracy of object recognition (detection accuracy).

[0003] Regarding the updating of the recognition model, Patent Document 1 discloses a learning system comprising: multiple vehicle-mounted devices installed in vehicles traveling on roads and performing image recognition processing; and a server that updates the parameters (recognition model) of the image recognition processing based on image data (learning images) used in the image recognition processing within the vehicle-mounted devices. The server has a main learning server that performs learning processing based on learning data from all driving areas, and multiple sub-learning servers that perform learning processing on learning data categorized according to groups of driving areas. The main learning server categorizes the learning data into groups corresponding to driving areas and sends the categorized learning data to the sub-learning servers corresponding to each group. Each sub-learning server uses the learning result from the main learning server as an initial value, performs learning processing (refinement) based on the received learning data, generates parameters for image recognition processing within the driving area corresponding to its own device, and sends these parameters to the main learning server. The main learning server uses the parameters corresponding to each driving area as update data and sends the update data to each vehicle-mounted device. Each vehicle-mounted device updates its image recognition processing parameters based on the received update data. Therefore, Patent Document 1 describes refining the parameters to adapt to the driving area of ​​the vehicle-mounted device, thereby optimizing the image recognition processing in each driving area of ​​the vehicle. Existing technical documents Patent documents

[0004] Patent Document 1: Japanese Patent Application Publication No. 2015-135552 Summary of the Invention

[0005] Typically, the progress of work at a work site changes locally. Furthermore, compared to the motor vehicle roads exemplified in Patent Document 1, the work site is characterized by a particularly rapid rate of change in terrain and landscape. For example, in cases of work involving significant terrain changes such as excavation, the landscape could drastically change within a few days. Therefore, in order to update the recognition model after the start of work to maintain recognition accuracy, it is crucial to decide whether to utilize the new recognition model based on the location of the images taken. That is, even if it is a new recognition model, if it is generated based on images taken in areas of the work site with little or no change, then even if the model is changed to that, the maintenance and improvement of recognition accuracy cannot be anticipated.

[0006] Although the system in Patent Document 1 changes the recognition model (parameters) for each driving area, it cannot determine where within the driving area the image was taken to generate each recognition model. Therefore, even if the technology in Patent Document 1 is applied to object recognition at a work site, the detection accuracy is likely to decrease significantly.

[0007] The purpose of this invention is to provide an object recognition system that can determine the object recognition model used in image recognition based on the location of the image taken at the work site.

[0008] This application includes multiple solutions to the aforementioned problems. For example, it provides an object recognition system comprising: a camera positioned at a work site to capture images of the work site; a first control device that inputs images acquired by the camera and uses a machine learning-based object recognition model to identify objects in the images; and a second control device that stores learning images captured by the camera within the work site and generates the object recognition model based on these learning images. The first control device utilizes the object recognition model sent from the second control device as a new object recognition model. In the object recognition system, the second control device also stores the shooting location information of the learning images. Based on the shooting location information of the learning images used in the generation of the object recognition model, it generates shooting location distribution data and displays it on a monitor. This shooting location distribution data represents the distribution of shooting locations of the learning images used in the generation of the object recognition model. Invention Effects

[0009] According to the present invention, it is possible to determine where the recognition model is generated based on the image taken at the work site, and therefore, it is possible to suppress the decrease in the recognition accuracy of the object being recognized as an accompanying update of the recognition model. Attached Figure Description

[0010] Figure 1 This is a schematic diagram of the object recognition system in the work site according to an embodiment of the present invention. Figure 2 This is a configuration diagram showing an example of the control device 22 and its peripheral equipment. Figure 3 This is a configuration diagram showing an example of the learning device 23 and its peripheral equipment. Figure 4 It is a schematic diagram of the work site 1 where the operating machinery 2, equipped with control device 22, camera 19 and monitor 21, is operating. Figure 5 yes Figure 4 A schematic diagram of the working machinery 2 in the diagram. Figure 6 This is a flowchart illustrating the processing of the object recognition unit 32b (control device 22). Figure 7 This is an example of a situation where an image captured by a camera contains an object (person) to be identified. Figure 8 This is a flowchart of the process by which the control device 22 saves the learning image and shooting location information acquired by the camera 19 through the image data storage unit 32a and sends it to the learning device 23. Figure 9 This is a flowchart illustrating the processing of the learning device 23 (recognition model generation unit 35d). Figure 10 This is an explanatory diagram of deep learning. Figure 11 This is a flowchart illustrating an example of the processing of the location distribution data generation unit 35c. Figure 12 This is a flowchart illustrating an example of the transmission processing of location distribution data and recognition model based on learning device 23. Figure 13 This is an example of a display showing the location distribution data 62a captured to the monitor 21 of the control device 22. Figure 14 This is an example of a display showing the location distribution data 62b of the camera to the monitor 21 of the control device 22. Figure 15 This is a schematic diagram of the object recognition system in the work site according to Embodiment 2. Figure 16 This is a schematic diagram of an example of an image acquisition device 6. Figure 17 This is a configuration diagram showing an example of the control device 122 and its peripheral equipment. Figure 18This is another example of a display showing the location distribution data 62a of the camera to the monitor 21 of the control device 22. Figure 19 This is a schematic diagram of the job progress management server 29 and its peripheral equipment in Implementation Method 3. Figure 20 This is a flowchart illustrating the processing of the learning device 23 (image data receiving unit 35a). Detailed Implementation

[0011] Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings.

[0012] <Object Recognition System> Figure 1 This is a schematic diagram of the object recognition system in the work site according to an embodiment of the present invention. Figure 1 The object recognition system has a camera 19 for capturing images of the work site, a control device (first control device) 22, a learning device (second control device) 23, and a monitor 21.

[0013] The control device 22 (first control device) is, for example, a computer having a processing unit (processor such as a CPU), a storage unit (e.g., a memory), and an input / output interface (input / output unit). The control device 22 receives an image acquired by the camera 19 and uses a machine learning-based object recognition model to identify objects (e.g., people) in the image. The control device 22 may be, for example, a controller (vehicle-mounted device) mounted on and used to control a hydraulic excavator or other work machinery at the work site, and a computer connected to a camera (field sensor) installed at the work site.

[0014] The learning device 23 (second control device) is, for example, a computer having a processing unit (processor such as a CPU), a storage unit (e.g., a memory), and an input / output interface (input / output unit). The learning device 23 stores multiple images (learning images) captured at the work site and generates an object recognition model based on these learning images. The generated object recognition model can be sent to the control device 22 for use in the recognition of objects by the control device 22.

[0015] The learning device 23 stores the location information of each learning image being captured, i.e., the capture location information. Based on the capture location information of the learning images used in the generation of the object recognition model, the learning device 23 generates capture location distribution data and displays it on the monitor 21. This capture location distribution data represents the distribution of capture locations of the learning images used in the generation of the object recognition model (details will be described later, but for example, in...). Figure 1(The image shown is labeled with reference numeral 62 in the attached drawing). The image location distribution data can be generated for each object recognition model. Preferably, before the control device 22 starts using a new object recognition model, the image location distribution data of the object recognition model is displayed on the monitor 21, and then it is confirmed that the learning image captured in the area where object recognition will be performed is included in the learning image used in the generation of the object recognition model.

[0016] Monitor 21 may be connected to control device 22, learning device 23, or a terminal different from control device 22 and learning device 23. However, in this embodiment, an example is given where monitor 21 is connected to control device 22 and another monitor 21b, different from monitor 21, is connected to control device 23.

[0017] In addition, the learning device 23 may be a server, for example, installed at the work site or in a monitoring center located in a remote location.

[0018] The control device 22 and the learning device 23 are configured to communicate with each other. For example, both the control device 22 and the learning device 23 have communication devices such as Wi-Fi antennas, and are connected wirelessly via a network 24. The network 24 is, for example, the Internet or a local network. Furthermore, the communication between the network 24 and the control device 22 and the learning device 23 is not limited to wireless communication, but can also be wired communication.

[0019] (Implementation Method 1) <Control Device 22 (First Control Device)> Figure 2 This is a configuration diagram showing an example of the control device 22 and its peripheral equipment. The control device 22 includes a processing unit 32, a storage unit 33, and an input / output unit (input / output interface) 31. A camera (shooting device) 19 for capturing images of the work site, a communication device 20, a monitor (display device) 21, a positioning device 26, and an input device 27 are connected to the control device 22.

[0020] The input / output unit 31 is an interface for connecting peripheral devices such as camera 19, communication device 20, monitor 21, and input device 27 to the control device 22.

[0021] The processing device 32 can be implemented, for example, by a combination of a CPU (Central Processing Unit) and a GPU (Graphics Processing Unit).

[0022] The storage device 33 can be configured with memory such as RAM (Random Access Memory) and storage devices such as HDD (Hard Disk Drive) and SSD (Solid State Drive). The storage device 33 is configured to connect to the processing device 32 and perform write and read operations based on the processing device 32. In addition, source files and programs used by the processing device 32 to perform various operations are also stored in the storage device 33.

[0023] The control device 22 can function as an image data storage unit 32a, an object recognition unit 32b, a recognition model update unit 32c, and a position calculation unit 32d by executing the program stored in the storage device 33 by the processing device 32. Figure 2 For convenience, the parts 32a-32d are shown inside the processing device 32. The processing that can be performed by each part 32a-32d will then be described.

[0024] <Camera 19> The camera 19 is preferably installed at the work site and fixed at a predetermined height in a predetermined posture. The posture of the camera 19 is preferably defined by a pitch angle (forward / backward rotation angle) and a yaw angle (left / right rotation angle). The fixed object of the camera can be a piece of machinery operating as a mobile body within the work site (e.g., a hydraulic excavator and a dump truck), or a support frame installed within the work site (where the support frame is portable). The camera 19 can, for example, be a digital camera with a CMOS (Complementary Metal Oxide Semiconductor) image sensor.

[0025] <Communication device 20> The communication device 20 is hardware used to communicate with other devices, including the learning device 23, via the network 24. The communication device 20 is also referred to as a network device, network controller, network interface card (NIC), communication module, etc. The communication device 20 may have a wired connection connector or a wireless communication interface. For example, the communication device 20 can be equipped with an antenna capable of LTE-Advanced and a WiFi antenna.

[0026] <Monitor 21> The monitor 21 displays position distribution data, which represents the distribution of the position of the object being identified on multiple images (e.g., images captured by camera 19) captured by the object recognition system.

[0027] The monitor 21 can also be a device that has input functionality in addition to display functionality. That is, for example, an electrostatic capacitive touch-sensitive display that allows input operations by the operator's fingers. However, it is not limited to a touch panel display; it can also be a combination of a display and input devices such as joysticks and buttons.

[0028] <Positioning Device 26> The positioning device 26 receives signals from multiple positioning satellites to calculate the position and sends the calculation result (position data) to the control device 22. The positioning device 26 can be, for example, an antenna and receiver compatible with GNSS (Global Navigation Satellite System). The receiver can calculate the antenna's position based on the positioning signals received by the antenna. For example, simply by setting the antenna on the camera 19, the position of the camera 19 can be calculated. When the camera 19 and the control device 22 are mounted on a moving body such as a work machine, and a desired position on the moving body, including the position of these devices, is to be accurately calculated, it is preferable to set two antennas on the moving body and have the receiver calculate the positions of these two antennas. Furthermore, when the camera 19 and the control device are mounted on a stationary body and the position of the stationary body is known, the positioning device 26 can be omitted from the system.

[0029] <Input Device 27> The input device 27 connected to the control device 22 is not limited in type, but may be, for example, a keypad or a touch panel. It may also combine a joystick and buttons, or function as a mouse and keyboard. Input signals are sent to the control device 22 based on input operations performed on the input device 27.

[0030] <Learning Device 23 (Second Control Device)> Figure 3 This is a configuration diagram showing an example of a learning device 23 and its peripheral devices. The learning device 23 includes a processing unit 35, a storage unit 36, and an input / output unit (input / output interface) 34. A communication device 20b, a monitor (display device) 21b, and an input device 25b are connected to the learning device 23.

[0031] The input / output unit 34 is an interface for connecting peripheral devices such as the communication device 20b, the monitor (display device) 21b, and the input device 25b to the learning device 23.

[0032] The processing device 35 can be implemented, for example, through a combination of a CPU and a GPU.

[0033] The storage device 36 can be configured using memory such as RAM and storage devices such as HDD and SSD. The storage device 36 is configured to connect to the processing device 35 and perform write and read operations based on the processing device 35. Furthermore, source files and programs used by the processing device 35 to perform various calculations are also stored in the storage device 36.

[0034] The learning device 23 functions as an image data receiving unit 35a, a position distribution data generating unit 35c, and a recognition model generating unit 35d by executing a program stored in the storage device 36 through the processing unit 35. Figure 3 For convenience, the parts 35a-35d are shown inside the processing device 35. The processing that can be performed by each part 35a-35d will then be described.

[0035] The location of the learning device 23 is not limited as long as it can be connected to the network 24. It can be set up outside or inside the work site.

[0036] <Communication Device 20b> Communication device 20b is hardware used to communicate with other devices, including control device 22, via network 24. Communication device 20b is referred to, for example, as a network device, network controller, network interface card (NIC), communication module, etc. Communication device 20b may have a wired connection connector or a wireless communication interface. For example, as communication device 20b, devices with antennas capable of LTE-Advanced and WiFi antennas can be used.

[0037] <Monitor 21b> The monitor 21b receives a signal from the learning device 23 and displays an image. The monitor 21b is, for example, a liquid crystal display (LCD). However, similar to the monitor 21 of the control device 22, the display can also be activated by touch using capacitive touch, for example.

[0038] <Input Device 25b> The input device 25b connected to the learning device 23 is not limited to any particular type, but may include, for example, a mouse and a keyboard. A joystick and buttons may also be used in combination. Input signals are sent to the learning device 23 based on input operations performed on the input device 25b.

[0039] Furthermore, when the learning device 23 is remotely connected from other devices connected to the same network 24 via protocols such as SSH (Secure Shell) and RDP (Remote Desktop Protocol), the monitor 21b and input device 25b can utilize the display and input devices of those other devices.

[0040] <Work Site 1> Figure 4 This is a schematic diagram of the work site 1 where the excavator (hydraulic excavator) 2, equipped with control device 22, camera 19, and monitor 21, operates. At work site 1, there are the excavator 2, the operator 3, and the transport equipment (dump truck) 4. The excavator 2 performs leveling, excavation, and loading operations (such as loading work objects 5) onto the transport equipment 4 within a predetermined area. The transport equipment 4, such as a dump truck, transports work objects 5 while moving in and out of work site 1. Meanwhile, the operator 3 performs various tasks around the excavator 2, including guiding other machinery, cleaning up work objects 5 that have fallen from the excavator 2, and transporting various small tools.

[0041] <Operating Machinery 2> Figure 5 yes Figure 4 A schematic diagram of the working machine (hydraulic excavator) 2 is shown. The working machine 2 includes: a lower traveling body 11; an upper rotating body 12 rotatably mounted on the lower traveling body; a front working device 13; the front working device 13 consisting of a boom 15, a stick 16, and a bucket 17; and an operator's cab 18 for the operator to enter and operate the working machine 2. The front working device 13 includes: a boom 15 rotatably mounted in front of the upper rotating body 12; a stick 16 rotatably mounted at the front end of the boom 15; and a bucket (working tool) 17 rotatably mounted at the front end of the stick 16. Furthermore, the bucket 17 is an example of a working tool (accessory) of the front working device 13, and can be replaced with a grab bucket, a shovel, etc.

[0042] The upper rotating body 12 is equipped with a camera 19 capable of acquiring images of the surroundings of the operating machine 2, a communication device 20, a monitor (display device) 21 located in the operator's cab 18, a positioning device 26, and a control device 22. In this embodiment, an example is given where the monitor 21 is installed in the operator's cab 18, but the location of the monitor 21 is not limited to the operator's cab 18. In addition, the monitor 21 can also be connected to the control device 22 wirelessly.

[0043] Camera 19 is mounted on the upper rotating body 12, and is configured, for example, as follows: Figure 5 The image shows the rear of the operating machine 2. The camera 19 does not need to be located at the rear of the upper rotating body 12, and there can be multiple cameras.

[0044] <Object recognition processing of control device 22> Next, use Figure 6 and Figure 7 This explains that the control device 22 performs the process of identifying the object from the image of the camera 19 through the object recognition unit 32b. Figure 6This is a flowchart illustrating the processing of the object recognition unit 32b (control device 22).

[0045] In step S201, the object recognition unit 32b (control device 22) uses the camera 19 to acquire an image and causes the processing to proceed to step S202.

[0046] In step S202, the object recognition unit 32b uses the image acquired in step S202 and the recognition model (object recognition model) stored in the storage device 33 to perform recognition processing to infer whether a recognition object exists in the acquired image, and then proceeds the processing to S203. Figure 7 As shown in (a), when an image including the object to be identified, i.e., the operator (person) 51, is acquired, recognition processing is performed using a recognition model stored in storage device 33. The recognition model is defined, for example, by a neural network and parameters. The recognition result includes, for example, the presence or absence of the object to be identified, the coordinates of the quadrilateral (bounding box) surrounding the object to be identified (that is, the position of the object to be identified in the image), the type of the object to be identified (e.g., a person, a transport machine, a work machine, etc.), and the reliability of the recognition. The coordinates of the bounding box can be obtained using the coordinates of any point related to the quadrilateral, such as the two vertices at the ends of the diagonal of the quadrilateral containing the object, the midpoint of the base of the quadrilateral, a vertex of the quadrilateral, or the center of the quadrilateral.

[0047] In step S203, the object recognition unit 32b outputs the recognition result obtained in step S202. For example, such as Figure 7 As shown in (b), the bounding box 52 and the object type 53 can be overlaid on the acquired image of the monitor 21 and output.

[0048] <Data storage and processing of control device 22> Next, use Figure 8 The flowchart illustrates the process by which the control device 22 saves the image data (learning image) captured by the camera 19 and the location data (capture location information) of the captured image data to the storage device 33 via the image data storage unit 32a and sends it to the learning device 23.

[0049] In step S101, the image data storage unit 32a (control device 22) determines whether an image save start operation (e.g., button operation) has been performed via the input device 27. If an operator has performed the image save start operation, an image save start signal is input to the image data storage unit 32a. After determining that an image save start operation has been performed, the image data storage unit 32a proceeds to step S102. If no image save start operation has been performed, step S101 is repeated.

[0050] In step S102, the image data storage unit 32a acquires the image captured by the camera 19, and the position calculation unit 32d calculates the position data (capture position information) of the image captured based on the position data input from the positioning device 26. The learning image is acquired from the camera 19, and the capture position information of the learning image is calculated. Then, the process proceeds to step S103.

[0051] return Figure 8 In step S103, the image data storage unit 32a associates the image (learning image) acquired in step S102 with the location data (shooting location information) calculated in the same step S102 and stores it in the storage device 33. Thus, the shooting location information is associated with each learning image. Furthermore, the image storage format is not limited; it can be a still image or a moving image. During storage, compression formats such as AVC (Advanced Video Coding) and JPEG (Joint Photographic Experts Group) can be performed, and the resolution can be specified. Then, processing proceeds to step S104.

[0052] In step S104, the image data storage unit 32a determines whether an image saving end operation (e.g., a button operation) has been input from the input device 27. If an image saving end operation has been input to the image data storage unit 32a (control device 22), the process proceeds to step S105. If no image saving end operation has been input, the process returns to step S102, and the process of acquiring and saving the learning image and shooting location information is repeated.

[0053] In step S105, the image data storage unit 32a uses the communication device 20 to send the learning image and shooting location information stored in the storage device 33 during the period from when the image saving start operation was input to when the image saving end operation was input to the learning device 23. Then, the process returns to step S101.

[0054] In addition, the images of the camera 19 mounted on the machine 2 are used as learning images. However, any images captured by a camera with the same sensor size, focal distance, setting height, and setting angle (tilt angle) as the camera 19 can also be used as learning images for the object recognition model of the machine 2.

[0055] Furthermore, the image save start operation and image save end operation are input via the input device 27 of the control device 22, but they can also be input via the input device 25b of the learning device 23 or other input devices. Alternatively, the signals input from the control device 22 and the learning device 23 when predetermined conditions are met can also be considered as the image save start operation and image save end operation.

[0056] <Data Storage and Processing of Learning Device 23> After receiving the learning image and shooting location information from the control device 22, the image data receiving unit 35a of the learning device 23 saves the received learning image and shooting location information in the storage device 36.

[0057] <Learning Processing of Learning Device 23 (Recognition Model Generation Processing)> Next, use Figure 9 and Figure 10 This explains the processing performed by the recognition model generation unit 35d based on the learning device 23.

[0058] Figure 9 This is a flowchart illustrating the processing of the learning device 23 (recognition model generation unit 35d).

[0059] In step S801, the recognition model generation unit 35d performs a determination process to determine whether a learning start operation has been input from the input device 25b. If the input of a learning start operation is received, the process proceeds to step S802; otherwise, the process returns to step S801.

[0060] In step S802, the recognition model generation unit 35d uses the learning images stored in the storage device 36 to perform machine learning and generate a new recognition model. The method of using the learning images is not limited, but for example, it could involve manually assigning bounding boxes to the objects in the learning images, manually assigning category information to the objects to generate teacher data, and then using this teacher data for learning. The teacher data is data obtained by assigning corrective information (such as the location and category information of the objects) to the learning images.

[0061] In the generation of this recognition model, deep learning was used, for example. When using deep learning, such as... Figure 10 As shown, for a predefined deep neural network 92, the parameters are adjusted to minimize the difference between the inference result of the input image 91 and the correct solution information 93 (e.g., the deviation of the bounding box coordinates), and learning (generating a new recognition model) is performed. The recognition model can, for example, be constructed and its parameters defined by the deep neural network used. Figure 10As shown, when the learned recognition model is used, the bounding box 93 and category information of the object to be recognized can be inferred from the input image 91, and the location and type of the object to be recognized can be inferred. After performing machine learning and generating a new recognition model, the process proceeds to step S803.

[0062] In step S803, the recognition model generation unit 35d sends a recognition model generation signal, indicating that a new recognition model has been generated, to the location distribution data generation unit 35c. At this time, it is also possible to further send the signal to the control device 22 using the communication device 20b. After the recognition model generation signal is sent, the process returns to step S801.

[0063] In addition to the above, the timing for starting learning using teacher data can also be a timing that occurs whenever the image data receiving unit 35a receives a new learning image. However, it is not limited to this timing; for example, the update status of new learning images can be checked at fixed intervals, and learning can begin when new learning images are available.

[0064] Furthermore, after the new recognition model is generated, the recognition information of the new recognition model can be assigned to each learning image used in the generation process. This is to facilitate the generation and processing of the location distribution data 62 described later.

[0065] <Data generation and processing of the location distribution of the identified objects based on learning device 23> The learning device 23 (position distribution data generation unit 35c) generates shooting position distribution data based on the shooting position information of the learning images used in the generation of the new recognition model stored in the storage device 36. This shooting position distribution data represents the distribution of shooting positions of the learning images used in the generation of the recognition model.

[0066] The image location distribution data refers to the data collected and statistically analyzed by the location distribution data generation unit 35c, which collects and displays the distribution of images used in the generation of a specified recognition model, including the images taken at various locations. This distribution is presented in the form of text (including numbers), charts, graphs, etc. The image location distribution data exists for each recognition model and can be displayed in various formats. For example, if the work site 1 is divided into multiple areas, the location distribution data is obtained by determining which area among these areas the learning images used in the generation of the specified recognition model were taken based on the image location information. For each area, the quantity (e.g., number of images and their proportion) of learning images containing image location information is calculated. This count (numerical value) is then correlated with the area to obtain the location distribution data. Therefore, a frequency distribution map showing the distribution of image locations used by the recognition model on a map of the work site 1 is also included in the image location distribution data. An example of this frequency distribution map will be described later. Figure 13 Location distribution data 62b.

[0067] Next, use Figure 11 This illustrates an example of the processing performed by the location distribution data generation unit 35c. Figure 11 This is a flowchart illustrating the processing of the location distribution data generation unit 35c.

[0068] In step S301, the location distribution data generation unit 35c (learning device 23) determines whether it has received a recognition model generation signal from the recognition model generation unit 35d. If it has received a recognition model generation signal, the process proceeds to step S302.

[0069] In step S302, the location distribution data generation unit 35c generates location distribution data 62 (see below) of the recognition model based on the shooting location information of the learning image used in the generation of a new recognition model triggered by the transmission of the recognition model generation signal. Figure 13 , 14 At this time, other information displayed on the monitor 21 (e.g., the shooting time information of the learning images used in the generation of the recognition model) can also be calculated together with the generation of the shooting location distribution data 62.

[0070] In step 303, the position distribution data generation unit 35c saves the shooting position distribution data 62 generated in step S302 into the storage device 36. Furthermore, if other information to be displayed on the monitor 21 is generated together with the shooting position distribution data 62 in step 302, it is preferable to also save this other information into the storage device 36.

[0071] <Transmission and Processing of Location Distribution Data and Recognition Model Based on Learning Device 23> The learning device 23 (position distribution data generation unit 35c) sends the position distribution data 62 of the recognition model specified by the sending command based on the position distribution data sending command. Furthermore, the learning device 23 (recognition model generation unit 35d) processes the sending of the recognition model specified by the subsequent input recognition model update command.

[0072] Figure 12 This is a flowchart illustrating an example of the transmission and processing of location distribution data and recognition models based on the learning device 23. The learning device 23 executes according to a predetermined control cycle. Figure 12 The processing.

[0073] If processing begins, in step S401, the learning device 23 determines whether a location distribution data transmission command has been received. If it is determined that a location distribution data transmission command has been received, the process proceeds to step S402. The location distribution data transmission command is an instruction output by the control device 22 or the learning device 23 to instruct the learning device 23 to transmit location distribution data of a specified recognition model, for example, by operating the respective input devices 27 and 25b. This operation includes a selection operation, which involves selecting a recognition model from the recognition models stored within the learning device 23.

[0074] In step S402, the learning device 23 selects the location distribution data 62 of the recognition model specified by the location distribution data transmission command from the storage device 36, and transmits it relative to the transmission source of the location distribution data transmission command (control device 22 or learning device 23). The location distribution data 62 of the recognition model is then displayed on the monitor of the transmission source of the location distribution data transmission command.

[0075] <Example of displaying shooting location distribution data 62> exist Figure 13 , 14 The image shows an example of the shooting location distribution data 62 being displayed on the monitor 21 of the control device 22. Figure 13 The display screen 101 shows the recognition information 61 of the recognition model, the shooting location distribution data 62a of the recognition model, an example of the shooting location distribution data 63a, the shooting time information (image acquisition date) of the learning images used in the generation of the recognition model 64, and the update button 65 of the recognition model.

[0076] Identification information 61 shows the identification information of the identification model specified by the location distribution data transmission instruction.

[0077] The image location distribution data 62a shows the distribution of locations where the learning images used in generating the recognition model, which is endowed with recognition information 61, were captured on a map of work site 1. In the example image, the map of work site 1 (which could also be a construction drawing, etc.) is divided into multiple cells (regions) using a grid, and the display of the cells is changed (e.g., shading patterns and colors) according to the amount of learning images captured within each cell. The display of each cell corresponding to the amount of learning images is shown in Example 63a. Figure 13 In the example, the display (shading pattern) of the cell is changed according to the number of shots in that cell relative to the total number of learning images used in the generation of the recognition model (that is, using a ratio).

[0078] The shooting time information 64 shows the shooting time information of the learning images used in the generation of the recognition model to which recognition information 61 is given. It is generated based on shooting time data associated with the learning images in the storage device 36. In the illustrated example, the shooting day (image acquisition day) is displayed as shooting time information. In addition, if the shooting day involves multiple days, the corresponding period can also be displayed on the screen.

[0079] The update button 65 is operated when the recognition model used by the control device 22 in object recognition is updated to the recognition model of the shooting position distribution data 62 displayed on the screen (the new recognition model). If the update button 65 is pressed, a recognition model update command is output to the learning device 23.

[0080] Furthermore, when the shooting location distribution data 62 is displayed on the monitor 21b of the learning device 23, it is preferable to display a selection request screen for the receiver's control device 22 when the update button 65 is pressed. If the receiver's control device 22 is selected on this screen, an update command for the recognition model is output along with the recognition information of the receiver's control device 22.

[0081] exist Figure 14 The display screen 102 shows the display format and Figure 13 Data 62b shows the distribution of shooting locations. This data, presented in a 3D bar chart, represents the number of images taken for learning purposes within each cell.

[0082] Furthermore, if the learning images stored in the storage device 36 of the learning device include images obtained from other work sites, the shooting location information of the learning images obtained from other work sites will be used as exclusion information, and only the shooting location information related to work site 1 will be displayed.

[0083] return Figure 12The explanation is as follows. In step S403, the learning device 23 determines whether it has received a recognition model update instruction. The recognition model update instruction is indicated by the update button 65 (see reference). Figure 13 , 14 The signal output when the operation is performed. This signal also contains recognition information of the new recognition model, which displays the shooting location distribution data 62. If it is determined in step S403 that a recognition model update instruction has been received, proceed to step S404.

[0084] In step S404, the learning device 23 selects the recognition model (new recognition model) specified by the recognition model update instruction from the storage device 36 and sends it relative to the control device 22 specified by the recognition model update instruction. Thus, the recognition model of the control device 22 specifying the target is updated to the sent model (new recognition model).

[0085] (Effects obtained from Implementation Method 1) If the system of Embodiment 1 is used, the image location distribution data 62 of the specified recognition model is displayed on a monitor for confirmation. This allows it to be determined from which location in the work site 1 the learning image was taken, and the recognition model can be updated accordingly. That is, when the recognition model is generated from a learning image taken from a predetermined area where the work will be carried out, the decrease in object recognition accuracy after updating to the recognition model can be suppressed. In other words, according to this embodiment, it is possible to determine from which location within the work site the recognition model was taken, thus suppressing any decrease in object recognition accuracy that accompanies the use of the recognition model. In other words, according to this embodiment, a recognition model that improves the recognition accuracy of objects within the work site can be selected.

[0086] Furthermore, the above description illustrates a scenario where the camera position distribution data 62 is displayed on the monitor 21 mounted on the work machinery 2, and the operator discusses whether the recognition model needs updating. However, it is also possible to display the camera position distribution data 62 on the monitor 21b of the learning device 23, and have the operator of the learning device 23 (e.g., a service provider) discuss whether the recognition model needs updating. Alternatively, the camera position distribution data 62 can be displayed on a monitor such as a mobile terminal, and an operator other than the operator at the work site (e.g., a site manager) can observe the monitor to discuss whether an update is needed. Moreover, it is also possible that, based on the distribution of camera positions in the camera position distribution data 62, a computer including the control device 22 and the learning device 23 automatically decides whether to update the recognition model (that is, in this case, no judgment is made via the "screen").

[0087] (Implementation Method 2) The following uses Figure 15 -17, Description of Embodiment 2 of the present invention. The main feature of this embodiment is that a learning image is captured by a camera 19 installed at the work site 1, and the learning image and the capture position information are sent from a control device (third control device) 122 connected to the camera 19 to the learning device 23. The learning device 23 has the same configuration as in Embodiment 1, so the description is omitted.

[0088] Figure 15 This diagram illustrates the relationship between the learning device (second control device) 23, the control device (first control device) 22, and the control device (third control device) 122 in this embodiment. The object recognition system of Embodiment 2 is equivalent to Embodiment 1 with the addition of the control device (third control device) 122. A camera 19, supported by a bracket or the like and installed at the work site 1, is connected to the control device 122.

[0089] Figure 16 This is a schematic diagram illustrating an example of an image acquisition device 6 having a camera 19 installed at work site 1 and a control device 122 connected to the camera 19. The illustrated image acquisition device 6 is a field sensor for a portable camera, having a sensor housing 105 and a bracket 106 supporting the sensor housing 105. The camera 19, a communication device 20, a control device 122 connected to the camera 19 and the communication device 20, and a positioning device 26 are connected to the sensor housing 105. The control device 122 may, for example, be capable of wireless communication with a mobile terminal (not shown), controlling the device based on operations input to the mobile terminal. This mobile terminal may also be carried by a worker at work site 1.

[0090] Figure 17 It means Figure 16 A configuration diagram of an example of a control device (third control device) 122 and its peripheral equipment. The control device 122 is configured as follows: from... Figure 2 In the first control device 22 of Embodiment 1 shown, the input device 27 and the monitor 21 are omitted as hardware, and the object recognition unit 32b and the recognition model update unit 32c are omitted as functions (software) executed by the processing unit 32.

[0091] The control device 122 is configured to execute Figure 8 The flowchart shown enables the transmission of learning images and their capture location information to the learning device 23. Furthermore, the image saving start operation in step S101 and the image saving end operation in step S104 can be input at desired timings from the control device 22 and the learning device 23 mounted on the work machinery 2, or from the aforementioned mobile terminal.

[0092] (Effects obtained from Implementation Method 2) As described above, in this embodiment, the device comprises: a camera (setup camera) 19 set at a desired location at the work site 1 to capture images for learning purposes; a positioning device 26 that receives signals from multiple positioning satellites and calculates the capture location information based on the images for learning purposes captured by the camera 19; and a control device (third control device) 122 connected to the setup camera 19 and the positioning device 26 and capable of communicating with the learning device (second control device) 23. The control device (third control device) 122 sends the images for learning purposes captured by the setup camera 19 and the capture location information calculated by the positioning device 26 to the learning device (second control device) 23.

[0093] This constitutes an object recognition system using the camera 19, achieving the same effects as Embodiment 1, which uses a camera 19 mounted on the work machinery 2. In particular, the ability to construct the object recognition system by placing the camera 19 at a desired location within the work site 1, independent of the work machinery 2, is an advantage. For example, the ability to position the camera 19 according to the site conditions (construction progress and whether the work involves manual intervention) is also an advantage.

[0094] Furthermore, preferably, the parameters of the camera (set camera) 19 of the field sensor 6, including sensor size, focal distance, setting height, and setting pitch angle, which are related to the camera specifications, differ from the same parameters of other cameras 19 used at the work site 1 within a specified range, and preferably, are as similar as possible to those other cameras 19 (i.e., the parameters are the same in this case). For example, if the camera is mounted on... Figure 5 The parameters of the camera 19 of the operating machine 2 are set to be the same as the parameters related to camera specifications, so that images captured by one camera 19 can be used as learning images for the other camera 19. Furthermore, the "difference" mentioned above is calculated for each parameter, and the "specified range" can be set for each parameter. Preferably, the "difference" is as small as possible.

[0095] Furthermore, the above describes the case where object recognition based on the control device 122 is not performed. However, it is also possible to have an object recognition unit 32b and a recognition model update unit 32, similar to the control device 22 in Embodiment 1, which can perform object recognition based on the control device 122.

[0096] (Implementation Method 3) The following uses Figure 18 -19, illustrating Embodiment 3 of the present invention. This embodiment is a variation of Embodiment 1, relating to other display formats of the shooting position distribution data 62 on the monitor 21.

[0097] Figure 18 Other examples of displays showing the shooting location distribution data 62 being displayed on the monitor 21 of the control device (first control device) 22 are shown. The same reference numerals are used for the parts that are the same as in the previous figures.

[0098] Figure 18 The text is a jumbled collection of characters and phrases, seemingly from different sources and lacking coherent sentences. A direct translation wouldn't be meaningful. Figure 13 The screen 101 shown includes a screen 101A with a display switching button 66, and a screen 103 that switches from screen 101A by pressing the display switching button 66 on screen 101A. The display switching button 66 is also displayed on screen 103. If the user presses the display switching button 66, the screen switches back to screen 101A.

[0099] In screen 103, the work progress status 67 of work site 1 is displayed on a map identical to that of screen 101A. The work progress status 67 can be generated based on the work progress data of each area in work site 1. The work progress data is managed, for example, according to cell units obtained by dividing the map of work site 1 into grids, and the work progress status 67 is displayed by changing the display format of the cells (e.g., shadow patterns and colors) according to the work progress of each cell.

[0100] (Effects obtained from implementation method 3) If configured as in Embodiment 3, the distribution data 62 of shooting locations and the status 67 of the operation can be confirmed on the same map, then the identification model of the updated object can be selected while comparing the operation progress and the shooting locations.

[0101] Furthermore, the date displayed on screen 103 for the work progress status 67 is preferably configured to allow selection of a desired date. For example, it is preferable to select a date that is the same as or close to the image acquisition date 64 of the shooting location distribution data displayed on screen 101A, or a date that is the same as or close to the current date. That is, the work progress status 67 displayed on the screen can be generated from work progress data with a timestamp that is the same as or close to the image acquisition date 64, or from work progress data with a timestamp that is the same as or close to the current date. This allows confirmation of the time when the learning image used in the recognition model creation was acquired and the terrain changes at the current work site 1. This information is useful for determining whether to use the generated recognition model or whether to re-acquire images to generate a new recognition model. By using this information, the updated recognition model can be appropriately selected, ensuring the detection accuracy based on the recognition model.

[0102] The display device for the shooting location distribution data 62 and the work progress status 67 is not limited to the monitor 21 of the control device 22, but may also be displayed on the monitor 21b of the learning device 23 and the monitors of other devices including mobile terminals.

[0103] Furthermore, while the above description shows the distribution data of shooting locations 62 and the work progress status 67, these two can be overlaid on the map of the work site 1 in a recognizable manner. In the case of overlay display, it is preferable, for example, to display the number of images taken for learning purposes as a numerical value and the work progress status as a heat map, thus making it easier to understand both.

[0104] Alternatively, it can be based on or replace the above-mentioned work progress status 67, and a cell that can be recognizablely displayed on a specified date (e.g., the day of the work) for which the work is performed.

[0105] Furthermore, the work progress data can be input from external terminals such as a server (work progress management server) that manages the work progress at work site 1 via a network, or transmitted from a removable medium such as flash memory, or stored in the control device 22 and the learning device 23. Next, an example of the configuration of the work progress management server will be described.

[0106] <Job Progress Management Server 29> Figure 19 This is a schematic diagram of the job progress management server 29 and its peripheral equipment in Implementation Method 3.

[0107] The work progress management server (fourth control device) 29 is connected to an input device 81, including an external input device such as a USB, and a communication device 84, which communicates with external terminals including the control device 22 and the learning device 23. The server 29 has an input / output unit 71, a processing unit (processor) 72, and a storage unit (e.g., a memory) 73. The processing unit 72 executes the program stored in the storage unit 73, thereby functioning as a work site information collection unit 72a that collects work site drawings and terrain information of the work site 1 via the input device 81 and the communication device 84 through the input / output unit 71; a grid division unit 72b that divides the work site 1 into multiple cells by a grid; a grid terrain change calculation unit 72c that calculates the height change of the terrain in each cell using the collected terrain information and the information of each cell; and a transmission information generation unit 72d that sends the calculated terrain change information of each cell to external terminals (e.g., the control device 22 and the learning device 23).

[0108] Server 29 can manage the work site 1 by cell, and by managing the height of each cell, it can calculate the terrain changes of the work site. For example, it can explore cells whose height has changed by more than 1m based on a certain point in time. Processing device 72 saves such grid changes to storage device 73, and, for example, generates transmission information by transmission information generation unit 92d according to a request from learning device 23 and sends it via communication device 84.

[0109] The configuration of the learning device 23 can be the same as that of embodiment 1. Figure 3 The configuration shown is the same. The learning device 23 is able to display the information on the terrain changes of the work site 1 received by the work progress management server 29 on the map of the work site 1.

[0110] (Implementation Method 4) The following uses Figure 20 The following describes Embodiment 4 of the present invention. The hardware configuration of Embodiment 4 is the same as that of Embodiment 1, and the main feature is that the processing performed by the image data receiving unit 35a (processing unit 35) of the learning device 23 is different.

[0111] Figure 20 This is a flowchart illustrating the processing of the learning device 23 (image data receiving unit 35a). The image data receiving unit 35a begins processing whenever it receives a learning image and shooting location information from an external terminal (e.g., control device 22). Figure 20 The processing.

[0112] In step S901, the image data receiving unit 35a (processing device 35) determines whether the received learning image meets specified conditions. These specified conditions may include, for example: an image taken at a specified work site; a link established between the shooting location information and the learning image; an image taken by a camera of a specified camera specification; an image taken during a specified period; an image taken in specified weather conditions; and an image taken at a specified time of day. Furthermore, any combination of these conditions may also be used as specified conditions.

[0113] In step S901, if it is determined that the received learning image meets the specified conditions, proceed to step S903; if it is determined that the specified conditions are not met, proceed to step S902.

[0114] In step S902, the image data receiving unit 35a saves the learning image and the shooting location information to the second database (hereinafter referred to as the second DB) in the storage device 36 configured in the learning device 23.

[0115] In step S903, after saving the received learning images to the first database (hereinafter referred to as the first DB) configured in the storage device 36, the image data receiving unit 35a determines whether the capacity of the first DB exceeds the upper limit. That is, it determines whether there is empty capacity in the first DB for saving the learning images.

[0116] If it is determined that the capacity of the first database (DB) is not exceeded, proceed to step S904 to save the learning image and shooting location information in the first database. Conversely, if it is determined that the capacity of the first database exceeds the limit, proceed to step S905 to save the learning image and shooting location information to the first database by sequentially overwriting data in order of image acquisition date from farthest to closest. Thus, learning images that meet the specified conditions and have the closest image acquisition date are saved in the first database.

[0117] Furthermore, if the determination in step S901 is yes, the flowchart can also be constructed by omitting step S903 and proceeding to step S904. Alternatively, before performing the processing in step S902, the same change in the storage format corresponding to the empty capacity of the second DB can be performed as in steps S903-905.

[0118] (Effects obtained from implementation method 4) If the learning device 23 is configured as in this embodiment, the learning image for the specified conditions is appropriately saved and updated in the first database. Therefore, if a recognition model is generated based on the learning image saved in the first database, a recognition model suitable for the specified conditions can be generated, and as a result, the reduction in recognition accuracy can be suppressed.

[0119] (other) Furthermore, the present invention is not limited to the embodiments described above, but includes various modifications without departing from its spirit and scope. For example, the present invention is not limited to having all the configurations described by the above embodiments, but may also include configurations in which a portion of the configuration is removed. In addition, a portion of the configuration of one embodiment can be added to or replaced with the configuration of other embodiments.

[0120] Furthermore, the constituent elements, functions, and execution processes of the aforementioned control devices 22, 23, 122, and 29 can be implemented, partially or entirely, through hardware (e.g., through integrated circuit design to execute the logic of each function). Additionally, the constituent elements of the aforementioned control devices 22, 23, 122, and 29 can also be read and executed by a processing device (e.g., a CPU) to realize the functions corresponding to each constituent element of the control devices 22, 23, 122, and 29. Information related to this program can be stored, for example, in semiconductor memory (flash memory, SSD, etc.), magnetic storage devices (hard disk drives, etc.), and recording media (disk drives, optical disks, etc.).

[0121] Furthermore, while control lines and information lines deemed necessary for describing these embodiments are shown in the descriptions of the various embodiments described above, this does not necessarily represent all control lines and information lines involved in the product. In fact, it can be assumed that almost all constituent elements are interconnected. Explanation of reference numerals in the attached figures

[0122] 1…Work site, 2…Working machinery (hydraulic excavator), 4…Transportation machinery (dump truck), 5…Working object, 6…Image acquisition device, 19…Camera, 20…Communication device, 20b…Communication device, 21…Monitor (display device), 21b…Monitor (display device), 22…Control device (first control device), 23…Learning device (second control device), 24…Network, 25…Input device, 25b…Input device, 26…Positioning device, 27…Input device, 29…Control device, 29…Work progress management server (fourth control device), 31…Input / output unit (input / output interface), 32…Recognition model update unit, 32…Processing device, 32a…Image data storage unit, 32b…Object recognition unit, 32c…Recognition model update unit, 32d…Position calculation unit, 33…Storage device 34… Input / output unit (input / output interface), 35… Processing device, 35a… Image data receiving unit, 35c… Location distribution data generation unit, 35d… Recognition model generation unit, 36… Storage device, 62… Shooting location distribution data, 64… Image acquisition date, 64… Shooting time information (image acquisition date), 65… Update button, 66… Display switching button, 67… Work progress status, 71… Input / output unit, 72… Processing device (processor), 72a… Work site information collection unit, 72b… Grid segmentation unit, 72c… Grid terrain change calculation unit, 72d… Transmit information generation unit, 73… Storage device, 101… Display screen, 101A… Display screen, 102… Display screen, 103… Display screen, 105… Sensor box, 106… Support, 122… Control device (third control device).

Claims

1. An object recognition system, comprising: A camera is installed at the work site to capture the work site; A first control device, which takes as input an image acquired by the camera, uses a machine learning-based object recognition model to identify objects in the image; and The second control device stores learning images captured by the camera at the work site and generates the object recognition model based on these learning images. The first control device will use the object recognition model sent from the second control device as a new object recognition model. The object recognition system is characterized by the following features: The second control device also stores the shooting position information of the learning images, and generates shooting position distribution data based on the shooting position information of the learning images used in the generation of the object recognition model, and displays it on a monitor. The shooting position distribution data represents the distribution of shooting positions of the learning images used in the generation of the object recognition model.

2. The object recognition system according to claim 1, characterized in that, The monitor displays a map of the work site. The distribution data of the shooting locations is displayed on a map of the work site.

3. The object recognition system according to claim 1, characterized in that, The monitor displays a map of the work site, divided into multiple areas. The map displays the amount of learning images taken in each region of the map, which are used in the generation of the object recognition model.

4. The object recognition system according to claim 3, characterized in that, Based on the work progress data of each area in the work site, the work progress status of the work site is also displayed on the map.

5. The object recognition system according to claim 1, characterized in that, The camera, the first control device, and the positioning device that receives signals from multiple positioning satellites and calculates the position are mounted on the work machinery operating at the work site. The learning images are captured by the camera. The shooting location information is calculated using the positioning device. The first control device and the second control device are capable of communicating with each other. The first control device sends the learning image and the shooting location information to the second control device.

6. The object recognition system according to claim 1, characterized in that, have: A camera is set up at a desired location in the work site to capture images for learning purposes; The positioning device receives signals from multiple positioning satellites and calculates the captured location information; as well as The third control device is connected to the camera and the positioning device, and is capable of communicating with the second control device. The third control device sends the learning image captured by the camera and the shooting position information calculated by the positioning device to the second control device.

7. The object recognition system according to claim 6, characterized in that, The differences between the set camera and the set camera, including parameters related to camera specifications such as sensor size, focal distance, setting height, and setting pitch angle, are within a specified range.

8. The object recognition system according to claim 1, characterized in that, The second control device sends the object recognition model to the first control device based on a sending command input after the shooting location distribution data is displayed on the monitor.

9. The object recognition system according to claim 2, characterized in that, The second control device generates the object recognition model based on a learning image of a specified work site from a learning image sent from the first control device.

10. The object recognition system according to claim 1, characterized in that, The second control device also stores the shooting time information of the learning images, and displays the shooting time information of the learning images used in the generation of the object recognition model on the monitor.

Images