Object detection system

Abstract

To provide an object detection system capable of efficiently collecting learning data that can improve object detection accuracy.SOLUTION: An object detection system includes: a control device 22 that inputs an image captured by a camera 19 and detects a detection object in the image using a machine-learned object detection model; and a learning device 23 that generates an object detection model based on teaching data including multiple images containing the detection object and its positional information in the images. The learning device 23 generates positional distribution data indicating a distribution of detection object's positions in the multiple images based on the teaching data and displays it on a monitor.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to an object detection system that is applied to, for example, a site management system or a work machine. [Background technology]

[0002] In recent years, with the improvement of computer performance, there has been a remarkable development in machine learning image recognition technology using deep neural networks, etc. At work sites, technology that uses machine learning to detect objects from images is being increasingly introduced in the form of on-board devices on hydraulic excavators and other work machines, and site sensors installed on-site.

[0003] The detection accuracy of image recognition technology using machine learning is closely related to the training data, and training data is extremely important. In general, detection accuracy is high in environments similar to the environment in which the training data was acquired, and low in untrained environments.

[0004] At work sites where work machines are operating, the backgrounds of the images acquired at each site vary greatly depending on the type of work and the region, making it difficult to create a model with high detection accuracy. For this reason, there is a need for technology that can improve detection accuracy by learning using image data from the work site.

[0005] Patent Document 1 proposes a system that collects learning data at a work site, transmits the data to a learning server for learning, and uses the learning results for object detection. [Prior art documents] [Patent documents]

[0006] [Patent Document 1] Japanese Patent Publication No. 2022-150641 Summary of the Invention [Problem to be solved by the invention]

[0007] The purpose of an object detection system is to accurately detect a predetermined detection target from an image. For example, to detect a worker as a detection target, it is necessary to collect images containing the worker as learning data and perform learning. By using the system described in Patent Document 1, images can be collected from a work site, the image data can be used for learning, and the learning results can be used for object detection at the work site.

[0008] However, if the training data collected is image data that only contains backgrounds and does not include the target, the system's detection performance may be degraded. Furthermore, the shape and size of the target in the image vary depending on the relative position between the camera and the target. Therefore, in order to accurately detect targets of various shapes and sizes, it is necessary to collect image data of targets placed in various relative positions.

[0009] However, the system in Patent Document 1 does not mention what kind of image data should be collected as learning data to improve detection accuracy, and it is difficult to expect improvement in detection accuracy by simply collecting image data randomly.

[0010] An object of the present invention is to provide an object detection system that can efficiently collect learning data that can improve object detection accuracy. [Means for solving the problem]

[0011] The present application includes multiple means for solving the above-mentioned problems. One example is an object detection system including a camera, a first control device that inputs an image acquired by the camera and detects a detection target in the image using a machine-learned object detection model, and a second control device that generates the object detection model based on training data including a plurality of images captured by the camera that show the detection target and position information of the detection target in the plurality of images. The second control device generates position distribution data that indicates the distribution of positions of the detection target in the plurality of images based on the training data, and the generated position distribution data displays on a monitor positions where the detection target is present and positions where the detection target is not present, distinguishing between them. [Effects of the Invention]

[0012] According to the present invention, it is possible to efficiently collect learning data that can improve object detection accuracy. [Brief explanation of the drawings]

[0013] [Figure 1] 1 is a schematic configuration diagram of an object detection system for a work site according to an embodiment of the present invention; [Figure 2] FIG. 2 is a configuration diagram showing an example of a control device 22 and its peripheral devices. [Figure 3] FIG. 2 is a configuration diagram showing an example of a learning device 23 and its peripheral devices. [Figure 4] 1 is a schematic diagram of a work site 1 where a work machine 2 equipped with a control device 22, a camera 19, and a monitor 21 is in operation. [Figure 5] FIG. 5 is a schematic configuration diagram of a work machine 2 in FIG. 4. [Figure 6] 2A is a side view of a revolving unit 12, and FIG. 2B is a diagram showing an example of an image acquired by a camera 19. [Figure 7] 10 is a flowchart showing the processing of an object detection unit 32b (control device 22). [Figure 8] FIG. 10 is a diagram showing an example of a case where a detection target (person) is included in an image captured by a camera. [Figure 9]10 is a flowchart of a process in which the control device 22 stores image data acquired by the camera 19 using the image data storage unit 32a and transmits the image data to the learning device 23. [Figure 10] FIG. 2 is a diagram showing an area on a monitor 21 where an image from a camera 19 is displayed. [Figure 11] 10 is a flowchart of an example of processing by a teacher data generating unit 35b of a learning device 23. [Figure 12] FIG. 10 is an explanatory diagram illustrating the generation of training data from an image 71 containing a detection target object 72. [Figure 13] 10 is a flowchart showing an example of processing by a position distribution data generating unit 35c. [Figure 14] 14 is a flowchart showing details of an example of the position distribution data creation process of S602 in FIG. 13. [Figure 15] FIG. 10 is an explanatory diagram of the process of creating position distribution data 83. [Figure 16] 10 is a flowchart showing the processing of a learning control unit 32d. [Figure 17] FIG. 6 is a diagram showing an example of a notification button 61 and a learning start request screen 62. [Figure 18] 10 is a flowchart showing the processing of the learning device 23 (detection model generation unit 35d). [Figure 19] An explanatory diagram of deep learning. [Figure 20] 10 is a flowchart showing the processing of a detection model update unit 32c. [Figure 21] FIG. 10 is a schematic configuration diagram of an example of a control device 22 and its peripheral devices according to a second embodiment. [Figure 22] FIG. 11 is a diagram showing the relationship between a learning device 23 and a plurality of control devices 22 in the third embodiment. [Figure 23] 10A and 10B are explanatory diagrams illustrating cases where a person is photographed with a camera at different heights and angles. [Figure 24] FIG. 11 is a diagram showing an example of a learning start request screen 62A according to the third embodiment. [Figure 25] FIG. 11 is a configuration diagram showing an example of a learning device 23A and its peripheral devices according to the third embodiment. [Figure 26]FIG. 10 is a configuration diagram showing an example of a control device 22A and its peripheral devices in the fourth embodiment. [Figure 27] FIG. 10 is a configuration diagram showing an example of a learning device 23B and its peripheral devices according to the fourth embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0014] Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

[0015] <Object detection system> Fig. 1 is a schematic diagram of an object detection system for a work site according to an embodiment of the present invention. The object detection system in Fig. 1 includes a camera 19 that captures images of the work site, a control device (first control device) 22, a learning device (second control device) 23, and a monitor 21.

[0016] The control device 22 (first control device) is, for example, a computer equipped with a processing device (a processor such as a CPU), a storage device (for example, a memory), and an input / output interface (input / output unit), and receives an image captured by the camera 19 and detects a detection target (for example, a person) in the image using a machine-learned object detection model. The control device 22 is, for example, a controller mounted on a work machine such as a hydraulic excavator at a work site and used to control the work machine, or a computer connected to a camera (site sensor) installed at the work site.

[0017] The learning device 23 (second control device) is, for example, a computer equipped with a processing device (a processor such as a CPU), a storage device (for example, a memory), and an input / output interface (input / output unit), and generates an object detection model based on training data including multiple images of detection targets at the work site and position information of the detection targets (for example, people) in the multiple images. The object detection model generated here is transmitted to the control device 22 and used by the control device 22 to detect the detection targets.

[0018] Based on the teacher data, the learning device 23 generates position distribution data (details will be described later, for example, a screen display indicated by reference numeral 83 in FIG. 1 ) which is part of the teacher data and indicates the distribution of positions of the detection target on a plurality of images in which the detection target appears, and displays the data on the monitor 21. The monitor 21 may be connected to the control device 22, the learning device 23, or a mobile terminal different from the control device 22 and the learning device 23, for example.

[0019] The learning device 23 is, for example, a server installed in a control center installed at a work site or in a remote location.

[0020] The control device 22 and the learning device 23 are connected to each other so that they can communicate with each other. For example, the control device 22 and the learning device 23 each have a communication device such as a Wi-Fi antenna, and are connected wirelessly via the network 24. The network 24 is, for example, the Internet or a local area network. Note that the form of communication between the network 24 and the control device 22 and the learning device 23 is not limited to wireless communication, and wired communication may also be used.

[0021] (Embodiment 1) <Control device 22 (first control device)> 2 is a configuration diagram showing an example of the control device 22 and its peripheral devices. The control device 22 includes a processing device 32, a storage device 33, and an input / output unit (input / output interface) 31. The control device 22 is connected to a camera (imaging device) 19 that captures images of the work site, a communication device 20, and a monitor (display device) 21.

[0022] The input / output unit 31 is an interface for connecting peripheral devices such as the camera 19 , the communication device 20 , and the monitor 21 to the control device 22 .

[0023] The processing device 32 can be realized by, for example, a combination of a CPU (Central Processing Unit) and a GPU (Graphics Processing Unit).

[0024] The storage device 33 can be configured with a memory such as a RAM (Random Access Memory) and a storage such as an HDD (Hard Disk Drive) or an SSD (Solid State Drive). The storage device 33 is connected to the processing device 32 and is configured to enable writing and reading processes by the processing device 32. Source files and programs for the processing device 32 to perform various calculations are also stored in the storage device 33.

[0025] The control device 22 can function as an image data storage unit 32a, an object detection unit 32b, a detection model update unit 32c, and a learning control unit 32d by executing a program stored in the storage device 33 in the processing device 32. For convenience, in Fig. 2, the units 32a-32d are illustrated inside the processing device 32. The processes that can be executed by the units 32a-32d will be described later.

[0026] <Camera 19> The camera 19 is preferably fixed at a predetermined height and in a predetermined attitude. The attitude of the camera 19 is preferably determined by a pitch angle (tilt angle) and a yaw angle (pan angle). The object to which the camera is fixed may be a mobile work machine (e.g., a hydraulic excavator or dump truck) operated within the work site, or a stand installed within the work site (however, the stand is portable). The camera 19 may be, for example, a digital camera equipped with a CMOS (Complementary Metal Oxide Semiconductor) image sensor.

[0027] <Communication device 20> The communication device 20 is hardware for communicating with other devices, including the learning device 23, via a network 24. The communication device 20 is also called, for example, a network device, a network controller, a network card, or a communication module. The communication device 20 may include a connector for wired connection or a wireless communication interface. For example, the communication device 20 may include an antenna compatible with LTE-Advanced, a Wi-Fi antenna, or the like.

[0028] <Monitor 21> Monitor 21 displays position distribution data that indicates the distribution of positions of detection targets on a plurality of images (e.g., images captured by camera 19) that show the detection targets of the object detection system. For example, if the detection targets are workers (people), the positions of the detection targets indicate the positions where the detected workers (people) are present, and the distribution of the positions of the detection targets indicates the distribution where workers are present.

[0029] The monitor 21 may be a device that has an input function in addition to a display function. That is, for example, a capacitive touch-enabled display that allows input operations with the operator's finger. However, the monitor 21 is not limited to a touch panel-enabled display, and may be configured by combining a display with an input device such as a joystick or switch.

[0030] <Learning device 23 (second control device)> 3 is a configuration diagram showing an example of learning device 23 and its peripheral devices. Learning device 23 includes processing device 35, storage device 36, and input / output unit (input / output interface) 34. Learning device 23 is connected to communication device 20b, monitor (display device) 21b, and input device 25b.

[0031] The input / output unit 34 is an interface for connecting peripheral devices such as the communication device 20b, the monitor (display device) 21b, and the input device 25b to the learning device 23.

[0032] The processing unit 35 can be realized by, for example, a combination of a CPU and a GPU.

[0033] The storage device 36 can be configured with a memory such as a RAM and a storage such as an HDD or SSD. The storage device 36 is connected to the processing device 35 and is configured to enable writing and reading processes by the processing device 35. Source files and programs used by the processing device 35 to perform various calculations are also stored in the storage device 36.

[0034] The learning device 23 can function as an image data receiving unit 35a, a teacher data generating unit 35b, a position distribution data generating unit 35c, and a detection model generating unit 35d by executing a program stored in the storage device 36 in the processing device 35. For convenience, in FIG. 3, the units 35a-35d are illustrated inside the processing device 35. The processes that can be executed by the units 35a-35d will be described later.

[0035] There are no restrictions on the location where the learning device 23 is installed, as long as it is a location that can be connected to the network 24. It may be installed either outside the work site or inside the work site.

[0036] <Communication device 20b> The communication device 20b is hardware for communicating with other devices, including the control device 22, via the network 24. The communication device 20b is also called, for example, a network device, a network controller, a network card, or a communication module. The communication device 20b may include a connector for wired connection or a wireless communication interface. For example, the communication device 20b may include an antenna compatible with LTE-Advanced, a Wi-Fi antenna, or the like.

[0037] <Monitor 21b> A signal is input from the learning device 23 to the monitor 21b, and the screen is displayed on the monitor 21b. The monitor 21b is, for example, a liquid crystal display. However, similar to the monitor 21 of the control device 22, for example, a capacitive touch display may also be used.

[0038] <Input device 25b> The input device 25b connected to the learning device 23 is not limited to a specific type, but may be, for example, a mouse or a keyboard. It may also be combined with other input devices such as a joystick or a switch. An input signal is sent to the learning device 23 in response to an input operation on the input device 25b.

[0039] In addition, if the learning device 23 is remotely connected to another device connected to the same network 24 using SSH (Secure Shell) or RDP (Remote Desktop Protocol), the monitor 21b and input device 25b can use the display device and input device of the other device.

[0040] <Worksite 1> 4 is a schematic diagram of a work site 1 where a work machine (hydraulic excavator) 2 equipped with a control device 22, a camera 19, and a monitor 21 operates. The work site 1 is equipped with the work machine 2, a worker 3, a transport machine (dump truck) 4, and the like. The work machine 2 performs tasks such as leveling and excavation work, and loading work objects 5 onto the transport machine 4 at predetermined locations. The transport machine 4 is, for example, a dump truck, and transports the work objects 5 and the like while entering and leaving the work site 1. The worker 3 also performs detailed tasks around the work machine 2, such as guiding various machines, cleaning up work objects 5 spilled from the work machine 2, and transporting various tools.

[0041] <Work Machine 2> Figure 5 is a schematic configuration diagram of the work machine (hydraulic excavator) 2 in Figure 4. The work machine 2 comprises a lower traveling body 11, an upper rotating body 12 rotatably mounted on the lower traveling body, a front working mechanism 13 comprising a boom 15, an arm 16 and a bucket 17, and a control room 18 where an operator rides to operate the work machine 2. The front working mechanism 13 comprises a boom 15 rotatably mounted in front of the upper rotating body 12, an arm 16 rotatably mounted at the tip of the boom 15, and a bucket (working implement) 17 rotatably mounted at the tip of the arm 16. The bucket 17 is an example of a working implement (attachment) of the front working mechanism 13 and can be replaced with a grapple, breaker, etc.

[0042] A camera 19 capable of acquiring images of the surroundings of the work machine 2, a communication device 20, a monitor (display device) 21 located inside the cockpit 18, and a control device 22 are installed on the upper rotating body 12. In this embodiment, an example has been given in which the monitor 21 is installed inside the cockpit 18, but the installation location of the monitor 21 is not limited to inside the cockpit 18. Furthermore, the monitor 21 may be connected to the control device 22 via wireless communication.

[0043] The camera 19 is installed on the upper rotating body 12 so as to capture an image of the area behind the work machine 2, as shown in Figure 5, for example. The installation position of the camera 19 does not have to be the rear of the upper rotating body 12, and there may be multiple cameras.

[0044] FIG. 6 shows (a) a side view of the revolving unit 12 and (b) an example of an image acquired by the camera 19. As shown in FIG. 6(a), in the work machine 2, the camera 19 is installed at the rear of the upper revolving unit 12 and is also installed facing downward so that it can capture images of the area around the work machine 2. If the relative position of the worker 3 to the camera 19 changes as shown in FIG. 6(a), the position of the worker and how the worker appears in the image captured by the camera 19 will change significantly. For example, as shown in FIG. 6(b), the size and shape of the worker in the image may differ depending on the relative position of the worker 3 and the camera 19.

[0045] <Object detection process of the control device 22> 7 and 8, a process in which the control device 22 detects a detection target object from an image captured by the camera 19 using the object detection unit 32b will be described. Fig. 7 is a flowchart showing the process of the object detection unit 32b (control device 22).

[0046] In step S201, the object detection unit 32b (control device 22) acquires an image using the camera 19, and the process proceeds to step S202.

[0047] In step S202, the object detection unit 32b detects the detection target using the image acquired in step S202 and a detection model (object detection model) stored in the storage device 33, and then proceeds to S203. As shown in FIG. 8(a), when an image including a worker (person) 51, which is the detection target, is acquired, detection processing is performed using the detection model stored in the storage device 33. The detection model is defined, for example, by a neural network and parameters. The detection result includes, for example, the coordinates of a rectangle (bounding box) surrounding the detection target, the type of the detection target (e.g., person, transport machine, construction machine, etc.), and the detection certainty. The coordinates of the bounding box can be any coordinate of any point related to the rectangle that contains the target, such as the two vertices located at both ends of the diagonal of the rectangle that contains the target, the midpoint of the base of the rectangle, any vertex of the rectangle, or the center of the rectangle.

[0048] In step S203, the object detection unit 32b outputs the detection result acquired in step S202. For example, as shown in Fig. 8(b), the bounding box 52 and the type of detected object 53 can be displayed and output superimposed on the acquired image on the monitor 21.

[0049] <Image storage process of the control device 22> Next, a process in which the control device 22 stores image data (learning data) acquired by the camera 19 in the storage device 33 and transmits the data to the learning device 23 using the image data storage unit 32a will be described using the flowchart of FIG.

[0050] In step S101, the image data storage unit 32a (control device 22) determines whether or not an input signal for starting image storage has been received through operation of the monitor 21. FIG. 10 shows the area on the monitor 21 where the image (camera image) of the camera 19 is displayed. This area can be enlarged or reduced as needed on the screen of the monitor 21. As shown in FIG. 10(a), a record button 42 is provided on the display screen of the monitor 21, and when the record button 42 is pressed by an operator, an image storage start signal is input to the image data storage unit 32a. When the record button 42 is pressed, the image data storage unit 32a determines that an image storage start signal has been received, and the process proceeds to step S102. If the record button 42 has not been pressed, it determines that an image storage start signal has not been input, and the process of step S101 is repeated.

[0051] In step S102, the image data storage unit 32a changes the screen of the monitor 21 to that shown in Fig. 10(b) (i.e., displays "Recording"), and starts capturing images with the camera 19. The screen in Fig. 10(b) has an image saving display 43 and a recording stop button 44. After capturing images (learning data) from the camera 19 via the input / output unit 31, the process proceeds to step S103.

[0052] Returning to FIG. 9, in step S103, the image data storage unit 32a stores the image (learning data) acquired in step S102 in the storage device 33. The image storage format is not limited, and may be a still image or a video. When storing the image, compression may be performed using AVC (Advanced Video Coding) or JPEG (Joint Photographic Experts Group), and the resolution may be specified. Thereafter, the process proceeds to step S104.

[0053] In step S104, the image data storage unit 32a determines whether or not an image storage end signal has been received. If the recording end button 44 (FIG. 10) on the monitor 21 is pressed and an image storage end signal has been input to the image data storage unit 32a (control device 22), the process proceeds to step S105. If an image storage end signal has not been received, the process returns to step S102, and the process of acquiring and storing images is repeated.

[0054] In step S105, the image data storage unit 32a changes the screen of the monitor 21 from FIG. 10(b) to FIG. 10(a), and uses the communication device 20 to transmit the image (learning data) stored in the storage device 33 to the learning device 23. Specifically, on the screen of the monitor 21, as shown in FIG. 10(a), the display 43 indicating that recording is in progress (images are being saved) shown in FIG. 10(b) is erased, and the recording stop button 44 shown in FIG. 10(b) is changed to the recording button 42. Thereafter, the process returns to step S101.

[0055] Note that, here, images from camera 19 mounted on work machine 2 were used as learning data, but images taken by another camera in which the difference in camera specification parameters, including the sensor size, focal length, installation height, and installation angle (pitch angle) of camera 19, falls within a specified range can also be used as learning data for the object detection model of work machine 2. It is preferable that the parameters of the other camera do not differ from those of camera 19 (in other words, in this case, the parameters will be the same). For example, if the camera specification parameters are the same as those of camera 19 mounted on work machine 2, it can be used as learning data for the object detection model of work machine 2. Note that the "difference" in the above is calculated for each parameter, and the "specified range" can be set for each parameter. It is preferable that the "difference" be as small as possible.

[0056] <Teacher data generation process of the learning device 23> The learning device 23 generates teacher data based on the image data (learning data) transmitted from the control device 22.

[0057] FIG. 11 shows a flowchart of the processing by the teacher data generating unit 35b of the learning device 23.

[0058] In step S501, the teacher data generation unit 35b (learning device 23) performs a process of reading image data (learning data) from the storage device 36. Specifically, it reads out, from the image data in the storage device 36, image data that has been transmitted from the control device 22 and does not have correct answer information attached (image data for which teacher data has not been generated). Thereafter, the process proceeds to step S502.

[0059] In step S502, the teacher data generation unit 35b performs a process of generating teacher data using the image data read in step S501. For example, the teacher data generation unit 35b reads an image from the storage device 36, adjusts it to a default resolution (size), and displays it on the monitor 21b. Then, a user viewing the monitor 21b uses the input device 25 to mark the detection target on the screen (for example, a bounding box 75 described below), and adds correct answer information (here, position information and class information (described below) of the detection target) to the image, thereby generating teacher data. Next, the generation of teacher data will be described in detail using drawings.

[0060] The generation of training data will be described using an example in which a detection object 72 (person) is captured in an image (training data) 71 as shown in FIG. 12(a). First, as shown in FIG. 12(b), a rectangular bounding box 75 defined by two points (x0, y0) and (x1, y1) is specified so as to surround the detection object 72 on the image 71. The coordinate values ​​of the two points of the bounding box 75 are preferably coordinate values ​​in a coordinate system set on the image 71 (for example, a coordinate system with the upper left corner of the image as the origin, as shown in FIG. 12(b)). The bounding box 75 may be specified by a user specifying two points via the input device 25, or the learning device 23 (training data generation unit 35b) may detect the detection object 72 and assign the bounding box 75 to surround (for example, circumscribe) the detection object 72. Second, class information is specified to indicate that the detection object 72 within the bounding box 75 is a worker (person). The class information is preferably specified by user input.

[0061] Here, the position information and class information of the detection target object 72 in the image 71 are taken as the correct answer information. Of the correct answer information, the position information of the detection target object 72 can be calculated, for example, from the coordinate values ​​of two points of the bounding box 75. In an example described below, the bottom side of the bounding box 75 is considered to be the position (X, Y) of the detection target object 72, and in the case of FIG. 12(b), the position (X, Y) of the detection target portion 71 can be expressed as x0≦X≦x1, Y=y1. Note that in the example of FIG. 12(b), the coordinate values ​​of the two points of the bounding box 75 are coordinate values ​​in an xy coordinate system set in the image 73. However, using the width w and height h of the image 73, relative values ​​(x0 / w, y0 / h) and (x1 / w, y1 / h) for the width w and height h may also be used as coordinate values.

[0062] Furthermore, instead of the two points constituting the bounding box 75, the coordinate values ​​of any point related to the bounding box 75, such as the midpoint of the base of the bounding box 75, a predetermined vertex among the four vertices constituting the bounding box 75, or the center of the bounding box 75, may be used as the position information of the detection target 72. Furthermore, instead of the bounding box 75, a polygon surrounding the detection target may be specified, and information about the vertices of the polygon (for example, the coordinate values ​​of each vertex) may be used as the position information (correct answer information) of the detection target 72.

[0063] The teacher data generating unit 35b performs a process of adding the correct answer information determined as above to the image data (learning data) in the storage device 36 to generate teacher data, and the process proceeds to step S503.

[0064] In step S503, the teacher data generating unit 35b performs processing to store in the storage device 36 the teacher data 73, which is a combination of the correct answer information and the image data.

[0065] It is preferable that the flow of FIG. 11 be repeated until there is no more learning data to which correct answer information can be assigned.

[0066] <Processing for generating position distribution data of detected objects by learning device 23> The learning device 23 (position distribution data generation unit 35c) performs a process of generating position distribution data (position distribution information) that indicates the distribution of positions of detection objects (e.g., people) on each image that constitutes the teacher data, based on the teacher data stored in the memory device 36.

[0067] Here, the images constituting the training data are referred to as "trainer images." Position distribution data is data generated by the position distribution data generator 35c, which aggregates the positions of detection targets on each training image and displays the distribution in the form of text (including numbers), graphs, figures, and so on. For example, the training image is divided into multiple independent regions, and each training image identifies which of the multiple regions the detection target is located in. The number of training images in each region in which the detection target is located is counted, and the count value (numerical value) is displayed in association with the region. Position distribution data also includes a frequency distribution diagram that shows the distribution of the positions of detection targets on multiple training images on a map that resembles the shape of the training image (camera image). An example of such a frequency distribution diagram is the target position distribution data 83b in Figure 15(e), which will be described later.

[0068] Next, the processing executed by the position distribution data generation unit 35c will be described with reference to Fig. 13 to Fig. 15. Fig. 13 is a flowchart showing the processing by the position distribution data generation unit 35c. This flowchart is preferably started at any timing after new teacher data is added to the storage device 36.

[0069] In step S601, the position distribution data generating unit 35c (learning device 23) reads out the teacher data stored in the storage device 36, and the process proceeds to step S602.

[0070] In step S602, the position distribution data generation unit 35c uses the training data to generate position distribution data of the detection targets related to the training data. Details of the position distribution data generation process will be described with reference to FIGS.

[0071] FIG. 14 is a flowchart showing details of an example of the position distribution data creation process in S602.

[0072] First, in step S701, the position distribution data generation unit 35c (learning device 23) performs initial settings for generating position distribution data. As the initial settings, the position distribution data generation unit 35c generates a sheet 82 that mimics the shape of the teacher image 81 shown in FIG. 15(a), as shown in FIG. 15(b), divides the sheet 82 into a grid of a predetermined format (e.g., an 8×10 grid), and proceeds to step S702. Note that to make it easier to intuitively grasp the distribution of detection targets, it is preferable that the teacher image 81 and the sheet 82 have the same aspect ratio, but they may also have different aspect ratios.

[0073] In step S702, the position distribution data generation unit 35c reads out the i-th (for example, i is an integer from 0 to n) training data stored in the storage device 36, and acquires the position information of the detection target object related to the training data. Then, the process proceeds to step S703.

[0074] In step S703, the position distribution data generation unit 35c updates the grid display on the sheet 82 using the position information of the detected object acquired in step S702. For example, as shown in FIG. 15(c), the sheet 82 is updated by hatching the grid where the bottom of the bounding box is located. That is, the hatched grids on the sheet 82 indicate grids for which training data showing the detected object at that position has been collected (collected grids). Note that in the example of FIG. 15, all grids where the bottom of the bounding box 75 is located are hatched. However, it is also possible to hatch only one grid where a specific point on the bottom of the bounding box (e.g., the midpoint of the bottom) is located. Other methods can also be used as appropriate for the update process based on the bounding box 75. Furthermore, the grids can be represented in any format other than hatching, such as by marking the grids or changing the grid color, as long as they can be identified.

[0075] If there are n pieces of training data stored in the storage device 36, the processes of steps S702 and S703 are performed for each of the n pieces of data. As a result, the positions of the detection objects in the n training images are reflected in the sheet 82, and position distribution data 83 such as that shown in Fig. 15(d) is generated. Hatched grids in the position distribution data 83 in Fig. 15(d) indicate that training data in which the detection objects are located in those grids has been collected.

[0076] In the example of Fig. 15(d), the position distribution data 83 is an example in which collected areas are displayed by hatching in a grid on the sheet 82, but the format of the position distribution data is not limited to this. For example, as shown in Fig. 15(e), position distribution data 83b may be generated on the sheet 82 as a frequency distribution diagram (heat map) showing the distribution of the number of pieces of training data in which the detection target is located in each grid. Displaying it as a frequency distribution diagram in this way makes it easy to determine which grids the detection targets are concentrated in.

[0077] Once the position distribution data 83 is generated, the process proceeds to step S603 in FIG.

[0078] In step S603, the position distribution data generator 35c transmits the position distribution data 83 and a learning start standby signal to the control device 22 via the communication device 20b. The learning start standby signal functions as a signal notifying the control device 22 that the position distribution data 83 can be displayed on the monitor 21 of the control device 22.

[0079] <Learning control process of the control device 22> Next, the processing by the learning control unit 32d of the control device 22 will be described with reference to Figures 16 and 17. Figure 16 is a flowchart showing the processing by the learning control unit 32d, which is repeatedly executed by the control device 22 at a predetermined control period.

[0080] In step S401, the learning control unit 32d (control device 22) determines whether a learning start standby signal has been received from the learning device 23. If a learning start standby signal has been received, the process proceeds to step S402, and if a learning start standby signal has not been received, the process of step S401 is repeated.

[0081] In step S402, the learning control unit 32d displays a notification button (announcement button) 61 on the screen of the monitor 21 to notify that the learning device 23 has generated the position distribution data 83 (which also means that the detection model can be generated (learned)). The notification button 61 is displayed on the screen as shown in, for example, FIG. 17(a).

[0082] In step S403, the learning control unit 32d checks whether a notification confirmation operation has been performed, i.e., whether the notification button 61 has been pressed. If the notification button 61 has been pressed on the monitor screen (if input of the notification confirmation operation has been detected), the learning control unit 32d proceeds to step 404, where it generates a screen (learning start request screen) 62 including position distribution data 83 as shown in FIG. 17(b) and displays it on the monitor 21. The learning start request screen 62 has a button (learning start button) 64. In the generated position distribution data 83, positions where detection objects exist (hatched grids, described below) and positions where detection objects do not exist (unhatched grids, described below) on the teacher data (plurality of teacher images) stored in the learning device 23 are displayed on the monitor in a distinguishable manner.

[0083] A hatched grid in the detection target position distribution data 83 on the learning start request screen 62 means that teacher data in which a detection target exists at that grid position is stored in the learning device 23. Therefore, by checking the position distribution data 83, it is possible to confirm which positions of the detection targets are included in the teacher data used to generate a detection model, and to determine whether or not to generate a detection model using the learning device 23.

[0084] Furthermore, if there is an unhatched grid (area) in the position distribution data 83, it can be recognized that there is no training data in which the detection object is located in that grid. This promotes the collection of image data in which the detection object is located in that grid and the supplementation of training data in order to improve the accuracy of the generated detection model. In other words, it is possible to efficiently collect training data that can improve the object detection accuracy of the detection model.

[0085] If the user determines that there is no problem with grids without training data, the detection model may be generated in this state. The threshold for the number of training data items used to distinguish between hatched and unhatched grids may be greater than zero. That is, grids with less training data than the threshold may be represented as blank (grids without hatching). In this case, grids without hatching can be recognized as having insufficient training data because they do not meet the threshold.

[0086] Returning to the description of Figure 16, in step S405, the learning control unit 32d checks whether a learning start operation has been performed, i.e., whether the learning start button 64 has been pressed. If the learning start button 64 has been pressed (if input of a learning start operation has been detected), the learning control unit 32d proceeds to step 406, where it transmits a learning start command from the control device 22 to the learning device 23. This learning start command triggers the learning device 23 to start generating (learning) a detection model (see Figure 18, described later). The learning control unit 32d also returns the screen of the monitor 21 to the state it was in before the learning start standby signal was transmitted (for example, the learning start request screen 62 is closed and the screen transitions to, for example, the screen of Figure 10(a)), and returns the process to step S401.

[0087] As shown in the figure, the learning start request screen 62 may be provided with a button (learning start hold button) 65 for holding off the generation (learning) of a detection model by the learning device 23. When the button 65 is pressed, the display screen returns to the state before the learning start standby signal was transmitted, and the process returns to step S401.

[0088] <Learning Process of the Learning Device 23 (Detection Model Generation Process)> Next, the processing performed by the detection model generation unit 35d of the learning device 23 will be described with reference to FIGS.

[0089] FIG. 18 is a flowchart showing the processing of the learning device 23 (detection model generating unit 35d).

[0090] In step S801, the detection model generation unit 35d performs processing to determine whether a learning start command has been input from the control device 22. If a learning start command has been received from the control device 22, the processing proceeds to step S802, and if no command has been received, the processing returns to step S801.

[0091] In step S802, the detection model generation unit 35d generates a detection model from the training data. That is, the learning device 23 generates a new detection model from the training data based on a learning start command (step S406 in FIG. 16) input after the position distribution data 83 is displayed on the monitor 21. The detection model generation unit 35d uses, for example, deep learning to generate the detection model. When deep learning is used, as shown in FIG. 19, for a predefined deep neural network 92, parameter tuning is performed to minimize the difference (e.g., the deviation in the coordinate values ​​of the bounding box) between the inference result of the training data image data 91 and the ground truth information 93, and learning is performed (a new detection model is generated). The detection model can be defined, for example, by the deep neural network structure and parameters used. As shown in FIG. 19, using the trained detection model makes it possible to infer information about the bounding box 93 and class information of the detection target from the input image 91, thereby inferring the position and type of the detection target. Once the detection model has been generated through learning, the process proceeds to step S803.

[0092] In step S803, the detection model generation unit 35d uses the communication device 20b to transmit the detection model and a detection model update signal to the control device 22. After transmission, the process returns to step S801.

[0093] <Detection model update process of the control device 22> Next, the processing by the detection model update unit 32c of the control device 22 will be described with reference to Fig. 20. Fig. 20 is a flowchart showing the processing by the detection model update unit 32c.

[0094] In step S301, the control device 22 (detection model update unit 32c) determines whether a detection model update signal has been input. The detection model update signal is transmitted from the learning device 23 and received via the communication device 20 and the input / output unit 31. If a detection model update signal has been received, the process proceeds to step S302; if no update signal has been received, the process of step S301 is repeated.

[0095] In step S302, the detection model update unit 32c performs a detection model update process. The detection model to be updated is received from the learning device 23 via the communication device 20 and the input / output unit 31, and the detection model is stored in the storage device 33. The object detection unit 32b may use the latest detection model among the detection models stored in the storage device 33, or may be able to select the latest detection model.

[0096] (Effects Obtained in Embodiment 1) By using the system of embodiment 1, it is possible to check the position distribution data and thereby confirm which positions on the image contain the detection target in the training data used to generate the detection model, and it is also possible to easily understand which positions on the image contain the detection target and are lacking in images. This makes it possible to collect learning data efficiently at the work site so as to improve the detection accuracy of the detection model.

[0097] It is not necessary to configure the training data so that the detection target is located in every grid on the sheet; for example, grids in which the detection target cannot exist may be ignored.

[0098] Furthermore, in this embodiment, the operator of the work machine, who is the operator of the control device 22, decides via screen 62 (see FIG. 17) whether to start learning by the learning device 23, but a worker other than the operator at the work site (e.g., a site manager, etc.) or an operator of the learning device 23 (e.g., a service provider) may also make the decision via a similar screen. Furthermore, a computer including the control device 22 and learning device 23 may automatically decide whether to start learning based on the distribution of detected objects in the position distribution data 83 (i.e., in this case, a decision is not made via a "screen").

[0099] (Embodiment 2) A second embodiment of the invention will be described below with reference to Figure 21. This embodiment is characterized in that a control device (first control device) 22 and a camera 19 are mounted on a site sensor 6, which is a portable camera installed at a work site 1, and a monitor 21 that displays position distribution data 83 is mounted on a terminal that can communicate with the control device 22 (for example, a mobile terminal 121 such as a smartphone or tablet that includes an input device 25 in addition to the monitor 21). The learning device 23 has the same configuration as in the first embodiment, so its description will be omitted.

[0100] FIG. 21 is a schematic diagram of an example of the control device 22 and its peripheral devices according to this embodiment.

[0101] The site sensor 6 includes a sensor box 101 and a stand 102 that supports the sensor box 101. The sensor box 101 is equipped with a camera 19, a communication device 20, and a control device 22 to which the camera 19 and the communication device 20 are connected. The control device 22 has the same configuration as in the first embodiment, but a monitor (display) 21 mounted on a mobile terminal 121 is configured to be able to communicate with the control device 22 wirelessly. The mobile terminal 121 equipped with the monitor 21 can also be carried by a site worker.

[0102] In this system, training data is generated from images captured by camera 19. After position distribution data 83 is displayed on monitor 21, learning device 23 is triggered by an operation of input device 25 of mobile terminal 121 (specifically, a learning start operation), and generates a new object detection model from the training data based on a learning start command input from mobile terminal 121.

[0103] (Effects Obtained in Embodiment 2) Even when an object detection system using a site sensor 6 as in this embodiment is configured, it is possible to obtain the same effects as in embodiment 1 which uses a work machine 2. A particular advantage is that an object detection system can be constructed by installing a camera 19 at a desired location within the work site 1, independent of the work machine 2.

[0104] Furthermore, if the parameters related to the camera specifications, including the sensor size, focal length, installation height, and installation pitch angle, are close to those of the camera 19 of the site sensor 6, and the difference in the parameters between the two cameras is within a specified range, the image can be used as learning data for the object detection model of the work machine 2, and the object detection model can be shared between the control device 22 in the sensor box 101 and the control device 22 of the work machine 2.

[0105] (Embodiment 3) Next, a third embodiment of the present invention will be described with reference to FIGS.

[0106] 22 is a diagram showing the relationship between a learning device 23 and multiple control devices 22 in this embodiment. The object detection system of embodiment 3 has a configuration in which the number of control devices 22 is increased compared to embodiment 1, and is made up of a learning device 23 and multiple control devices 22. A camera 19 is connected to each control device 22. Furthermore, each control device 22 may be mounted on either the work machine 2 or the site sensor 6.

[0107] For example, when multiple cameras are used in which at least one of the camera specification parameters, including sensor size, focal length, installation height, and installation pitch angle (installation angle), is outside a predetermined range, the images from each camera will show different shapes and sizes of detected objects. For example, when a worker is photographed using two cameras 119a and 119b with different installation heights and installation pitch angles as shown in FIG. 23(a), camera 119a will capture an image such as that shown in FIG. 23(b), and camera 119b will capture an image such as that shown in FIG. 23(c). When an object is detected from an image captured by camera 119b using an object detection model generated based on image data collected by camera 119a, the detection accuracy will decrease because the shape of the detected object in the training data differs from the shape of the detected object in the captured image. Therefore, when multiple cameras with different camera specifications are used, it is necessary to manage image data (including training data, training images, and training data) for each camera with the same camera specifications.

[0108] On the other hand, for cameras 19 with the same camera specifications, the way the detected object is captured is the same, so the learning data, position distribution data 83, and detection model can be shared. For example, if the installation height and installation angle of the camera 19 of the site sensor 6 are set to the same as those of the camera 19 installed on the work machine 2, the learning data can be shared. Therefore, even if image data is not collected by the work machine 2, it is possible to generate a detection model by collecting image data using the site sensor 6.

[0109] Therefore, in this embodiment, when saving images in step S103 of Fig. 9, the image saving unit 32a of each control device 22 saves the images with the unique information of the camera 19 attached so that the specifications (camera specifications) of each camera 19 can be identified. For example, by including the unique number of each camera 19 in the file name of the image data of each camera 19 and saving it, it is possible to identify which camera 19 the image data was acquired from. Note that the same unique information may be attached to image data of multiple cameras 19 with the same camera specifications.

[0110] In addition, in the learning control process of Figure 16 by the learning control unit 32d of the control device 22, information on the camera specifications related to the teacher data on which the position distribution data 83 is based is also transmitted from the learning device 23 to the control device 22, along with the position distribution data 83, and when generating a learning start request screen in step S404, a screen 62A may be generated that includes the camera specifications (for example, the unique number (No. 251) of the camera 19) along with the position distribution data 83, as shown in Figure 24.

[0111] 25 is a diagram showing an example of the configuration of the learning device 23A and its peripheral devices in this embodiment. A camera information management unit 35e is added as a function of the processing device 35 to the configuration of the learning device 23 in the first embodiment.

[0112] The camera information management unit 35e (learning device 23A) can manage information (camera information) related to the specifications of the cameras 19 connected to each control device 22 connected to the network 24. For example, camera information data including the unique number, sensor size, focal length, installation height, and installation angle (pitch angle) of each camera 19 within the work site 1 is stored in the storage device 36 for management. If the camera's unique number is added to the image data (learning data) sent from each control device 22, it is possible to identify other camera specifications from the unique number. The camera information management unit 35e can also modify the camera information data. Identification information for each camera specification may be added to the camera information data.

[0113] When the image data receiving unit 35a (learning device 23A) receives image data from each control device 22 via the communication device 20b, it may classify the image data based on the unique information of the camera 19 linked to each image data and store it in the storage device 36. For example, a directory is generated for each camera 19 to manage the image data. Note that a directory may also be generated for each camera specification to manage the image data.

[0114] The teacher data generation unit 35b (learning device 23A) may store the teacher data so that, in addition to providing correct answer information, it can also identify the unique information and specification information of the camera 19 that captured the image (teacher image) that is the source of the teacher data. For example, the teacher data is managed by creating a directory for each camera or each camera specification. As a result, the storage device 36 stores teacher images captured by multiple cameras with the same camera specifications, including sensor size, focal length, installation height, and installation pitch angle.

[0115] 13, the position distribution data generator 35c (learning device 23A) reads out teacher data with the same camera specifications in step S601, generates position distribution data 83 in step S602, and then transmits a learning start standby signal and the position distribution data 83 to at least one control device 22 connected to a camera 19 with the same camera specifications related to the teacher data used to generate the position distribution data (step S603). In the flow of FIG. 13, the operator or the learning device 23A may be configured to select from which teacher data of which camera specifications the position distribution data 83 is to be generated.

[0116] When the detection model generation unit 35d (learning device 23A) receives the learning start command, it generates a detection model based on training data having the same camera specifications as the camera 19 connected to the control device 22 that sent the learning start command. The generated model is sent to the control device 22 that sent the learning start command. If the latest detection model has already been generated when the learning start command is received, the detection model may be sent to the control device 22 that sent the learning start command.

[0117] (Effects Obtained in Embodiment 3) By using the system of this embodiment, position distribution data and a detection model can be generated appropriately even in a work site where multiple cameras 19 are used.

[0118] Furthermore, if there are multiple combinations of cameras 19 and control devices 22 within the work site 1, and the cameras 19 have common camera specifications, the position distribution data and detection models can be generated using image data captured by the other cameras 19. This allows image data to be collected efficiently, making it possible to efficiently generate highly accurate detection models.

[0119] (Embodiment 4) Hereinafter, a fourth embodiment of the present invention will be described with reference to FIGS.

[0120] The object detection system according to this embodiment includes a control device 22A shown in FIG. 26 and a learning device 23B shown in FIG. 27. This embodiment is characterized in that the learning device 23B executes the learning control process shown in the flowchart of FIG. 16, a learning start request screen 62 including position distribution data 83 is displayed on a monitor 21b connected to the learning device 23B, and an operator of the learning device 23B (e.g., a system provider or a service personnel of the system) determines whether to press a learning start button 64 (i.e., whether or not a new object detection model needs to be generated by the learning device 23B). Therefore, the learning device 23B includes a learning control unit 35f (see FIG. 27) that executes the process shown in FIG. 16. Unlike the first embodiment, the learning control unit 32d is removed from the control device 22A (see FIG. 26). The learning device 23B generates a new object detection model from the teacher data based on a learning start command input after displaying the position distribution data on the monitor 21b.

[0121] Other configurations are the same as those in embodiment 1. That is, the camera 19 and the control device 22A are mounted on the work machine 2, the control device 22A and the learning device 23B are capable of communicating with each other, and the teacher data stored in the storage device 36 of the learning device 23B is generated from images acquired by the camera 19.

[0122] After generating the position distribution data 83, the position distribution data generator 35c of the learning device 23 transmits a learning start standby signal and information about the position distribution data to the learning controller 35f. The learning controller 35f performs the same process as the learning controller 32d of the control device 22 in the first embodiment, as shown in Fig. 16. The learning controller 35f also uses the monitor 21b and input device 25b of the learning device as an output destination and an input source.

[0123] (Effects Obtained in the Fourth Embodiment) In this embodiment, a learning start request screen 62 including position distribution data 83 is displayed on the monitor 21b connected to the learning device 23B, and the operator of the learning device 23B can determine whether or not it is necessary for the learning device 23B to generate a new object detection model. In other words, the same effects as those of the first embodiment can be achieved.

[0124] The present invention is not limited to the above-described embodiments, and includes various modifications within the scope of the gist of the present invention. For example, the present invention is not limited to those having all of the configurations described in the above-described embodiments, and includes those in which some of the configurations are omitted. Furthermore, it is possible to add or replace some of the configurations of one embodiment with the configurations of another embodiment.

[0125] Furthermore, the components related to the control device 22 and learning device 23, as well as the functions and execution processes of the components, may be partially or entirely implemented by hardware (for example, by designing logic for executing each function using an integrated circuit). The components related to the control device 22 and learning device 23 may also be implemented as a program (software) that is read and executed by a processing device (for example, a CPU) to implement the functions of the components of the control device 22 and learning device 23. Information related to the program can be stored, for example, in semiconductor memory (flash memory, SSD, etc.), magnetic storage device (hard disk drive, etc.), and recording medium (magnetic disk, optical disk, etc.).

[0126] In addition, in the above description of each embodiment, the control lines and information lines are those that are considered necessary for the description of the embodiment, but they do not necessarily represent all the control lines and information lines related to the product. In reality, it can be considered that almost all components are interconnected. [Explanation of symbols]

[0127] 1...work site, 2...work machine (hydraulic excavator), 3...worker, 4...transport machine (dump truck), 5...work object, 6...site sensor, 18...operation room, 19...camera (imaging device), 21...monitor (display device), 21b...monitor (display device), 22...control device (first control device), 22A...control device, 23...learning device (second control device), 23A, 23B...learning devices, 24...network, 25...input device, 25b...input device, 31...input / output unit (input / output interface), 32...processing device, 32a...image storage unit, 32a...image data storage unit, 32b...object detection unit, 32c...detection model update unit, 32d...learning control unit, 33...storage device, 34...input / output unit (input / output interface), 35...processing device, 35a...image data receiving unit, 35b...teaching data generation unit, 35c ...position distribution data generation unit, 35d...detection model generation unit, 35e...camera information management unit, 35f...learning control unit, 36...storage device, 42...recording button, 44...recording stop button, 51...worker (person), 52...bounding box, 61...notification button (notification button), 62...learning start request screen, 64...learning start button, 65...learning start hold button, 71...image (learning data), 71...detection target part, 72...detection target object, 73...teaching data, 75...bounding box, 81...teaching image, 82...sheet, 83...position distribution data, 83b...position distribution data, 91...image data, 91...input image, 92...deep neural network, 93...correct answer information, 93...bounding box, 101...sensor box, 102...stand, 121...mobile terminal

Claims

1. A camera and a first control device that receives an image acquired by the camera and detects a detection target in the image using a machine-learned object detection model; a second control device that generates the object detection model based on training data including a plurality of images captured by the camera and showing the detection target and position information of the detection target on the plurality of images, the second control device generates position distribution data indicating a distribution of positions of the detection targets on the plurality of images based on the teacher data; In the generated position distribution data, positions where the detection target exists and positions where the detection target does not exist are displayed on the monitor in a distinguishable manner. An object detection system comprising:

2. 2. The object detection system of claim 1, The position distribution data is a frequency distribution diagram showing the distribution of the positions of the detection targets on the plurality of images on a map that mimics the shape of the images in the training data. An object detection system comprising:

3. 2. The object detection system of claim 1, The second control device newly generates the object detection model based on a learning start command input after the position distribution data is displayed on the monitor. An object detection system comprising:

4. 2. The object detection system of claim 1, the camera, the first control device, and the monitor are mounted on a work machine; the first control device and the second control device are connected to each other so as to be able to communicate with each other; The training data is generated from an image acquired by the camera, The second control device newly generates the object detection model from the training data based on a learning start command input from the first control device after the position distribution data is displayed on the monitor. An object detection system comprising:

5. 2. The object detection system of claim 1, the camera is connected to the first control device and installed at a work site; the monitor is mounted on a mobile terminal capable of communicating with the first control device, the first control device and the second control device are connected to each other so as to be able to communicate with each other; The training data is generated from an image acquired by the camera, The second control device generates the object detection model anew from the training data based on a learning start command input from the mobile terminal after the position distribution data is displayed on the monitor. An object detection system comprising:

6. 2. The object detection system of claim 1, the plurality of images included in the training data are images taken by a plurality of cameras with the same camera specifications, including a sensor size, a focal length, an installation height, and an installation pitch angle; The second control device generates the object detection model and the position distribution data based on training data having the same camera specifications. An object detection system comprising:

7. 2. The object detection system of claim 1, the camera and the first control device are mounted on a work machine, the second control device is a server connected to the monitor, the first control device and the second control device are connected to each other so as to be able to communicate with each other; The training data is generated from an image acquired by the camera, The second control device generates the object detection model anew from the training data based on a learning start command input after the position distribution data is displayed on the monitor. An object detection system comprising:

Images