Work machine, image processing device, and method for detecting specific object from image

Abstract

A work machine is provided with: a machine body; an image acquisition device; and an image processing device that detects a specific object on the basis of the image acquired by the image acquisition device. The image processing apparatus detects a boundary of the sky and a ground object other than the sky and the object from the image, estimates an inclination of the body based on a position of the boundary in the image, and determines a position of the object based on a position of the object in the image and the estimated inclination. The position of the object in a coordinate system fixed to the ground is estimated. According to the work machine, the object detection precision can be improved.

Description

Technical Field

[0001] The present disclosure relates to a work machine, an image processing device, and a method for detecting a specific object from an image. Background Art

[0002] As next-generation agriculture, research and development of smart agriculture that utilizes ICT (Information and Communication Technology) and IoT (Internet of Things) is being promoted. Research and development of automation and unmanned operation of agricultural machines such as tractors, harvesters, transplanters, and agricultural drones used in farmland is also being promoted. For example, an agricultural machine that moves in a farmland by autonomous driving while performing agricultural operations using a positioning system such as GNSS (Global Navigation Satellite System) that can perform precise positioning is gradually being put into practical use.

[0003] Patent Document 1 discloses an agricultural work machine that includes a photographing device capable of photographing the front in the traveling direction of the machine body in a farmland. The agricultural work machine can detect the presence of an obstacle object in the farmland based on the photographed image and identify the type of the obstacle object. Further, the agricultural work machine can perform output control based on a control mode selected from a plurality of control modes according to the type of the obstacle object (for example, deceleration of the machine body, stop, warning to the obstacle object, etc.).

[0004] Prior Art Documents

[0005] Patent Document 1: Japanese Unexamined Patent Application Publication No. 2020-178619 Summary of the Invention

[0006] Problems to be Solved by the Invention

[0007] In work machines such as agricultural machines and construction machines that detect a specific object based on an image (for example, a visible light image, an infrared image, or a point cloud image) obtained by an image acquisition device such as a photographing device (camera) or LiDAR, improvement in the detection performance of the object is required.

[0008] The present disclosure provides a technique for further improving the detection performance of an object.

[0009] Means for Solving the Problems

[0010] The method according to an embodiment of the present disclosure is a method executed by an image processing device that detects a specific object from an image obtained by an image acquisition device mounted on a work machine. The method includes: obtaining the image from the image acquisition device; obtaining information indicating the operation state of the work machine; extracting from the image a partial image representing a region of interest determined based on the operation state of the work machine; generating an input image based on the partial image; and detecting the object by inputting the input image into a pre-generated learned model.

[0011] The method according to another embodiment of the present disclosure is a method executed by an image processing device that detects a specific object from an image obtained by an image acquisition device mounted on a work machine. The method includes: detecting, from the image, a boundary between the sky and the ground object other than the sky and the object; estimating the inclination of the machine body based on the position of the boundary in the image; and estimating the position of the object in a coordinate system fixed to the ground based on the position of the object in the image and the estimated inclination.

[0012] The method according to still another embodiment of the present disclosure is a method executed by an image processing device that detects a specific object from an image obtained by an image acquisition device mounted on a work machine. The method includes: generating the input image based on the image obtained by the image acquisition device; and detecting the object by inputting the input image into one learned model selected from a plurality of learned models according to the environment around the work machine.

[0013] The method according to still another embodiment of the present disclosure is a method for detecting a specific object from an image obtained by an image acquisition device mounted on a work machine. The method includes: generating the input image based on the image obtained by the image acquisition device; detecting the object by inputting the input image into the learned model; and performing relearning of the learned model based on one or more images in which the object is detected among a plurality of images obtained by the image acquisition device during the operation of the work machine.

[0014] The general or specific aspects of the present disclosure can be implemented by a device, a system, a method, an integrated circuit, a computer program, or a non-transitory computer-readable storage medium, or any combination thereof. The computer-readable storage medium may include both volatile storage media and non-volatile storage media. The device may also be composed of multiple devices. When the device is composed of two or more devices, the two or more devices may be arranged within one device or separately arranged within two or more separate devices.

[0015] Advantages of the Invention

[0016] According to an embodiment of the present disclosure, in a working machine that detects a specific object based on an image acquired by an imaging device, the detection performance of the object can be improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 is a block diagram showing a schematic structure of an agricultural machine as an example of a working machine in an exemplary embodiment of the present disclosure.

[0018] Figure 2 is a block diagram showing an example of the structure of an image processing device.

[0019] Figure 3 is a flowchart showing an example of the operation of the image processing device.

[0020] Figure 4 is a side view schematically showing an example of an agricultural machine.

[0021] Figure 5 is a block diagram showing an example of the structure of an agricultural machine.

[0022] Figure 6 is a diagram showing an example of a path when an agricultural machine travels while performing an operation of harvesting ( ) crops in a farm field.

[0023] Figure 7 is a diagram schematically showing a situation where an object (person) is detected using a camera mounted on an agricultural machine.

[0024] Figure 8 is a diagram schematically showing a flow of processing executed by an ECU (image processing device).

[0025] Figure 9A is a diagram showing an example of a captured image acquired by a camera.

[0026] Figure 9B is a diagram showing an example of a region of interest that can be selected when an agricultural machine turns right.

[0027] Figure 9C is a diagram showing a partial image corresponding to the Figure 9B region of interest shown.

[0028] Figure 9D is a diagram showing an example of a region of interest that can be selected when the agricultural machine turns left.

[0029] Figure 9E is a diagram showing a partial image corresponding to the Figure 9D region of interest shown.

[0030] Figure 10A is a diagram showing an example of a region of interest that can be selected when the agricultural machine moves forward at a relatively high speed.

[0031] Figure 10B is a diagram showing a partial image corresponding to the Figure 10A region of interest shown.

[0032] Figure 11 is a diagram showing an example of an agricultural machine traveling at the outermost periphery of the working area in the farmland.

[0033] Figure 12 is a three-dimensional diagram schematically showing the configuration relationship of the camera coordinate system Σc, the vehicle coordinate system Σv, the world coordinate system Σw, and the reference plane Re.

[0034] Figure 13 is a flowchart showing an example of the image processing performed by the image processing device in Embodiment 2.

[0035] Figure 14 is a diagram showing an example of the boundary between the sky and the ground objects detected from the captured image.

[0036] Figure 15 is a flowchart showing a modified example of Embodiment 2.

[0037] Figure 16 is a flowchart showing another modified example of Embodiment 2.

[0038] Figure 17 is a block diagram showing a structural example of the image processing device in Embodiment 3.

[0039] Figure 18 is a table showing an example of the correspondence between multiple learned models stored in the memory and the environment.

[0040] Figure 19 is a flowchart showing an example of the operation of the image processing device in Embodiment 3.

[0041] Figure 20It is a flowchart showing an example of the operation of an image processing device in the case of selecting a learned model whenever a direction change is made in an agricultural machine.

[0042] Figure 21 It is a flowchart showing an example of the model selection process in step S320.

[0043] Figure 22 It is a flowchart showing a specific example of the operation of the image processing device in the present embodiment.

[0044] Figure 23 It is a diagram showing an example of the detection result of an object.

[0045] Figure 24 It is a diagram showing an example of a construction work vehicle. Detailed Embodiment

[0046] (Definition of Terms)

[0047] In the present disclosure, an "operation machine" means a machine used for a specific purpose such as agriculture or construction. In the present disclosure, an "agricultural machine" means an operation machine used for agricultural purposes. A "construction machine" means an operation machine used for construction purposes. "Operations" include, for example, agricultural operations, construction operations, demolition operations of rubble, snow removal operations, etc. The operation machines of the present disclosure can be mobile machines that can perform operations while moving. Examples of agricultural machines include tractors, harvesters, transplanters, ride-on management machines, vegetable transplanters, lawn mowers, seeders, fertilizer spreaders, agricultural mobile robots, and agricultural unmanned aerial vehicles (e.g., drones). Examples of construction machines include backhoes, wheel loaders, carriers, construction mobile robots, and construction unmanned aerial vehicles. Either an agricultural work vehicle or a construction work vehicle such as a tractor or a combine harvester can function as an "operation machine" alone, or a working machine mounted on or towed by a work vehicle (a working machine mounted on the front of a tractor ( )) and the entire work vehicle can function as one "operation machine". Agricultural machines perform agricultural operations such as tilling, seeding, weed control, fertilizing, transplanting, or harvesting crops on the ground in a farmland. Construction machines perform operations such as transporting sand, gravel, and other items at a construction site. Sometimes these operations are called "ground operations" or simply "operations". Sometimes, a vehicle-type operation machine driving while performing operations is called "operation driving".

[0048] "Autopilot" means controlling the movement of a working machine such as an agricultural machine through the operation of a control device instead of manual operation by a driver. An agricultural machine performing autopilot is sometimes referred to as an "autopilot agricultural machine" or a "robot agricultural machine". In autopilot, not only can the movement of the working machine be automatically controlled, but also the operation actions (e.g., the actions of the working implement mounted on the working machine) can be automatically controlled. In the case where the working machine is a vehicle-type machine, the travel of the working machine through autopilot is called "automatic driving". The control device can control at least one of steering, adjustment of travel speed, start of movement, and stop required for the movement of the working machine. In the case of controlling a working machine equipped with a working implement, the control device can also control actions such as the lifting of the working implement, the start and stop of the actions of the working implement. The movement based on autopilot can include not only the movement of the working machine along a specified path towards a destination, but also the movement following a following target. The working machine performing autopilot can also move partially based on the instructions of the user. In addition, the working machine performing autopilot can operate not only in the autopilot mode but also in the manual driving mode where it moves through the manual operation of the driver. Steering the working machine through the operation of the control device instead of manually is called "automatic steering". Part or all of the control device can also be located outside the working machine. Communication of control signals, commands, or data, etc. can be carried out between the control device located outside the working machine and the working machine. The working machine performing autopilot can also sense the surrounding environment and move autonomously without human participation in the control of the movement of the working machine. The working machine capable of autonomous movement can travel in a field or outside the field (e.g., on a road) in a unmanned manner. In autonomous movement, detection of obstacles and avoidance actions of obstacles can also be carried out.

[0049] The "image acquisition device" in the present disclosure means a device capable of acquiring an image or information similar thereto. The image acquisition device can be, for example, a photographing device such as a camera capable of acquiring visible light images, infrared images, ultraviolet images, etc. through photographing, a LiDAR sensor capable of acquiring data of a point cloud image, and a radar that can acquire information similar to an image using electromagnetic waves with short wavelengths such as millimeter waves.

[0050] An example of the "control device" in the present disclosure is a computing device having at least one processor and at least one memory storing a computer program (code) that defines a control process to be executed by the processor. Another example of the "control device" is a computing device having a hardware accelerator such as an FPGA (Field-Programmable Gate Array), an ASSP (Application Specific Standard Product), or an ASIC (Application-Specific Integrated Circuit) configured to execute a control process.

[0051] Similarly, an example of the "image processing device" in the present disclosure is a computing device having at least one processor and at least one memory storing a computer program (code) that defines an image processing process to be executed by the processor. Another example of the "image processing device" is a computing device having a hardware accelerator such as an FPGA or an ASIC configured to execute an image processing process.

[0052] The "processor" in the present disclosure is a hardware electronic circuit such as a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a DSP (Digital Signal Processor), an ISP (Image Signal Processor), or an NPU (Neural Network Processing Unit). The "memory" is a hardware electronic circuit such as a ROM (Read Only Memory) or a RAM (Random Access Memory). A part of the memory may also be a storage medium connected to the processor via wiring or a network. These hardware electronic circuits can be implemented by one or more integrated circuits (ICs) or large-scale integrated circuits (LSIs). Each functional unit or block and related components within the electronic circuit may be manufactured individually as separate integrated circuit chips, or a part or all of these functional units or blocks may be combined and manufactured as a single integrated circuit chip.

[0053] A program that defines the actions of a processor is designed such that the processor executes one or more functions, operations, steps, or processes in an embodiment of the present invention.

[0054] Hereinafter, embodiments of the present disclosure will be described. However, sometimes detailed descriptions that are not necessary will be omitted. For example, sometimes detailed descriptions of well-known matters and repeated descriptions related to substantially the same structures will be omitted. This is to prevent the following descriptions from becoming unnecessarily lengthy and to facilitate the understanding of those skilled in the art. In addition, the inventors provide the accompanying drawings and the following descriptions in order for those skilled in the art to fully understand the present disclosure, and do not intend to limit the subject matter described in the scope of the claims by these. In the following descriptions, the same reference numerals are given to structural elements having the same or similar functions.

[0055] The following embodiments are examples, and the technology of the present disclosure is not limited to the following embodiments. For example, the numerical values, shapes, materials, steps, order of steps, layout of display screens, etc. shown in the following embodiments are merely examples, and various changes can be made as long as there is no technical contradiction. In addition, one embodiment and other embodiments can be combined.

[0056] Hereinafter, several embodiments in which the technology of the present disclosure is applied to an agricultural machine as an example of a work machine will be described. In the following descriptions, various technologies for describing agricultural machines can also be applied to construction machines such as construction work vehicles used at construction sites, work vehicles used at disaster sites, snow removal vehicles used in snowy areas, unmanned aerial vehicles (UAVs) that perform operations such as transporting or monitoring articles, and the like.

[0057] (Embodiment 1)

[0058] Figure 1 is a block diagram showing a schematic structure of an agricultural machine 100 in an exemplary embodiment of the present disclosure. Figure 1 The shown agricultural machine 100 includes a photographing device 10, an image processing device 20, and a control device 30. The agricultural machine 100 is a work machine configured to perform agricultural work while moving. Although not shown in Figure 1 , the agricultural machine 100 may include a power source such as an internal combustion engine or a drive motor, and various devices required for movement (such as traveling or flying) such as a traveling device with tires and wheels or a propeller.

[0059] The imaging device 10 is a device such as a camera that acquires an image in the moving direction of the agricultural machine 100. The imaging device 10 is mounted on the agricultural machine 100 so as to be able to acquire an image in the direction in which the agricultural machine 100 moves (for example, the front, rear, right, or left, etc.). The orientation of the imaging device 10 (that is, the direction of the optical axis of the optical system in the imaging device 10) does not necessarily need to be the same as the moving direction of the agricultural machine 100, and may be inclined with respect to the moving direction. For example, the imaging device 10 may be disposed to be inclined downward with respect to the front, rear, right, or left of the agricultural machine 100.

[0060] The agricultural machine 100 may also include a plurality of imaging devices 10 mounted in different directions. The imaging device 10 generates image data by taking pictures during the movement of the agricultural machine 100. In a certain embodiment, the agricultural machine 100 is configured to generate data of a moving image, for example, at a specified frame rate such as 30 fps or 60 fps.

[0061] In Figure 1 this example, the imaging device 10 is used as an example of an image acquisition device. Instead of or in addition to the imaging device 10, a device such as a LiDAR sensor that can acquire point cloud data similar to an image may be used. Alternatively, a radar that can acquire distribution information of surrounding objects similar to an image may be used. A device that can acquire an image or data similar thereto can be used as the "image acquisition device". In this specification, the image acquisition device generating data of a moving image, a still image, or data similar to an image is expressed as "acquire an image".

[0062] The image processing device 20 is a computing device that processes the image acquired by the imaging device 10. The image processing device 20 may include one or more processors and one or more memories. The image processing device 20 may be configured or programmed to perform processing for detecting a specific object from the image acquired by the imaging device 10 (hereinafter, sometimes referred to as "captured image"). The specific object may be, for example, a person, an animal, another agricultural machine, a vehicle, a crop, an obstacle such as a stone or a rock, or any combination of these obstacles. In a certain embodiment, the specific object is a person, and the image processing device 20 is configured or programmed to perform processing for detecting a person from the captured image.

[0063] The control device 30 is a device for controlling the movement of the agricultural machinery 100. The control device 30 may be, for example, a computing device such as an electronic control unit (ECU). The control device 30 in this embodiment controls the movement of the agricultural machinery 100 based on the detection result of the object performed by the image processing device 20. For example, the control device 30 may be configured to stop the movement of the agricultural machinery 100 or to cause a sound output device such as a buzzer to emit a warning sound when a specific object is detected by the image processing device 20. Through such control, it is possible to avoid a collision between the agricultural machinery 100 and an object, or to alert the object (such as a person). In the case where the agricultural machinery 100 has an automatic driving function, the control device 30 may be configured to also perform automatic driving control.

[0064] Figure 2 It is a block diagram showing an example of the structure of the image processing device 20. Figure 2 The image processing device 20 shown has one or more processors 22 and one or more memories 24. The processor 22 is, for example, an electronic circuit (calculation circuit) such as a CPU, a GPU, or an NPU that performs calculation processing. The image processing device 20 may also have multiple processors. The image processing described below may also be performed by multiple processors in a collaborative manner. The memory 24 may be, for example, a ROM such as an EPROM (Erasable Programmable Read-Only Memory) or an EEPROM (Electrically Erasable Programmable Read-Only Memory), or a RAM such as a DRAM (Dynamic Random Access Memory) or an SRAM (Static Random Access Memory). The memory 24 stores a computer program 25 executed by the processor 22 and a learned model 27 for detecting an object from an input image. The program 25 and the learned model 27 may also be stored in a plurality of memories in a dispersed manner. The processor 22 executes the processing of detecting a specific object from the captured image by executing the computer program 25 stored in the memory 24. In addition, a hardware accelerator such as an FPGA or an ASIC configured to execute the image processing in this embodiment may be provided in the image processing device 20 instead of the processor 22 such as a CPU or a GPU. Such a hardware accelerator can replace the processing executed by the processor in the following description.

[0065] The learned model 27 can be, for example, a machine learning model (hereinafter, sometimes also referred to as an "AI model") trained by an algorithm based on machine learning or artificial intelligence (AI) technologies such as a convolutional neural network (CNN) or a Vision Transformer (ViT). The learned model 27 can be, for example, a model that detects an object from an image based on an object detection algorithm such as SSD (Single Shot Multibox Detector), YOLO (You Only Look Once), R-CNN (Regions with Convolutional Neural Network), Fast R-CNN, Faster R-CNN, or RetinaNet.

[0066] The image processing device 20 generates an input image for input to the model 27 by performing required preprocessing on the captured image generated by the imaging device 10. The image processing device 20 detects a specific object (such as a person) from the image by inputting the input image into the model 27.

[0067] The image processing device 20 in the present embodiment does not detect an object from the entire captured image obtained by the imaging device 10, but detects an object from a region of interest that is a part of the captured image. Specifically, the image processing device 20 extracts a partial image representing the region of interest from the captured image, and generates an input image for the learned model 27 by performing prescribed preprocessing on the partial image. At this time, the image processing device 20 dynamically changes at least one of the position and size of the region of interest according to the operation state of the agricultural machine 100. For example, the image processing device 20 shifts the region of interest to the right when the agricultural machine 100 turns right, and shifts the region of interest to the left when the agricultural machine 100 turns left. Alternatively, the image processing device 20 can shift the region of interest downward when the attitude of the agricultural machine 100 tilts upward, and shift the region of interest upward when the attitude of the agricultural machine 100 tilts downward. In addition, the image processing device 20 can also change the size of the region of interest according to the moving speed of the agricultural machine 100. For example, it can be that the higher the moving speed of the agricultural machine 100, the smaller the region of interest. By such processing, it is possible to more accurately detect a specific object such as a person.

[0068] Generally, in the case of processing for detecting a specific object from an image using a machine learning model, the image is resized to a specified number of pixels (e.g., 300×300 pixels, etc.) suitable for the model to be used and input into the model. The number of pixels of the resized input image is generally smaller than that of the original image. Therefore, if the original image is large, the reduction in fineness when converted into the input image may become significant, and the performance of object detection may be significantly reduced.

[0069] Therefore, the image processing device 20 in the present embodiment extracts a partial image representing a region of interest determined based on the operation state of the agricultural machine 100 from the captured image, and generates an input image based on the partial image. In the presence of an object such as a person, a region that may become an obstacle to the movement of the agricultural machine 100 is adaptively selected as the region of interest according to the operation state of the agricultural machine 100. When resizing the partial image to the specified number of pixels suitable for the model, compared with the case of resizing the original captured image, the reduction in fineness of the region of interest where an object may exist is suppressed. Thereby, an object that may become an obstacle to the movement of the agricultural machine 100 can be detected more accurately.

[0070] Figure 3 is a flowchart showing an example of the operation of the image processing device 20. In Figure 3 the example shown, the image processing device 20 detects a specific object from the captured image by performing the operations of steps S110 to S150 during the operation of the agricultural machine 100, and sends the detection result to the control device 30. The operations of steps S110 to S150 are repeatedly performed during the operation of the agricultural machine 100.

[0071] In step S110, the image processing device 20 acquires the image obtained by the imaging device 10 and information indicating the operation state of the agricultural machine 100. The information indicating the operation state of the agricultural machine 100 may be, for example, information related to at least one of the moving speed, turning state, and tilting state of the agricultural machine 100. The image processing device 20 may be configured to acquire the information indicating the operation state of the agricultural machine 100 from the control device 30, for example. The control device 30 may be configured to generate the information indicating the operation state of the agricultural machine 100 based on signals from one or more sensors mounted on the agricultural machine 100.

[0072] In step S120, the image processing device 20 determines a region of interest from the images acquired by the imaging device 10 based on the operating state of the agricultural machine 100, and extracts a partial image representing the region of interest. The region of interest is the region in the image that is the object of the detection process of the object. The image processing device 20 can be configured, for example, to acquire information related to at least one of the moving speed, turning state, and tilting state of the agricultural machine 100, and change the region of interest based on this information. For example, the higher the moving speed of the agricultural machine 100, the smaller the region of interest the image processing device 20 makes. Alternatively, the image processing device 20 can also move the region of interest to the right when the agricultural machine 100 turns right, and move the region of interest to the left when the agricultural machine 100 turns left. In this case, the greater the steering angle during the turn of the agricultural machine 100, the greater the movement of the region of interest by the image processing device 20. In addition, the agricultural machine 100 can also be equipped with a tilt sensor that measures the tilt amount of the agricultural machine 100. In this case, the image processing device 20 can also change the position of the region of interest according to the measured tilt amount. In a certain embodiment, the tilt sensor measures the pitch angle of the agricultural machine 100 as the tilt amount. In this case, the image processing device 20 can also move the region of interest up or down according to the measured pitch angle. For example, when the agricultural machine 100 is tilted upward as when driving uphill, the upper side of the captured image is mostly likely to be the region corresponding to the sky. In this case, since the object to be detected may be located on the lower side of the image, the image processing device 20 can also move the region of interest downward. Conversely, when the agricultural machine 100 is tilted downward, the image processing device 20 can also move the region of interest upward.

[0073] In step S130, the image processing device 20 generates an input image for input to the learned model 27 based on the extracted partial image. The image processing device 20 generates the input image through preprocessing that includes compressing (resizing) the number of pixels of the partial image to a preset number of pixels. In addition to resizing the image, the preprocessing can also include various processes such as normalization, color space conversion, and noise removal. Thus, an input image representing the region of interest appropriately selected according to the operating state of the agricultural machine 100 is generated.

[0074] In step S140, the image processing device 20 detects a specific object (such as a person) by inputting the input image into the learned model 27 stored in the memory 24. For example, the image processing device 20 may be configured to output, as a detection result, coordinate information indicating the position of a rectangular box (referred to as a "bounding box") that encloses the region where the object exists, as well as information on the width and height of the bounding box, in the case where a specific object exists in the image. Alternatively, the image processing device 20 may output, as a detection result, a signal indicating whether a specific object exists in the input image. In the case where the distance from the imaging device 10 to the object calculated based on the position of the bounding box in the input image is less than a threshold value, the image processing device 20 may also determine that an object exists in the input image.

[0075] In step S150, the image processing device 20 sends the detection result of the object to the control device 30. For example, the image processing device 20 may also send a signal indicating whether a specific object exists in the input image to the control device 30. Alternatively, the image processing device 20 may send a signal indicating that meaning to the control device 30 only when a specific object is detected in the input image. In addition, the image processing device 20 can not only detect a specific object from the input image, but also estimate the distance from the imaging device to the object based on the input image, and send a signal indicating the existence of the object to the control device 30 only when the distance is less than a threshold value.

[0076] If the control device 30 receives a signal indicating the detection result from the image processing device 20, the control device 30 controls the operation of the agricultural machine 100 according to the detection result. For example, the control device 30 may be configured to stop the movement of the agricultural machine 100 or cause a buzzer or a speaker to emit a warning sound if it receives a signal indicating that an object has been detected. By such an operation, it is possible to avoid a collision between the agricultural machine 100 and the object, or to alert the object (such as a person) or the rider of the agricultural machine 100.

[0077] Hereinafter, an embodiment in which the technology of the present disclosure is applied to a combine harvester as an example of the agricultural machine 100 will be described. The technology of the present disclosure is not limited to a combine harvester, and can also be applied to other types of agricultural machines, such as tractors, transplanters, or agricultural drones. In addition, the technology of the present disclosure can also be applied to work machines used in non-agricultural applications (such as construction work vehicles, snow removal vehicles, or mobile work robots). In the following description, a camera is used as an example of the image acquisition device, but other types of image acquisition devices such as a LiDAR sensor capable of acquiring point cloud data similar to an image, or a radar capable of acquiring distance distribution information of surrounding objects similar to an image can also be used.

[0078] [1. Structure]

[0079] Figure 4 is a side view schematically showing an example of the agricultural machine 100. The agricultural machine 100 in the present embodiment is, for example, a harvester such as a combine harvester. The agricultural machine 100 performs operations such as cutting crops in a farm field, threshing the cut crops, and discharging the harvested material after threshing. The crops are, for example, plants capable of harvesting grains such as rice, wheat, corn, and soybeans. Figure 4 The symbols F, B, U, and D shown respectively represent front, rear, upper, and lower.

[0080] The agricultural machine 100 includes a machine body 101 and a traveling device 102. Figure 4 The shown traveling device 102 includes a plurality of wheels (crawlers) equipped with tracks. The traveling device 102 may instead of tracks include wheels with tires. Above the machine body 101, a cabin 110 is provided.

[0081] In front of the traveling device 102, a cutting device 103 for cutting crops is provided so that its height can be adjusted. Above the cutting device 103, a reel 109 for lifting the stem portions of the ( ) crops is provided so that its height can be adjusted. Behind the cabin 110, a threshing device 105 and a box 106 for storing the harvested material are arranged side by side in the left - right direction. The threshing device 105 threshes the cut crops. The box 106 stores harvested materials such as grains obtained by threshing. Behind the threshing device 105, a straw discharge processing device 108 is provided. The straw discharge processing device 108 shreds the stem portions and the like after the harvested materials such as grains are removed and discharges them to the outside.

[0082] A conveying device 104 for conveying the cut crops is provided between the cutting device 103 and the threshing device 105. On the box 106, a discharging device 107 for discharging the harvested material is provided. The harvested material is discharged to the outside from a discharge port 117 located at the top end of the discharging device 107 having a cylindrical shape. The discharging device 107 can perform a pitching motion and a rotating motion and can change the position of the discharge port 117. The structures and operations of various devices for performing harvesting operations such as the cutting device 103, the conveying device 104, the threshing device 105, the discharging device 107, the straw discharge processing device 108, and the reel 109 are well - known, so detailed descriptions thereof are omitted here.

[0083] The agricultural machine 100 in the present embodiment can operate in both a manual driving mode and an autonomous driving mode. In the autonomous driving mode, the agricultural machine 100 can travel in an unmanned manner while performing operations of harvesting crops in a farm field.

[0084] As Figure 4 shown, the agricultural machine 100 includes an engine (engine) 111 and a transmission (transmission) 112. Inside the cabin 110, a driver's seat, an operating lever, an operation terminal (terminal monitor), and a set of switches for operation are provided.

[0085] The agricultural machine 100 includes a plurality of sensing devices for sensing the environment around the agricultural machine 100. In Figure 4 the example shown, the plurality of sensing devices include a laser sensor 125, a plurality of cameras 126, and a plurality of millimeter-wave radars 127.

[0086] The laser sensor 125 is a ranging device that can measure the distance to a reflection point by emitting laser light and detecting the reflected light, and is also called a LiDAR sensor. The laser sensor 125 can obtain information on the distance distribution to surrounding ground objects by changing the emission direction of the laser light. Figure 4 The exemplified laser sensor 125 is disposed at the front part of the agricultural machine 100. The laser sensor 125 can also be disposed at the side part or the rear part of the agricultural machine 100. The laser sensor 125 may include a light source that generates laser light, a detector that detects the reflected light, and a processing circuit that processes the signal of the detected reflected light. The laser sensor 125 may also include a beam scanner that changes the direction of the emitted light beam. The laser sensor 125 can be configured to generate sensor data such as point cloud data representing the distance and direction from an object in the environment around the agricultural machine 100 to each measurement point, or the three-dimensional or two-dimensional coordinate values of each measurement point. The sensor data output from the laser sensor 125 is processed by the control device of the agricultural machine 100. The control device can, based on the sensor data, measure the height or the degree of lodging of the crops existing around the agricultural machine 100, and adjust the height of the cutting device 103 or the vehicle speed according to the height or the degree of lodging of the crops. It can also utilize the point cloud data output from the laser sensor 125 for the detection of an object.

[0087] The camera 126 is an example of a photographing device that photographs the environment around the agricultural machine 100 and generates image data. The camera 126 can be disposed, for example, at the front, rear, left, and right of the agricultural machine 100. The image obtained by the camera 126 is sent to the control device mounted on the agricultural machine 100. This image is used to detect obstacles such as people existing around the agricultural machine 100 by image processing during autonomous driving.

[0088] The millimeter-wave radar 127 is a sensor for detecting metal objects such as vehicles existing around the agricultural machine 100. InFigure 4 In the example shown, two millimeter-wave radars 127 are provided at the front part and the rear part of the agricultural machine 100. The millimeter-wave radars 127 may also be arranged at other parts such as the side part of the agricultural machine 100.

[0089] The agricultural machine 100 further includes a GNSS unit 120. The GNSS unit 120 includes a GNSS receiver and functions as a positioning device for obtaining the positioning data of the agricultural machine 100. The GNSS receiver may include: an antenna that receives signals from GNSS satellites; and a processor that calculates the position of the agricultural machine 100 based on the signals received by the antenna. The GNSS unit 120 receives satellite signals transmitted from multiple GNSS satellites and performs positioning based on the satellite signals. GNSS is a general term for satellite positioning systems such as GPS (Global Positioning System), QZSS (Quasi-Zenith Satellite System, such as Michibiki), GLONASS, Galileo, and BeiDou. In the present embodiment, the GNSS unit 120 is provided on the upper part of the cabin 110, but it may also be provided at other positions.

[0090] The GNSS unit 120 may also include an inertial measurement unit (IMU). The position data can be supplemented by using the signals from the IMU. The IMU can measure the inclination and minute movements of the agricultural machine 100. By using the data obtained by the IMU to supplement the position data based on satellite signals, the positioning performance can be improved. The IMU may also be arranged at a position different from that of the GNSS unit 120.

[0091] The engine 111 may be, for example, a diesel engine. An electric motor may also be used instead of the diesel engine. The transmission device 112 can change the driving force and moving speed of the agricultural machine 100 by shifting gears. The transmission device 112 can also switch the forward and backward movement of the agricultural machine 100.

[0092] In the form where the agricultural machine 100 is equipped with a crawler-type traveling device 102, the traveling direction of the agricultural machine 100 can be changed by making the rotational speeds of the left and right wheels equipped with crawlers different from each other, or by making the rotational directions of the left and right wheels different from each other. In the mode where the agricultural machine 100 is equipped with a traveling device including wheels with tires, the control device of the agricultural machine 100 can control the steering angle of the steering wheel by controlling a power steering device ( )(The rudder angle changes, causing a change in the driving direction of the agricultural machine 100.)

[0093] Figure 4 The agricultural machine 100 shown can be driven by a human, but can also correspond only to driverless operation. In this case, structural elements such as the cabin 110, the steering device, and the driver's seat, which are only required for human driving, may not be provided in the agricultural machine 100. The driverless agricultural machine 100 can travel through autonomous driving or remote operation by the user.

[0094] Figure 5 It is a block diagram showing a structural example of the agricultural machine 100. Figure 5 The agricultural machine 100 shown includes a GNSS unit 120, a laser sensor 125, a camera 126, a millimeter-wave radar 127, a terminal monitor 131, an operation switch group 132, a buzzer 133, a drive device 140, a power transmission mechanism 141, a lamp 142, a sensor group 150, a control system 160, and a communication device 190. These structural elements are communicably connected to each other via a bus.

[0095] The GNSS unit 120 includes a GNSS receiver 121, an RTK receiver 122, an inertial measurement unit (IMU) 123, and a processing circuit 124. The sensor group 150 includes various sensors such as a vehicle speed sensor 151, a steering angle sensor 152, and an illuminance sensor 153. The control system 160 includes a storage device 164, electronic control units (ECUs) 165, 166, 167. In Figure 5 structural elements with a relatively high relevance to the operation of the driverless operation of the agricultural machine 100 are shown, and illustrations of other structural elements are omitted.

[0096] The GNSS receiver 121 included in the GNSS unit 120 receives satellite signals transmitted from multiple GNSS satellites and generates GNSS data based on the satellite signals. The GNSS data is generated in a prescribed format such as the NMEA-0183 format, for example. The GNSS data may include values representing the identification number, elevation angle, azimuth angle, and reception intensity of each satellite from which the satellite signal is received, for example.

[0097] Figure 5The illustrated GNSS unit 120 can use RTK (Real Time Kinematic)-GNSS to position the agricultural machine 100. In RTK-GNSS-based positioning, in addition to satellite signals transmitted from multiple GNSS satellites, correction signals transmitted from a reference station are also used. The reference station can be set near the farmland where the agricultural machine 100 operates (for example, at a position within 10 km from the agricultural machine 100). The reference station generates a correction signal in, for example, RTCM format based on the satellite signals received from multiple GNSS satellites and transmits it to the GNSS unit 120. The RTK receiver 122 includes an antenna and a modem and receives the correction signal transmitted from the reference station. The processing circuit 124 of the GNSS unit 120 corrects the positioning result based on the GNSS receiver 121 based on the correction signal. By using RTK-GNSS, positioning can be performed with an accuracy of, for example, several centimeters of error. Position data including information on latitude, longitude, and altitude is obtained through high-precision positioning based on RTK-GNSS. The GNSS unit 120 calculates the position of the agricultural machine 100, for example, at a frequency of about once to ten times per second.

[0098] In addition, the positioning method is not limited to RTK-GNSS, and any positioning method (such as an interference positioning method or a relative positioning method) that can obtain position data with the required accuracy can be used. For example, positioning using VRS (Virtual Reference Station) or DGPS (Differential Global Positioning System) can also be performed. In the case where position data with the required accuracy can be obtained even without using the correction signal transmitted from the reference station, the correction signal can be not used to generate position data. In this case, the GNSS unit 120 may not include the RTK receiver 122.

[0099] The IMU 123 may include a three-axis acceleration sensor and a three-axis gyroscope. The IMU 123 may also include an orientation sensor such as a three-axis geomagnetic sensor. The IMU 123 can function as a motion sensor and output signals representing various quantities such as the acceleration, speed, displacement, and attitude of the agricultural machine 100. In addition to satellite signals and calibration signals, the processing circuit 124 can estimate the position and orientation of the agricultural machine 100 with higher accuracy based on the signals output from the IMU 123. The signals output from the IMU 123 can be used for correcting or supplementing the position calculated based on satellite signals and calibration signals. The IMU 123 outputs signals at a frequency higher than that of the GNSS receiver 121. Using this high-frequency signal, the processing circuit 124 can measure the position and orientation of the agricultural machine 100 at a higher frequency (e.g., 10 Hz or more). A three-axis acceleration sensor and a three-axis gyroscope can also be provided separately instead of the IMU 123. In addition, the IMU 123 can be provided as a device different from the GNSS unit 120. The IMU 123 functions as an inclination sensor that measures the amount of inclination (e.g., pitch angle, roll angle, yaw angle) with respect to the reference attitude of the agricultural machine 100.

[0100] The camera 126 is an example of a photographing device that photographs the surroundings of the agricultural machine 100. The camera 126 includes, for example, an image sensor such as a CCD (Charge Coupled Device) or a CMOS (Complementary Metal Oxide Semiconductor). The camera 126 may further include an optical system including one or more lenses and a signal processing circuit. The camera 126 photographs the surroundings of the agricultural machine 100 during the travel of the agricultural machine 100 and generates data of an image (e.g., a moving image ( )). The camera 126 can photograph a moving image at a frame rate of, for example, 3 frames per second (fps: frames per second) or more. The image generated by the camera 126 is used, for example, in the detection of obstacles such as people. The image generated by the camera 126 can also be used in positioning or remote monitoring. Multiple cameras 126 can be provided at different positions of the agricultural machine 100, or one camera can be provided. A visible light camera that generates a visible light image and an infrared camera that generates an infrared image can also be provided separately. Both a visible light camera and an infrared camera can be provided. The infrared camera can be used for obstacle detection at night.

[0101] The millimeter-wave radar 127 is set to detect obstacles containing metals such as vehicles existing around the agricultural machine 100. When an object exists at a distance closer to the millimeter-wave radar 127 than a specified distance, the millimeter-wave radar 127 outputs a signal indicating the presence of an obstacle. As Figure 4 shown, multiple millimeter-wave radars 127 can also be set at different positions of the agricultural machine 100. By having multiple millimeter-wave radars 127, it is possible to reduce the dead angle in the surveillance of obstacles around the agricultural machine 100.

[0102] The buzzer 133 is a sound output device for emitting a warning sound notifying an abnormality. The buzzer 133 emits a warning sound when an obstacle is detected, for example, during autonomous driving. The buzzer 133 is controlled by the control system 160.

[0103] The drive device 140 includes various devices required for driving the agricultural machine 100 to travel, such as the engine 111 and the transmission 112. The engine 111 can be equipped with an internal combustion engine such as a diesel engine, for example. The drive device 140 can also be equipped with a traction motor instead of or together with the internal combustion engine.

[0104] The power transmission mechanism 141 transmits the power generated by the engine 111 to various devices performing the harvesting operation. The devices performing the harvesting operation are the cutting device 103, the conveying device 104, the threshing device 105, the discharging device 107, the discharged straw processing device 108, and the reel 109, etc. The agricultural machine 100 can also be equipped with a power source (such as an electric motor) that provides power to at least one of these devices performing the harvesting operation separately from the engine 111.

[0105] The lamp 142 is a device for illuminating the surroundings of the agricultural machine 100, such as a headlight or a work lamp, for example. Multiple lamps 142 can be mounted on the agricultural machine 100. The lamp 142 includes one or more light sources. Each light source can be a light-emitting diode (LED), a halogen lamp, or a xenon lamp, for example.

[0106] The vehicle speed sensor 151 is a sensor that measures the traveling speed of the agricultural machine 100. The vehicle speed sensor 151 measures the rotational speed of the wheels or axles, for example, and calculates the vehicle speed based on the measured value. The steering angle sensor 152 is a sensor that measures the steering angle of the steering wheel. The illuminance sensor 153 is a sensor that measures the illuminance of the surrounding environment and can be configured outside or inside the cabin of the agricultural machine 100.

[0107] The storage device 164 includes, for example, one or more storage media such as flash memory or a magnetic disk. The storage device 164 stores various data generated by the GNSS unit 120, the laser sensor 125, the camera 126, the millimeter-wave radar 127, the sensor group 150, and the ECUs 165, 166, 167. Among the data stored in the storage device 164, there may be included map data of the area of the farmland where the agricultural machine 100 performs agricultural operations and data of the target path for autonomous driving.

[0108] The ECU 165 controls the overall operation of the agricultural machine 100. The ECU 165 controls the operation of the agricultural machine 100 by controlling the engine 111, the transmission 112, the traveling device 102, the power transmission mechanism 141, etc. included in the driving device 140.

[0109] The ECU 166 performs calculations and controls for realizing autonomous driving based on the data output from the GNSS unit 120, the laser sensor 125, the camera 126, the millimeter-wave radar 127, and the sensor group 150. For example, the ECU 166 determines the position and orientation of the agricultural machine 100 based on the data output from the GNSS unit 120. During autonomous driving, the ECU 166 performs calculations required for the agricultural machine 100 to travel along the set target path based on the position and orientation of the agricultural machine 100. The ECU 166 may also execute a process of generating a target path from the starting point to the destination of the movement of the agricultural machine 100.

[0110] The ECU 167 performs a process of detecting a specific object (such as a person) located around the agricultural machine 100 based on the image acquired by the camera 126. The ECU 167 in the present embodiment is an edge computing device mounted on the agricultural machine 100, but at least a part of the functions of the ECU 167 may also be executed by an external computer (such as a server computer on the cloud) of the agricultural machine 100.

[0111] Through the operations of these ECUs, the control system 160 realizes autonomous driving and the crop harvesting operation. During autonomous driving, the control system 160 controls the driving device 140 based on the measured position and orientation of the agricultural machine 100 and the target path. Thereby, the control system 160 can make the agricultural machine 100 travel along the target path.

[0112] The multiple ECUs included in the control system 160 can communicate with each other according to a vehicle bus standard such as CAN (Controller Area Network), for example. A faster communication method such as in-vehicle Ethernet (registered trademark) may be used instead of CAN. Figure 3In this case, each of the ECUs 165, 166, and 167 is shown as a separate block, but their respective functions can also be implemented by multiple ECUs. An in-vehicle computer integrating at least part of the functions of the ECUs 165, 166, and 167 can also be provided. The control system 160 can also include ECUs other than the ECUs 165, 166, and 167, and any number of ECUs can be provided according to functions. Each ECU includes a processing circuit including one or more processors.

[0113] In Figure 5 the example shown, the camera 126 functions as the Figure 1 imaging device 10 shown. The ECU 167 functions as the Figure 1 image processing device 20 shown. The combination of the ECU 165 and the ECU 166 functions as the Figure 1 control device 30 shown.

[0114] The communication device 190 is a device including a circuit for communicating with an external device. The communication device 190 includes a circuit for performing wireless communication. The communication device 190 can include, for example, an antenna and a communication circuit for transmitting and receiving signals via a network between an external terminal device or an external server computer. The network can include, for example, a cellular mobile communication network such as 3G, 4G, or 5G and the Internet. The communication device 190 can also have a function of communicating with a portable terminal used by a monitor located near the agricultural machine 100. Between such a portable terminal, any wireless communication standard-compliant communication such as Wi-Fi (registered trademark), 3G, 4G, or 5G cellular mobile communication or Bluetooth (registered trademark) (Bluetooth) can be performed.

[0115] The terminal monitor 131 is a terminal for a user to perform operations related to the travel and work of the agricultural machine 100, and is also called a virtual terminal (VT). The terminal monitor 131 can include a display device such as a touch screen and / or one or more buttons. The display device can be, for example, a liquid crystal or an organic light-emitting diode (OLED) display. The user can perform various operations such as switching on (ON) / off (OFF) of the automatic driving mode, recording or editing of map data of farmland, setting of a target path, setting of a crop type, and setting of a work type by operating the terminal monitor 131. At least a part of these operations can also be realized by operating the operation switch group 132. The terminal monitor 131 can also be configured to be detachable from the agricultural machine 100. A user located away from the agricultural machine 100 can also operate the detached terminal monitor 131 to control the operation of the agricultural machine 100. The user can also operate a computer installed with required application software to control the operation of the agricultural machine 100 instead of the terminal monitor 131.

[0116] The operation switch group 132 includes a plurality of switches for operating the agricultural machine 100. In this specification, "switch" broadly means devices such as levers, pedals, and buttons that are used by the driver during operation. The operation switch group 132 may include, for example, a switch for switching between the autonomous driving mode and the manual driving mode, a switch for switching between forward and reverse, an accelerator pedal, a brake pedal, a lever for switching gears, a switch for switching the on / off of a light, etc.

[0117] [2. Operation]

[0118] Next, the operation of the agricultural machine 100 will be described.

[0119] Figure 6 FIG. is an example of a path when the agricultural machine 100 travels while performing an operation of harvesting crops in the farmland 70. In the present embodiment, the agricultural machine 100 travels in the farmland 70 in an autonomous driving manner while harvesting crops. In the farmland 70, the agricultural machine 100 performs an operation of harvesting crops while traveling along a set target path 74. The ECU 166 for autonomous driving control performs steering control of the agricultural machine 100 to eliminate the deviation between the position and orientation of the agricultural machine 100 determined based on the data output from the GNSS unit 120 and the position and orientation of the target path 74. Thereby, the agricultural machine 100 can travel along the target path 74.

[0120] In Figure 6 the example shown, the farmland 70 includes an operation area 71 where the agricultural machine 100 performs an operation of harvesting crops and a turning area 72 (Japanese: makiba) near the outer peripheral edge of the farmland 70. Map data indicating which area in the farmland 70 corresponds to the operation area 71 and which area corresponds to the turning area 72 (also referred to as "farmland map (map)") and the target path 74 can be determined by the ECU 166 for autonomous driving control. For example, if the agricultural machine 100 travels on the outermost path 73 of the operation area 71 in a manual driving manner while performing cutting, the ECU 166 generates a farmland map based on the trajectory of the path 73 and generates a target path 74 for autonomous driving inside the path 73. Thereby, starting from the second week, the agricultural machine 100 can perform operation travel automatically (for example, in an unmanned manner) based on the autonomous driving control of the ECU 166. The agricultural machine 100 automatically travels along Figure 6 the shown target path 74. In addition, Figure 6The target path 74 shown is just an example, and the method for determining the target path 74 is arbitrary. In addition, the method for generating the farmland map and the target path 74 is not limited to the above method. For example, the user may also be enabled to set the farmland map and the target path 74 by operating the terminal monitor 131.

[0121] In the agricultural machine 100 of the present embodiment, during the operation and travel, an operation of detecting an obstacle using the camera 126 and the millimeter-wave radar 127 is performed. The camera 126 is mainly used to detect specific objects such as people. The millimeter-wave radar 127 is mainly used to detect metal objects such as other vehicles. In the present embodiment, the ECU 167 can detect a person with high accuracy in the farmland 70 where crops exist by performing image processing based on the captured image obtained by the camera 126. The control system 160 in the present embodiment is configured to detect people and vehicles, but does not react to obstacles such as birds that have a relatively low impact on the operation and travel.

[0122] Figure 7 It is a diagram schematically showing a situation where an object 76 (a person in this example) is detected using the camera 126 mounted on the agricultural machine 100. Figure 7 The symbols F, B, R, and L shown respectively represent front, rear, right, and left. Figure 7 The dashed line in shows an example of the range captured by the camera 126. As Figure 7 shown, when there is an object 76 in the traveling direction of the agricultural machine 100, the ECU 167 detects the object 76 from the image captured by the camera 126 and sends a signal indicating the presence of the object 76 to the ECU 165. For example, the ECU 167 may also calculate the position of the object 76 in the coordinate system fixed to the ground based on the position of the object 76 in the image, the position of the camera 126 in the agricultural machine 100, and the orientation information, and send a signal indicating the presence of the object 76 to the ECU 165 when the distance between this position and the agricultural machine 100 or the camera 126 is less than a threshold value. In addition, the measured value of other distance measuring sensors such as the laser sensor 125 may also be used in the determination of the distance. However, in the agricultural machine 100 that can be used for purposes such as harvesting relatively tall crops (such as rice) as in the present embodiment, in the method using other distance measuring sensors such as the laser sensor 125, it is sometimes impossible to accurately detect an object 76 such as a person located in the crops. Even in such a case, by using the captured image, the distance to the object 76 can be estimated more accurately.

[0123] If the ECU 165 receives a signal indicating the presence of an object, it controls the drive device 140 to stop the running of the agricultural machine 100 and makes the buzzer 133 emit a warning sound. Additionally, even when a vehicle or other obstacle is detected within a specified distance from the agricultural machine 100 based on the signal output from the millimeter-wave radar 127, the ECU 165 also stops the running of the agricultural machine 100 and makes the buzzer 133 emit a warning sound. Through such control, a collision between the agricultural machine 100 and the obstacle can be avoided.

[0124] The ECU 167 in the present embodiment corresponds to Figure 1 the image processing device 20 shown. That is, the ECU 167 has Figure 2 the structure shown and executes Figure 3 the actions shown. The image processing device 20 can detect an object 76 such as a person with high accuracy by appropriately extracting a partial image to be input to the learned model 27 from the image of the camera 126 with a wide field of view according to the operation state of the agricultural machine 100. Hereinafter, this process will be described in more detail.

[0125] Figure 8 is a diagram schematically showing the flow of the process executed by the ECU 167 (i.e., the image processing device). In Figure 8 the example shown, the ECU 167 first acquires the captured image obtained by the camera 126 (step S801). Next, the ECU 167 determines the region of interest (ROI) that is the object of the detection process for the object in the captured image (step S802). The ECU 167 determines the ROI with reference to various information indicating the operation state of the agricultural machine 100. In Figure 8 the example shown, the ECU 167 also determines the ROI with reference to the map data (farmland map) including the area of the farmland where agricultural operations are performed by the agricultural machine 100. The information indicating the operation state of the agricultural machine 100 may include, for example, information indicating the steering angle or steering operation value measured by the steering angle sensor 152, information indicating the speed measured by the vehicle speed sensor 151, and information indicating the inclination amount of the agricultural machine 100 measured by the IMU 123 (i.e., the inclination sensor). The ECU 167 determines the position and size of the ROI in the captured image based on at least a part of this information.

[0126] The ECU 167 cuts out from the captured image ( )(Part image corresponding to the determined region of interest (step S803). The ECU 167 generates an input image by performing prescribed preprocessing (such as resizing, normalizing, noise removal, etc.) on the part image. The ECU 167 performs processing to detect a specific object by inputting the input image into an AI model (i.e., a learned model), and outputs a signal representing the detection result to the ECU 165.)

[0127] Hereinafter, Figure 8 several specific examples of the processing shown will be described.

[0128] Figure 9A Fig. shows an example of a captured image obtained by the camera 126. When the agricultural machine 100 travels during operation, the camera 126 repeatedly obtains Figure 9A the captured image shown by shooting at a prescribed frame rate. In Figure 9A the captured image shown, an object 76 (a person in this example) is captured. The ECU 167 extracts a part image representing the region of interest determined based on the operation state of the agricultural machine 100 from such a captured image.

[0129] Figure 9B Fig. shows an example of a region of interest 77 that can be selected when the agricultural machine 100 turns right. Figure 9C Fig. shows Figure 9B the part image corresponding to the region of interest 77 shown. When the agricultural machine 100 turns right, as shown in Figure 9B Fig., the ECU 167 determines the right - hand - side region in the captured image as the region of interest 77, and as shown in Figure 9C Fig., extracts the part image representing the region of interest 77. The ECU 167 generates an input image to be input to the learned model 27 by performing preprocessing such as reducing the number of pixels (resizing) on the Figure 9C part image shown. The ECU 167 detects the object 76 by inputting the generated input image into the learned model 27. As described above, any object detection algorithm such as SSD, YOLO, or R - CNN can be used in this object detection process.

[0130] Figure 9D Fig. shows an example of a region of interest 77 that can be selected when the agricultural machine 100 turns left. Figure 9E Fig. shows Figure 9D the part image corresponding to the region of interest 77 shown. Contrary to the above example, when the agricultural machine 100 turns left, the ECU 167 extracts a part image with the left - hand - side region in the captured image as the region of interest 77.

[0131] In this way, in Figures 9B to 9EIn the example shown, when the agricultural machine 100 turns right, the ECU 167 moves the region of interest 77 to the right, and when the agricultural machine 100 turns left, the ECU 167 moves the region of interest 77 to the left. Through such processing, it is possible to more accurately detect the object 76 that may affect the traveling of the agricultural machine 100 during a right turn or a left turn.

[0132] In the prior art, a relatively small-sized input image (e.g., 300×300 pixels) generated by preprocessing a relatively large-sized captured image (e.g., 1280×960 pixels) shown is input to the AI model. In this case, the clarity of the image is greatly reduced, so particularly in the distance or at the edges of the image, there may be a problem of reduced detection accuracy of the object. Figure 9A In contrast, in the present embodiment, first, a partial image corresponding to the region of interest 77 where an object that may affect the operation traveling of the agricultural machine 100 may exist is cut out from the captured image, and an input image generated by preprocessing this partial image is input to the learned model. The number of pixels of the input image is the same as in the prior example (e.g., 300×300 pixels), but since the range captured in the input image is narrowed, a reduction in the clarity of the image can be suppressed. Therefore, the detection accuracy of the object can be improved.

[0133] The number of pixels of the region cut out as the partial image depends on the operation state of the agricultural machine 100, but for example, it can be 2 / 3 or less, 1 / 2 or less, 1 / 3 or less, or 1 / 4 or less of the number of pixels of the original captured image.

[0134] When performing the above processing, it is also possible that the larger the steering angle during a turn of the agricultural machine 100, the greater the ECU 167 moves the region of interest 77. In other words, it is also possible that the greater the increase in the magnitude of the steering angle during a right turn, the greater the movement of the region of interest 77 to the right, and the greater the increase in the magnitude of the steering angle during a right turn, the greater the movement of the region of interest 77 to the left. The ECU 167 can obtain information on the steering angle at that moment from the measured value of the steering angle sensor 152. Data such as a table showing the relationship between the magnitude of the steering angle during a right turn or a left turn and the movement amount of the region of interest 77 in the image can be stored in advance in the memory 24 or the storage device 164. The ECU 167 can determine the movement amount based on this data and the measured steering angle. In

[0135] In the example shown, the longitudinal dimension of the cut-out partial image is the same as the longitudinal dimension of the original captured image, and the lateral dimension of the partial image is smaller than the lateral dimension of the captured image. Not limited to such a cutting method, a region smaller than the captured image can also be cut out as the partial image for the longitudinal direction. Figures 9B to 9E In the example shown, the longitudinal dimension of the cut-out partial image is the same as the longitudinal dimension of the original captured image, and the lateral dimension of the partial image is smaller than the lateral dimension of the captured image. Not limited to such a cutting method, a region smaller than the captured image can also be cut out as the partial image for the longitudinal direction.

[0136] The region of interest 77 is not limited to the turning state of the agricultural machine 100, and can also be determined based on other states. For example, the region of interest 77 can also be determined based on the traveling speed of the agricultural machine 100.

[0137] Figure 10A An example of the region of interest 77 that can be selected when the agricultural machine 100 is moving forward at a relatively high speed is shown. Figure 10B is shown in connection with Figure 10A A partial image corresponding to the region of interest 77 shown is shown. In this example, the ECU 167 changes the size of the region of interest 77 according to the traveling speed of the agricultural machine 100. Specifically, the higher the traveling speed of the agricultural machine 100, the smaller the ECU 167 makes the region of interest 77, and the lower the traveling speed of the agricultural machine 100, the larger the ECU 167 makes the region of interest 77. When the agricultural machine 100 is moving forward at a high speed, it is required to accurately detect an object 76 farther away. Therefore, the higher the traveling speed, the smaller the ECU 167 makes the region of interest 77 to improve the detection accuracy of the object 76 in the distance. Not only does the size of the region of interest 77 change according to the moving speed of the agricultural machine 100, but the position of the region of interest 77 can also change according to the moving speed. For example, it can also be that the higher the moving speed, the more the position of the region of interest 77 is moved upward. By such processing, it becomes easier to detect an object in the distance when the agricultural machine 100 is traveling at a high speed. The ECU 167 obtains information on the traveling speed at that moment from the measured value of the vehicle speed sensor 151. Data such as a table representing the relationship between the traveling speed and the size and / or position of the region of interest 77 in the image can be pre-stored in the memory 24 or the storage device 164. The ECU 167 can determine the size and / or position of the region of interest 77 based on this data and the measured traveling speed.

[0138] The region of interest 77 can also be determined based on the tilt amount of the agricultural machine 100. The tilt amount of the agricultural machine 100 refers to, for example, the pitch angle (i.e., the rotation angle about the axis in the left-right direction) or the roll angle (i.e., the rotation angle about the axis in the front-back direction), etc., which is the magnitude of the tilt angle relative to the reference attitude. Here, the reference attitude refers to the attitude of the agricultural machine 100 when it is located on a horizontal ground. The ECU 167 can also move the region of interest up or down according to the tilt amount such as the pitch angle measured by the IMU 123. The pitch angle takes a positive value when the agricultural machine 100 is tilted upward relative to the horizontal plane as when going uphill, and takes a negative value when the agricultural machine 100 is tilted downward relative to the horizontal plane as when going downhill. When the agricultural machine 100 is tilted upward (i.e., the pitch angle takes a positive value), the region of interest in the captured image can also be moved downward. On the contrary, when the agricultural machine 100 is tilted downward (i.e., the pitch angle is negative), the region of interest can also be moved upward. Thereby, it is possible to avoid a lot of sky or ground being included in the region of interest and improve the detection performance of the object. It can also be that the larger the absolute value of the pitch angle, the larger the movement amount of the region of interest by the ECU 167. In this case, the data representing the relationship between the absolute value of the pitch angle and the movement amount is stored in the memory 24 or the storage device 164. The ECU 167 can determine the movement amount of the region of interest based on this data and the measurement value of the IMU 123.

[0139] The ECU 167 can also determine the region of interest based on the map data of the farmland. For example, the ECU 167 can also determine the region of interest based on the map data, the positioning data output from the GNSS unit 120, and the operation state of the agricultural machine 100.

[0140] Figure 11 FIG. shows an example of the agricultural machine 100 traveling on the outermost periphery of the operation area 71 in the farmland. In this example, there is a person as the object 76 outside the operation area 71 in the farmland. Since the agricultural machine 100 during operation does not travel outside the operation area 71, such an object 76 does not need to be detected as an obstacle. Therefore, the ECU 167 can also determine the area corresponding to the operation area 71 where agricultural operations are performed in the farmland from the captured image based on the map data and the positioning data, and determine the region of interest from this area based on the operation state of the agricultural machine 100.

[0141] Here, an example of a method for determining the area corresponding to the operation area 71 from the captured image will be described.

[0142] Figure 12It is a perspective view schematically showing the configuration relationship of the camera coordinate system Σc fixed to the camera 126, the vehicle coordinate system Σv fixed to the agricultural machine 100, the world coordinate system Σw fixed to the ground, and the reference plane Re extending parallel to the horizontal plane. The camera coordinate system Σc has mutually orthogonal Xc axis, Yc axis, and Zc axis. The vehicle coordinate system Σv has mutually orthogonal Xv axis, Yv axis, and Zv axis. The world coordinate system Σw has mutually orthogonal Xw axis, Yw axis, and Zw axis. In Figure 12 In the example of, the Xw axis and the Yw axis of the world coordinate system Σw are located on the reference plane Re. The camera 126 is fixed to the agricultural machine 100. Therefore, the position and orientation of the camera coordinate system Σc relative to the vehicle coordinate system Σv are fixed in a known state. The camera coordinate system Σc is tilted such that its Zc axis intersects the reference plane Re obliquely. When the agricultural machine 100 does not rotate in the pitch and roll directions, the plane containing the Xv axis and the Yv axis of the vehicle coordinate system Σv is parallel to the reference plane Re.

[0143] At a position that is at a distance equal to the focal length of the camera 126 from the origin O of the camera coordinate system Σc in the Zc axis direction, there is a hypothetical image plane Im. The image plane Im is orthogonal to the Zc axis and the optical axis λ of the camera 126. The pixel positions on the image plane Im are defined by an image coordinate system having mutually orthogonal u axis and v axis. For example, let the coordinates of points P1 and P2 located on the reference plane Re in the world coordinate system Σw be (X1, Y1, Z1) and (X2, Y2, Z2), respectively. In Figure 12 In the example of, the Xw axis and the Yw axis of the world coordinate system Σw are located on the reference plane Re. Therefore, Z1 = Z2 = 0.

[0144] Points P1 and P2 on the reference plane Re are respectively converted into points p1 and p2 on the image plane Im of the camera 126 through perspective projection of the pinhole camera model. In the image plane Im, points p1 and p2 are respectively located at pixel positions represented by the coordinates (u1, v1) and (u2, v2).

[0145] If the configuration relationship of the camera coordinate system Σc with respect to the reference plane Re in the world coordinate system Σw is provided, then through a homography transformation, it is possible to obtain the corresponding point (X, Y, 0) on the reference plane Re based on any point (u, v) on the image plane Im. If the coordinates of a point are represented in a homogeneous coordinate system, such a homography transformation is defined by a 3-row × 3-column transformation matrix H. When the coordinates of a point on the image plane Im of the camera 126 are (u, v, 0), the coordinates (X, Y, 0) of the corresponding point on the reference plane Re are associated with the point (u, v, 0) through the homography transformation matrix H as shown in the following equation (1).

[0146] [Equation 1]

[0147]

[0148] The content of the transformation matrix H depends on the configuration relationship of the camera coordinate system Σc with respect to the reference plane Re in the world coordinate system Σw. If the position of the reference plane Re changes, the content of the transformation matrix H also changes. The reference plane Re can be set to be in contact with the ground or at a height of a specified distance from the ground. When the object to be detected is a person, for example, the reference plane Re can also be set to a height of more than 1 meter and less than 2 meters from the ground.

[0149] By using such a homography transformation, it is possible to associate the coordinates of any point on the image plane Im of the camera 126 with the coordinates of a point on the reference plane Re.

[0150] During the operation of the agricultural machine 100, the ECU 167 can obtain the position and attitude of the agricultural machine 100 based on the positioning data output from the GNSS unit 120. In addition, the configuration relationship between the vehicle coordinate system Σv and the camera coordinate system Σc is known. Therefore, the ECU 167 can obtain the configuration relationship between the world coordinate system Σw and the camera coordinate system Σc based on the positioning data and can determine the transformation matrix H based on this configuration relationship.

[0151] The ECU 167 can determine which pixel region in the captured image corresponds to the working area 71 based on the result of the operation of Equation (1) and the position information of the working area included in the map data of the farmland. The ECU 167 can be configured to exclude the pixel regions corresponding to the outside of the working area 71 in the captured image from the objects to be detected, and determine the region of interest from the pixel regions corresponding to the working area 71. Thereby, it is possible to avoid the agricultural machine 100 from stopping due to detecting an object 76 that does not affect the working travel of the agricultural machine 100. In addition, the ECU 167 may instead of determining the region of interest from the pixel regions corresponding to the working area 71 in the captured image, determine the region of interest from the pixel regions corresponding to the unworked area in the working area 71. This is because even if there is an object 76 in the completed working area, it will not hinder the travel of the agricultural machine 100, so the completed working area can also be excluded from the detected objects.

[0152] (Embodiment 2)

[0153] Next, an agricultural machine according to an exemplary second embodiment of the present disclosure will be described.

[0154] The agricultural machine in this embodiment is the same as the agricultural machine in Embodiment 1, and includes a machine body, a photographing device, and an image processing device that detects a specific object based on an image obtained by the photographing device (image acquisition device). The image processing device performs the following steps (S21)-(S23).

[0155] (S21) Detect the boundary between the sky and the ground objects other than the sky and the object from the image.

[0156] (S22) Estimate the inclination of the machine body based on the position of the boundary between the sky and the ground objects other than the sky in the image.

[0157] (S23) Estimate the position of the object in the coordinate system fixed to the ground based on the position of the object in the image and the estimated inclination of the machine body.

[0158] Through such operations, it is possible to estimate the inclination of the machine body and the position of the object based on the image without using a sensor for measuring the inclination of the machine body. Hereinafter, the detailed information of the agricultural machine in this embodiment will be described.

[0159] The agricultural machine in this embodiment has the same structure as the Figure 1 or Figure 5 shown agricultural machine 100. In this embodiment, the image processing device 20 (for example, Figure 5 the shown ECU 167) also has Figure 2The structure shown. If a specific object is detected based on the image acquired by the imaging device 10, the image processing device 20 in the present embodiment estimates the position of the object in the coordinate system fixed to the ground (the aforementioned world coordinate system Σw) and the distance from the agricultural machine 100 to the object based on the position of the object in the image. As will be described with reference to Figure 12 As described, this estimation process is performed by coordinate transformation from the image coordinate system to the world coordinate system Σw. Coordinate transformation requires information on the configuration relationship of the camera coordinate system Σc with respect to the reference plane Re in the world coordinate system Σw. Therefore, during the travel of the agricultural machine 100, it is sought to use a sensor (such as Figure 5 shown IMU 123, etc.) that can measure the inclination of the agricultural machine 100 to sequentially estimate the attitude of the imaging device 10 in the agricultural machine 100, and use the information on this attitude to determine the transformation matrix H in the above formula (1).

[0160] However, in the case of using the measurement values of sensors such as the IMU 123, there is a problem that it is difficult to obtain the time ( ) synchronization between the measurement values of the sensors and the images (i.e., each frame in the video) generated by the imaging device 10. Sometimes, due to the time lag, it is impossible to accurately determine the attitude of the agricultural machine 100 corresponding to each frame.

[0161] To solve this problem, the image processing device 20 in the present embodiment estimates the inclination (tilt angle) of the agricultural machine 100 based on the captured image acquired by the imaging device 10. Specifically, the boundary between the sky and the ground objects other than the sky is detected from the captured image, and based on the temporal change in the position of this boundary, the tilt angle (for example, the pitch angle) of the agricultural machine 100 is estimated. Thereby, it is possible to more accurately determine the attitude of the agricultural machine 100 corresponding to each frame of the captured image and improve the estimation accuracy of the position or distance of the object.

[0162] In addition, in the present embodiment, the agricultural machine 100 may also be provided with an inclination sensor (such as Figure 5The IMU shown (123). The image processing device 20 may also calculate the difference between the inclination of the agricultural machine 100 estimated based on the position of the boundary between the sky and the ground objects other than the sky in the image and the inclination measured by the inclination sensor. When this difference is less than the threshold value, based on the position of the object in the image and the estimated inclination, the position of the object in the coordinate system fixed to the ground is estimated. In this case, the image processing device 20 may be configured to, when the difference between the inclination of the agricultural machine 100 estimated based on the position of the boundary between the sky and the ground objects other than the sky in the image and the inclination measured by the inclination sensor is above the threshold value, discard the estimation result of the former inclination and not perform the processing based on the inclination. In this way, the measurement result of the inclination sensor can also be used to evaluate the reliability of the inclination estimated based on the image.

[0163] Hereinafter, while referring to Figure 13 , a more specific example of the operation of the image processing device 20 in the present embodiment will be described.

[0164] Figure 13 is a flowchart showing an example of the image processing executed by the image processing device 20 in the present embodiment. The image processing device 20 repeatedly executes the processing shown in steps S210 to S270 during the operation of the agricultural machine 100.

[0165] In step S210, the image processing device 20 acquires the image generated by the imaging device 10.

[0166] In step S220, the image processing device 20 executes the processing of detecting the boundary between the sky and the ground objects other than the sky and a specific object (such as a person) from the acquired image. The boundary between the sky and the ground objects other than the sky may be, for example, the boundary between the sky and the ground (horizon) or the boundary between the sky and the mountains. The boundary between the sky and the ground objects can be detected, for example, by various methods such as hue analysis, edge detection, and / or texture analysis of the image. The processing of detecting a specific object can be executed, for example, by the same method as the processing in steps S120 to S140 shown in Figure 3 . In addition, a learned model for detecting the boundary between the sky and the ground objects and a specific object from the image may be used to detect both at once. Further, in the present embodiment, the extraction processing of partial images in step S120 may be omitted. That is, the image processing device 20 may also detect a specific object and / or the boundary between the sky and the ground objects by inputting the input image generated by performing predetermined preprocessing on the captured image into the learned model.

[0167] Figure 14This is a diagram showing an example of the boundary between the sky and the ground objects detected in the captured image. In this example, the boundary 78 between the sky and the mountains is represented by a thick curve. The image processing device 20 can be configured to perform the process of detecting such a boundary 78 for each frame of the image (video) generated by the imaging device 10.

[0168] In the next step S230, the image processing device 20 determines whether an object is detected from the image. If an object is detected, the process proceeds to step S240. If no object is detected, the process returns to step S210.

[0169] In step S240, the image processing device 20 estimates the inclination of the body of the agricultural machine 100 based on the position of the boundary 78 between the sky and the ground objects in the image. For example, the image processing device 20 estimates the inclination of the body based on the position of the boundary 78 in the image and the displacement amount from the position of the boundary 78 detected in the image acquired by the imaging device 10 at a past time. The past time ( ) can be a time when the inclination measured by the inclination sensor is less than the reference value. The image processing device 20 estimates, for example, the pitch angle of the body as the inclination of the body. More specifically, the image processing device 20 can also estimate the inclination angle from the reference posture of the agricultural machine 100 based on the temporal change in the position of the boundary 78 detected for each frame of the video generated by the imaging device 10. The reference posture can be the posture of the agricultural machine 100 when the agricultural machine 100 is located on a flat ground. The image processing device 20 can use the position of the boundary 78 in the image frame acquired when the agricultural machine 100 is in the reference posture as a reference, and estimate the pitch angle of the agricultural machine 100 based on how many pixels the boundary 78 has been displaced up and down in the image from this position. For example, the image processing device 20 can estimate the pitch angle through the following process.

[0170] (1) Based on the current frame, determine the corresponding point for each of a plurality of pixels (or a part of the pixels such as the central part) arranged along the boundary 78 in the reference frame acquired when the agricultural machine 100 is in the reference posture.

[0171] (2) Calculate the displacement amount from the position in the reference frame for each corresponding point.

[0172] (3) Calculate the average value of the displacement amounts of the plurality of corresponding points as the displacement amount of the boundary 78.

[0173] (4) Calculate the pitch angle based on the displacement amount of the boundary 78.

[0174] The correspondence relationship between the up-and-down displacement amount of the boundary 78 in the image and the pitch angle can be stored in advance, for example, in Figure 2In the memory 24 shown. The image processing device 20 can estimate the pitch angle based on this correspondence. In addition, the image processing device 20 can also estimate the roll angle by obtaining the rotation angle between frames of the boundary 78 based on the temporal change of the position of the boundary 78 in the image. By estimating the roll angle in addition to the pitch angle, the attitude of the photographing device 10 can be estimated more accurately.

[0175] In step S250, the image processing device 20 estimates the position of the object in the coordinate system fixed to the ground (world coordinate system) based on the position of the object in the image and the estimated inclination of the aircraft body. Specifically, the image processing device 20 determines the transformation matrix H in the above formula (1) based on the estimated inclination of the aircraft body, the known arrangement relationship of the photographing device 10 relative to the aircraft body, and the position and orientation of the aircraft body obtained from a positioning device (for example Figure 5 the GNSS unit 120 in). By performing the operation of formula (1) on the position of the object in the image coordinate system (for example, the position of the bounding box), the position of the object in the world coordinate system can be calculated.

[0176] In step S260, the image processing device 20 estimates the distance from the agricultural machine 100 to the object based on the position of the object in the world coordinate system. The position of the agricultural machine 100 in the world coordinate system is measured by a positioning device such as the GNSS unit 120. The image processing device 20 calculates the distance between the position of the agricultural machine 100 and the position of the object in the world coordinate system. Here, the "position" of the agricultural machine 100 is, for example, the position of a specific part in the agricultural machine 100 such as the position of the GNSS unit 120 that is determined in advance.

[0177] In step S270, the image processing device 20 sends information indicating the distance estimated in step S260 to the control device 30. The control device 30 that receives this information determines whether this distance is less than a specified value, and in the case where this distance is lower than the specified value, performs control such as stopping the movement of the agricultural machine 100 or sounding a warning tone from a buzzer.

[0178] In this way, the image processing device 20 detects the boundary between the sky and the ground objects other than the sky from the image obtained by the photographing device 10, and estimates the inclination of the aircraft body based on the temporal change of the position of this boundary. Based on the estimated inclination, the position coordinates of the object in the image are converted into the position coordinates in the coordinate system fixed to the ground, and based on the converted position coordinates, the distance to the object is estimated. The control device 30 controls the operation of the agricultural machine 100 based on the estimated distance.

[0179] In addition, Figure 13The process of step S260 shown can also be performed by the control device 30 instead of the image processing device 20. In this case, the image processing device 20 can be configured to send information representing the position coordinates of the object in the world coordinate system to the control device 30 after step S250. In this case, the control device 30 controls the operation of the agricultural machine based on the position of the object in the world coordinate system. The control device 30 can be configured to perform at least one of stopping the agricultural machine 100, decelerating the agricultural machine 100, and outputting a warning when the distance estimated based on the position of the object in the world coordinate system is less than a specified value.

[0180] Figure 15 is a flowchart showing a modification example of the present embodiment. In Figure 15 the example, it is the image processing device 20 instead of the control device 30 that determines whether the distance from the agricultural machine 100 to the object is less than a specified value. Figure 15 The example of Figure 14 differs from the example of

[0181] Figure 16 in that step S265 is added after step S260, and step S270 is replaced with step S275. In step S265, the image processing device 20 determines whether the distance from the agricultural machine 100 to the object is less than a specified value. If the determination result is Yes, the process proceeds to step S275. If the determination result is No, the process returns to step S210. In step S275, the image processing device 20 sends a signal indicating the presence of the object to the control device 30. If the control device 30 receives this signal, it performs controls such as stopping the movement of the agricultural machine 100 or sounding a warning tone from a buzzer. Figure 16 the example shown, the inclination (e.g., pitch angle) measured by an inclination sensor (e.g., Figure 5 the IMU123 shown) is also used. Figure 16 The example of Figure 14 differs from the example of

[0182] In step S245, the image processing device 20 determines whether the difference between the inclination of the vehicle body estimated in step S240 and the inclination measured by the inclination sensor is less than a threshold value. The inclination angles to be compared here may be, for example, the pitch angle or the roll angle. It is also possible to compare the value obtained by averaging the inclination measured by the inclination sensor over the entire specified time (for example, on the order of 0.1 second to several seconds) with the inclination of the vehicle body estimated in step S240. When the difference between the two inclinations is less than the threshold value, the process proceeds to step S250, and then the same processing as in the example of Figure 14 is performed. When the difference between the two inclinations is equal to or greater than the threshold value, the process proceeds to step S255. The fact that the difference between the two inclinations is equal to or greater than the threshold value means that there is a possibility that the reliability of the inclination of the vehicle body estimated based on the image is low. Therefore, in such a case, the image processing device 20 estimates the position of the object in the coordinate system fixed to the ground based on the inclination measured by the inclination sensor rather than the inclination estimated based on the image. After step S255, the process proceeds to step S260.

[0183] In addition, step S270 in the example of Figure 16 can be replaced with steps S265 and S275 in Figure 15 . In this case, it is the image processing device 20 rather than the control device 30 that determines whether the distance from the agricultural machine 100 to the object is less than a specified value.

[0184] (Embodiment 3)

[0185] Next, an agricultural machine according to an exemplary third embodiment of the present disclosure will be described.

[0186] The agricultural machine in this embodiment, like the agricultural machine in Embodiment 1, includes an imaging device, an image processing device that detects a specific object from an image obtained by the imaging device, and a control device that controls the operation of the agricultural machine based on the detection result of the object. In this embodiment, the image processing device is different from that in Embodiment 1 in that it includes a memory that stores a plurality of learned models corresponding to the environment around the agricultural machine, and switches and uses those models according to the environment. The image processing device performs the following steps (S31)-(S33).

[0187] (S31) Generate an input image based on the image obtained by the imaging device.

[0188] (S32) Select one learned model from the plurality of learned models according to the environment around the agricultural machine.

[0189] (S33) Detect the object by inputting the input image into the selected learned model.

[0190] By such an operation, it is possible to use an appropriate learned model corresponding to the environment around the agricultural machine to detect an object from a captured image with higher accuracy.

[0191] The environments around agricultural machines are diverse. For example, if environmental conditions such as time period, weather, type of crop to be worked on, and type of farmland change, the characteristics of the images acquired by the imaging device (e.g., brightness, saturation, and hue of each pixel) change. Therefore, when a single learned model is used in various environments, the detection performance of the object may decrease depending on the environment. In order to maintain high detection performance in various environments, it is necessary to prepare a large amount of learning data to train the model. However, even so, it is sometimes difficult to handle multiple environments with just one model.

[0192] Therefore, in the present embodiment, the image processing device is configured to prepare in advance a plurality of learned models corresponding to a plurality of environments and use those models separately according to the situation. More specifically, the image processing device generates an input image based on the image acquired by the imaging device, and detects an object by inputting the input image into one learned model selected from a plurality of learned models according to the environment around the agricultural machine. By such an operation, even when the environment around the agricultural machine changes, it is possible to use an appropriate learned model corresponding to the environment to detect an object from the image with relatively high accuracy.

[0193] The agricultural machine in the present embodiment has the same structure as the Figure 1 or Figure 5 agricultural machine 100 shown. In the present embodiment, the image processing device 20 (e.g., Figure 5 the ECU 167 shown) also has the same hardware structure as the Figure 2 example shown. However, in the present embodiment, the memory 24 stores a plurality of learned models, which is different from the Figure 2 example.

[0194] Figure 17The figure is a block diagram showing a structural example of the image processing apparatus 20 in the present embodiment. In this example, the memory 24 of the image processing apparatus 20 stores a computer program 25 executed by the processor 22 and a plurality of learned models 27 for detecting an object from an input image. The plurality of learned models 27 are stored in association with various environmental conditions around the agricultural machine 100. For example, a plurality of learned models 27 associated with different luminances and / or different time periods of the environment around the agricultural machine 100 may be stored in the memory 24. The image processing apparatus 20 selects and uses one learned model corresponding to the luminance of the current environment or the current time period from the plurality of learned models 27. The image processing apparatus 20 can determine the luminance of the current environment based on at least one of an image acquired by the imaging device 10 (e.g., a histogram or average luminance of pixel values in the image), an output from an illuminance sensor provided in the agricultural machine 100 (e.g., Figure 5 the illuminance sensor 153 shown), an input from a user (e.g., the type of crop and / or the type of work area input via Figure 5 the terminal monitor 131 shown), and the lighting state of the lamp of the agricultural machine 100. The illuminance sensor may be mounted on the body of the agricultural machine 100 (e.g., near the imaging device 10). In addition, the image processing apparatus 20 can obtain information on the current time period, for example, according to a clock function provided in the processor 22 or a timing circuit such as a real-time clock that can be provided separately from the processor 22.

[0195] In Figure 17 the example shown, the plurality of learned models 27 include daytime models 27A and 27B and a nighttime model 27C. In this case, the image processing apparatus 20 selects the daytime models 27A or 27B during the daytime time period and selects the nighttime model 27C during the nighttime time period. In Figure 17 the example, for the daytime models, two types of models, a non-backlight model 27A and a backlight model 27B, are prepared. In this case, the image processing apparatus 20 may be configured to determine whether it is backlit based on at least one of an image acquired by the imaging device 10, an input from a user, and an output from the illuminance sensor during the daytime time period, select the backlight model 27B in the case of backlighting, and select the non-backlight model 27A in the case of non-backlighting.

[0196] Examples of the models are not limited to Figure 17The example shown. For example, multiple learned models associated with different weathers such as a sunny-day model, a cloudy-day model, and a rainy-day model can also be stored in the memory 24. In this case, the image processing device 20 selects one learned model corresponding to the current weather from those multiple learned models for use. The image processing device 20 can determine the current weather based on the image acquired by the imaging device 10 or weather-related information obtained from an external device (for example, a server computer connected via the Figure 5 communication device 190 and network shown). Alternatively, multiple learned models associated with different types of crops can be prepared. In this case, the image processing device 20 selects one learned model corresponding to the type of crop to be worked on from the multiple learned models 27. The multiple models corresponding to the types of crops can include various models corresponding to the types of crops that are the objects of operations such as harvesting, such as a rice model, a wheat model, and a soybean model. In addition, multiple learned models corresponding to the types of paddy fields, vegetable fields, grasslands, etc., can also be prepared. Depending on the type of crop and the type of work area, the hue and texture of the image acquired by the imaging device 10 may vary greatly. Therefore, by preparing multiple models according to the type of crop or work area that is the object of the operation, the environmental adaptability can be improved. The image processing device 20 can determine the type of crop or the type of work area to be worked on based on the image acquired by the imaging device 10 or the input from the user. Further, different learned models can be created in advance for each combination of any two or more selected from the brightness around the agricultural machine, the time period, the weather, the type of crop, and the type of work area, and those models can be switched and used according to the environmental conditions. By switching such multiple models according to the environment, higher-precision object detection can be performed.

[0197] Figure 18 is a table showing an example of the correspondence between the multiple learned models stored in the memory 24 and the environment. In this example, different multiple learned models A to L are stored in the memory 24 according to the combination of three items: whether the time period is the daytime or the nighttime, whether the weather is sunny or cloudy, and whether it is backlit in the case of sunny weather, and the type of crop. Each model is pre-trained with multiple learning data corresponding to the respective environments and is stored in the memory 24. The correspondence between the model and the environment is not limited to Figure 18 the example shown and can be variously changed. For example, not only for day and night, but also a model corresponding to evening can be created. Or, models corresponding to other weathers such as rain or snow can also be created. In Figure 18In the example, the types of crops are three types, namely Crop A, Crop B, and Crop C, but it can also be one type, two types, or four or more types. Further, models can also be created for each type of working land such as paddy fields, vegetable fields, and grasslands.

[0198] Each of the multiple learned models 27 can be, for example, a machine learning model trained using a deep learning-based algorithm such as a CNN or a Vision Transformer. Each model has a common model architecture and different weight sets. By determining the combination of weight sets for each model using appropriate learning data and learning software, multiple learned models 27 corresponding to various environments can be created. The multiple learned models 27 can be generated either by the image processing device 20 itself or by an external device such as a server computer. The multiple learned models 27 are stored in the memory 24 before the operation of the agricultural machine 100 starts or before using the model.

[0199] Figure 19 It is a flowchart showing an example of the operation of the image processing device 20 in the present embodiment. In Figure 19 In the example, the image processing device 20 repeatedly executes the operations of steps S310 to S340 during the operation of the agricultural machine 100.

[0200] In step S310, the image processing device 20 generates an input image to be input to the learned model based on the captured image generated by the imaging device 10. The image processing device 20 generates the input image, for example, by performing prescribed preprocessing (such as resizing, normalizing, noise removal, etc.) on the captured image. Alternatively, the image processing device 20 can also generate the input image by performing the Figure 3 processing of steps S110 to S130 shown. By continuing the processing of steps S110 to S130, the area of the detection object can be reduced according to the operation state of the agricultural machine 100, and the detection accuracy of the object is improved.

[0201] In step S320, the image processing device 20 selects one learned model corresponding to the surrounding environment of the agricultural machine 100 from the multiple learned models 27 stored in the memory 24. A specific example of the processing of step S320 will be described later.

[0202] In step S330, the image processing device 20 detects a specific object (such as a person) by inputting the input image into the selected learned model. This step is the same as Figure 3The processing of step S140 is the same. The image processing device 20 detects an object such as a person by inputting an input image into the learned model 27 stored in the memory 24. For example, the image processing device 20 can be configured to output, as a detection result, coordinate information indicating the position of a rectangular frame (bounding box) that encloses the area where the object exists, and information on the width and height of the bounding box, when a specific object exists in the image. Alternatively, the image processing device 20 can also output a signal indicating whether a specific object exists in the input image as a detection result. The image processing device 20 can also determine that an object exists in the input image when the distance from the imaging device 10 to the object calculated based on the position of the bounding box in the input image is less than a threshold value.

[0203] In step S340, the image processing device 20 sends the detection result to the control device 30. This step is the same as the Figure 3 processing of step S150. The image processing device 20 sends the detection result of the object to the control device 30. For example, the image processing device 20 can also send a signal indicating whether a specific object exists in the input image to the control device 30. Alternatively, the image processing device 20 can also send a signal indicating this only when a specific object is detected from the input image to the control device 30. In addition, the image processing device 20 can also send a signal indicating the existence of the object to the control device 30 only when the distance to the detected object is less than a specified value. The distance to the object can be estimated, for example, using the Figure 15 or Figure 16 methods shown.

[0204] If the control device 30 receives a signal indicating the detection result from the image processing device 20, the control device 30 controls the operation of the agricultural machine 100 based on the detection result. For example, the control device 30 can be configured to perform at least one of stopping the agricultural machine 100, decelerating the agricultural machine 100, and outputting a warning to a device such as a buzzer or a display when receiving a signal indicating that an object has been detected. By such an operation, it is possible to avoid a collision between the agricultural machine 100 and the object or to alert the object (such as a person) or the rider of the agricultural machine 100.

[0205] The processing of steps S310 to S340 can be performed by the imaging device 10 (for example Figure 5The shooting by the camera 126) shown is periodically performed. Alternatively, the processes of steps S310 to S340 may be performed for every specific number of frames (for example, 10 frames, 30 frames, 60 frames, etc.) or for every specified time (for example, 0.5 seconds, 1 second, 2 seconds, 3 seconds, etc.). The process of step S320 may also be performed before step S310. In addition, the process of step S320 may be performed at a frequency lower than those of the processes of steps S310, S330, and S340. For example, the process of step S320 may be performed for every specified time (for example, every few seconds, every dozens of seconds, every few minutes, etc.) or every time the agricultural machine 100 makes a direction change. The environment around the agricultural machine 100 does not change in a short period of time, and whether it is backlight can change according to the orientation of the agricultural machine 100. Therefore, it is not necessarily required to select the model at the same frequency as object detection, and the model selection may also be performed at the timing of a direction change such as a turn.

[0206] As Figure 6 shown, the control device 30 of the agricultural machine 100 (for example Figure 5 the ECU 166 shown) may be configured to move the agricultural machine 100 along the set target path 74. The target path 74 is not limited to Figure 6 the example shown. For example, it may also be a path including a round-trip section. The image processing device 20 may also select the learned model every time the agricultural machine 100 makes a direction change or every time the traveling direction is reversed in the round-trip section. The image processing device 20 can obtain a signal indicating that the agricultural machine 100 has made a direction change from the control device 30. The image processing device 20 can select an appropriate model by performing the process of step S320 in response to this signal. In this case, the same model is used for object detection until the next turn.

[0207] Figure 20 is a flowchart showing an example of the operation of the image processing device 20 in the case where the learned model is selected every time the agricultural machine 100 makes a direction change. In Figure 20In the example, the image processing device 20 first selects, in step S320, one learned model corresponding to the surrounding environment of the agricultural machine from among a plurality of learned models. Thereafter, the processes of steps S310, S330, and S340 are executed. After step S340, the process proceeds to step S350, where the image processing device 20 determines whether the agricultural machine 100 has changed its direction. When a signal indicating that the agricultural machine 100 has changed its direction is received from the control device 30, the image processing device 20 determines that the agricultural machine 100 has changed its direction. When a direction change has occurred, the process returns to step S320 to reselect the learned model. When no direction change has occurred, the process returns to step S310, and the previously selected learned model is used to perform the object detection process based on the captured image again.

[0208] Next, with reference to Figure 21 , the model selection process in step S320 will be described in more detail.

[0209] Figure 21 FIG. is a flowchart showing an example of the model selection process in step S320. Step S320 in this example includes steps S321 to S326.

[0210] In step S321, the image processing device 20 acquires information related to the type of the work site and the type of the crop set by the user. Such information can be set, for example, through an input device such as the terminal monitor 131 shown in Figure 5 .

[0211] In step S322, the image processing device 20 acquires information related to the date and time. For example, the image processing device 20 can acquire information related to the date and time based on the clock function of the processor 22 of the image processing device 20 or a real-time clock that can be separately provided from the processor 22.

[0212] In step S323, the image processing device 20 acquires information related to whether the lights of the agricultural machine 100 are on or off. Information related to whether the lights are on or off can be acquired from the control device 30. The user can, for example, switch the on and off states of the light 142 by operating the changeover switch of the light 142 included in the operation switch group 132 shown in Figure 5 . A signal indicating whether the light 142 is on or off can be sent from the control device 30 (e.g., the ECU 165 shown in Figure 5 ) to the image processing device 20.

[0213] In step S324, the image processing device 20 acquires information from an illuminance sensor (e.g., Figure 5The measured value of the illuminance sensor 153) shown. The measured value of the illuminance sensor indicates the degree of the brightness of the environment around the agricultural machine 100.

[0214] In step S325, the image processing device 20 generates a histogram of the pixel values in the image. For example, when the pixel values of each pixel are represented by 256 gray levels (8 bits) from 0 to 255 for each of red, green, and blue, a histogram representing the frequency for each pixel value from 0 to 255 for each color can be generated. The image processing device 20 can evaluate the brightness, hue, saturation, etc. of the entire image based on this histogram. In this step, the image processing device 20 can also, for example, instead of generating a histogram based on the entire image, use the method in Embodiment 2 to detect the area corresponding to the sky from the image and generate a histogram of the area corresponding to the sky.

[0215] In step S326, the image processing device 20 selects the learned model to be used based on the type of the work area, the type of the crop, the date and time, the presence or absence of the lamp being lit, the measured value of the illuminance sensor, and the histogram of the image. The image processing device 20 can estimate whether the time period is day, evening, or night, or whether the weather is sunny, cloudy, rainy, or snowy based on this information or signal, and select a learned model corresponding to those estimation results and the selected type of the work area and the crop.

[0216] In addition, the order of the processes in steps S321 to S325 can also be interchanged with each other. Further, at least one of the processes in steps S321 to S325 can also be omitted. In this case, the image processing device 20 selects the learned model without considering the information that has been omitted. The algorithm for estimating the type of the work area, the type of the crop, the time period, the weather, etc. from the acquired information such as the image is not limited to a specific algorithm and can be arbitrarily designed. An AI model for determining the optimal model based on the acquired information such as the image can also be used.

[0217] As described above, according to the present embodiment, the detection process of the object is performed using the learned model appropriately selected from a plurality of learned models according to the environment. Thereby, compared with the case of using one learned model, the detection performance of the object can be significantly improved. Further, when each model is learned using relatively little learning data, the detection accuracy in each environment can also be improved. For example, the accuracy of image recognition can be ensured regardless of day or night, or weather.

[0218] (Embodiment 4)

[0219] Next, an agricultural machine according to an exemplary 4th embodiment of the present disclosure will be described.

[0220] In the present embodiment, the image processing device mounted on the body of the agricultural machine has a function of performing re-learning (hereinafter, also referred to as "optimization learning") of the learned model. Through re-learning, the learned model can be improved to adapt to the actual use environment of the agricultural machine, and the detection performance of the object can be improved.

[0221] The agricultural machine in the present embodiment is the same as the agricultural machines in the foregoing embodiments, and includes a photographing device, an image processing device that detects a specific object (such as a person) from the image obtained by the photographing device, and a control device that controls the operation of the agricultural machine based on the detection result of the object. The image processing device stores one or more learned models for detecting an object from an input image. The image processing device in the present embodiment performs re-learning of the learned model by executing the following steps (S41)-(S43).

[0222] (S41)Generate an input image based on the image obtained by the photographing device.

[0223] (S42)Detect the object by inputting the input image into the learned model.

[0224] (S43)Based on one or more images in which the object is detected among the multiple images obtained by the photographing device during the operation of the agricultural machine, perform re-learning of the learned model.

[0225] According to the above structure, the learned model can be improved to match the actual use environment of the agricultural machine, and the detection performance of the object can be improved.

[0226] The structure of the agricultural machine in the present embodiment is the same as Figure 1 or Figure 5 shown in the structure of the agricultural machine 100. The structure of the image processing device is the same as Figure 2 or Figure 17 shown in the structure of the image processing device 20. In the present embodiment, the image processing device 20 has a function of improving the learned model 27 through re-learning, which is different from the foregoing embodiments in this regard.

[0227] Figure 2 The learned model 27 shown in Figure 17 or each of the multiple learned models 27 shown in

[0228] However, the actual usage environments of the agricultural machine 100 are diverse, and sometimes it is difficult to construct a model that can maintain high detection performance in multiple usage environments solely through prior learning. Therefore, the image processing device 20 in the present embodiment is configured to perform re-learning of the learned model using the images in which the object is actually detected in the images obtained in the actual usage environment of the agricultural machine 100. Thereby, the model can be continuously improved, and the detection performance of the object can be enhanced.

[0229] In the present embodiment, the image processing device 20, which is an edge computing device mounted on the agricultural machine 100, is configured to perform re-learning of the learned model. Therefore, it is possible to optimize and learn the learned model suitable for the usage environment of the agricultural machine 100 without communicating with an external computer such as a cloud server.

[0230] Hereinafter, the operation of the image processing device 20 in the present embodiment will be described in more detail.

[0231] The image processing device 20 in the present embodiment determines whether the detection result of the object is correct for each of one or more images in which the object is detected among the multiple images obtained by the imaging device 10 during the operation of the agricultural machine 100. Whether the detection result of the object is correct can be determined, for example, based on the time from when the movement of the agricultural machine 100 stops after the detection process of the object until the movement restarts. This determination method is based on the following idea: in the case where the object is detected in the image although the object actually does not exist (i.e., in the case of false detection), after the agricultural machine 100 temporarily stops, the user of the agricultural machine 100 should immediately restart the movement.

[0232] When it is determined that the detection result of the object is correct, the image processing device 20 uses this image in the re-learning of the learned model. More specifically, the image processing device 20 does not use the image in which the object is falsely detected although the object actually does not exist in the re-learning, and only uses the image in which the object is correctly detected in the re-learning.

[0233] When an object is detected from an image, the image processing device 20 sends a signal indicating the detection of the object to the control device 30. If a signal indicating the detection of the object is received, the control device 30 performs specific control such as stopping the agricultural machine 100, for example. At this time, the detection result is correct when the object actually exists on or near the path of the agricultural machine 100, and conversely, the detection result is incorrect when the object does not actually exist. When the object actually exists, after the object moves to a position where it does not interfere with the movement of the agricultural machine 100, the user of the agricultural machine 100 performs an operation to restart the movement of the agricultural machine 100. On the other hand, when the object does not actually exist, after confirming the absence of the object, the user immediately restarts the movement of the agricultural machine 100. The user operates, for example, Figure 5 the operation switch group 132 or an input device such as the terminal monitor 131 shown in FIG., and provides a restart instruction to the control device 30 (ECU 165 in the Figure 5 example), and this restart instruction indicates the restart of the movement. The control device 30 restarts the movement of the agricultural machine 100 in response to the input restart instruction. The image processing device 20 can be configured to measure the time from when the agricultural machine 100 stops until it restarts moving, and determine whether the detection result of the object is correct based on this time. For example, it can be configured to determine that the detection result of the object is correct if the measured time is longer than a threshold, and determine that the detection result of the object is incorrect if the measured time is below the threshold.

[0234] Figure 22 is a flowchart showing a specific example of the operation of the image processing device 20 in the present embodiment. In the Figure 22 example, the image processing device 20 performs the detection process of the object based on the dynamic image obtained during the movement of the agricultural machine 100 and the relearning of the learned model by executing the operations of steps S410 to S500. Figure 22 The operation shown in FIG. starts, for example, when an instruction to start the operation of the agricultural machine 100 is provided from the user via the input device.

[0235] In step S410, the image processing device 20 generates an input image to be input to the learned model based on the captured image generated by the imaging device 10. The image processing device 20 generates the input image, for example, by performing prescribed preprocessing (such as resizing, normalizing, noise removal, etc.) on the captured image. Alternatively, the image processing device 20 can also perform Figure 3The processes of steps S110 to S130 are performed to generate an input image. By performing the processes of steps S110 to S130, the area of the detection target can be reduced according to the operation state of the agricultural machine 100, and the detection accuracy of the object is improved.

[0236] In step S420, the image processing device 20 performs the process of detecting the object by inputting the input image into the learned model stored in the memory 24. As in Embodiment 3, one learned model corresponding to the environment may be selected from multiple learned models and applied to the image, or as in Embodiments 1 and 2, the learned model to be used may be determined in advance.

[0237] In step S430, the image processing device 20 determines whether an object is detected from the input image. If an object is detected, the process proceeds to step S440. If no object is detected, the process proceeds to step S490.

[0238] Figure 23 This is a diagram showing an example of the detection result of the object. In this example, the image processing device 20 detects a specific object from the input image by executing software for object detection. For example, a label indicating the type of the object (such as Person if it is a person), the coordinate values of the representative points of the bounding box indicating the position of the object in the image (such as the upper left vertex or the center point), and a numerical value indicating the reliability of the detection result may be output as the detection result.

[0239] In step S430, the image processing device 20 may also determine whether an object is detected based on the distance between the agricultural machine 100 and the object. For example, based on the position of the object in the input image and the position and orientation information of the imaging device 10 in the agricultural machine 100, the position of the object in the coordinate system fixed to the ground may be calculated, and it may be determined that an object is detected when the distance between this position and the agricultural machine 100 is less than a threshold value. Here, the distance may also be estimated by Figure 15 or Figure 16 the method shown. In addition, in the determination of the distance, the measurement value of a distance measurement sensor such as the laser sensor 125 shown in Figure 5 may also be used.

[0240] In step S440, the image processing device 20 sends a signal indicating that an object has been detected to the control device 30. If the control device 30 receives this signal, the movement of the agricultural machine 100 is stopped, and a signal indicating that the movement has stopped is sent to the image processing device 20.

[0241] In step S450, the image processing device 20 receives a signal from the control device 30 indicating that the movement of the agricultural machine 100 has stopped. If this signal is received, the image processing device 20 stands by until the movement of the agricultural machine 100 restarts.

[0242] As described above, after the movement of the agricultural machine 100 stops, the user of the agricultural machine 100 confirms whether there is actually an object in the traveling direction of the agricultural machine 100. In the case where there is actually no object (i.e., in the case of false detection), the user restarts the movement of the agricultural machine 100 within a relatively short time. On the other hand, in the case where there is actually an object, the user restarts the movement of the agricultural machine 100 after confirming that the object no longer exists on the predetermined path of the agricultural machine 100. If the movement of the agricultural machine 100 restarts, the control device 30 sends a signal indicating the restart of the movement to the image processing device 20.

[0243] In step S460, the image processing device 20 receives a signal from the control device 30 indicating the restart of the movement of the agricultural machine.

[0244] In step S470, the image processing device 20 compares the time from the stop of the movement of the agricultural machine to the restart of the movement (hereinafter referred to as "standby time") with a predetermined threshold. In the case where the standby time is longer than the threshold, the process proceeds to step S480. In the case where the standby time is below the threshold, it is determined that an object has been falsely detected, and the process proceeds to step S490. In the case where the standby time is longer than the threshold, the object is processed as being normally detected. In the case where the standby time is below the threshold, the object is processed as being falsely detected.

[0245] In step S480, the image processing device 20 records the input image in the memory 24 as an image for re-learning. At this time, information indicating the detection result may also be recorded in association with the input image. The information indicating the detection result may include a label indicating the type of the object and coordinate values indicating the position of the object within the input image (e.g., coordinate values of a bounding box).

[0246] In step S490, the image processing device 20 determines whether it has received a signal indicating the completion of the agricultural operation based on the agricultural machine 100 (hereinafter referred to as "operation completion signal"). The operation end signal may be sent from the control device 30 to the image processing device 20 during the agricultural operation. The so-called "when the agricultural operation is completed" may refer to, for example, along a target path within the farmland by the agricultural machine 100 (e.g., Figure 6When the operation travel of the target path 74) shown is completed, or when the discharge operation of the harvested product after the operation travel is completed. When the image processing device 20 does not receive the operation completion signal, it returns to step S410. The image processing device 20 repeats the processing from step S410 to S490 until it receives the operation completion signal. When the image processing device 20 receives the operation completion signal, it proceeds to step S500.

[0247] In step S500, the image processing device 20 performs re-learning of the model based on the input image group recorded in the memory 24 as the re-learning images, creates a new learned model, and records it in the memory 24. For example, the image processing device 20 sets data including each of the input images recorded as the re-learning images, the position information of the object in each input image, and the information of the label indicating the type of the object in each input image as the learning data, and creates a new learned model by executing the software for re-learning.

[0248] As described above, during the agricultural operation by the agricultural machine 100, the image processing device 20 in the present embodiment generates an input image based on the image obtained by the imaging device 10, detects the object by inputting the input image into the learned model, and repeatedly performs the action of recording the image when the object is detected. After receiving the signal indicating the end of the agricultural operation, the image processing device 20 performs re-learning of the learned model based on one or more images in which the object is detected. By such processing, for example, every time the agricultural operation in a farmland is completed, the model can be re-learned to improve the model.

[0249] In addition, the timing for performing the re-learning of the model in step S500 may not be the timing when the agricultural operation by the agricultural machine 100 is completed. For example, the re-learning of the model in step S500 may be performed at the timing of switching from the automatic driving mode to the manual driving mode or at the timing when the power of the agricultural machine 100 is turned off (OFF).

[0250] In addition, as a method for determining whether the detection result of the object is correct, a method different from the method based on the time from when the movement of the agricultural machine 100 stops until the movement restarts after the object detection process may be used. For example, the image processing device 20 may also be based on information from an object detection sensor mounted on the agricultural machine 100 (for example, Figure 5Based on the output of the laser sensor 125 and / or the millimeter-wave radar 127) in the example, it is determined whether the object detected from the image actually exists. When there is a person as an object among the crops to be harvested, object detection sensors such as laser sensors or millimeter-wave radars may not be able to distinguish between people and crops, but can detect the presence of some object. Therefore, by using both the detection of objects based on images and the detection of objects based on object detection sensors, it is possible to determine whether the detection result of objects based on images is correct.

[0251] In Figure 22 In the example, when the time period from the stop to the restart of the movement of the agricultural machine 100 is longer than the threshold (i.e., when it is determined that the detection result of the object based on the image is correct), the image processing device 20 stores the input image in the memory 24. Not limited to such processing, the image processing device 20 may also attach information indicating whether the detection result of the object is correct to the input image and store it in the memory 24. For example, when it is determined that the detection result of the object based on the input image is incorrect, information indicating that the detection result is incorrect may be associated with the input image and stored in the memory 24. In addition, when it is determined that the detection result of the object based on the input image is correct, the image processing device 20 may also associate information indicating that the detection result is correct with the input image and store it in the memory 24. Such information can be stored as metadata associated with the input image. The image processing device 20 can determine the images to be used in the relearning of the learned model based on this information. For example, the image processing device 20 may not use the images associated with the information indicating incorrect detection results in the relearning, and use the images associated with the information indicating correct detection results in the relearning.

[0252] In step S480, the image processing device 20 may also associate and record the position information of the object in each of one or more images in which the object is detected with the image. The position information may be, for example, the coordinate values of the representative point (e.g., the upper left vertex or the center point) of the bounding box indicating the position of the object in the image. By attaching and recording such position information to the image, the relearning of the learned model can be efficiently performed. For example, the image processing device 20 may store the position information of the object in the image and the label indicating the type of the object in the memory 24. By doing so, the relearning of the learned model can be made more efficient using the position information of the object in the image, the label indicating the type of the object, and the data for annotation.

[0253] After the relearning in step S500, the image processing apparatus 20 can either update the existing learned model with the learned model after the relearning or make the two coexist. In the case of making the two coexist, the image processing apparatus 20 can also be able to select which one of the existing learned model and the learned model after the relearning to use for the detection process of the object according to the user's operation. In addition, the image processing apparatus 20 can also update the existing learned model with the learned model after the relearning according to an instruction from the user. The instruction from the user can be provided, for example, by an operation of an input device such as the terminal monitor 131 shown in Figure 5 According to such a configuration, it is possible to update the model after the user confirms that the performance of the learned model after the relearning is improved compared to the performance of the existing learned model, or return to the original learned model in the case where the effect of the relearning cannot be confirmed.

[0254] In the above-described first to fourth embodiments, the examples in the case where the work machine is an agricultural machine such as a combine harvester have been mainly described, but the above-described various techniques can also be applied to work machines other than agricultural machines. For example, part or all of the functions of the above-described first to fourth embodiments can also be implemented on the construction work vehicle 200 shown in Figure 24 In addition, the image acquisition device is not limited to the imaging device 10 such as the camera 126, and can also be a device such as a LiDAR sensor that can acquire a point cloud image.

[0255] As described above, the present disclosure includes the agricultural machine, the image processing apparatus, and the image processing method described below.

[0256] [Item A1]

[0257] A work machine that is a work machine that performs work while moving, the work machine including:

[0258] An image acquisition device that acquires an image in the moving direction of the work machine;

[0259] An image processing apparatus that detects a specific object from the image; and

[0260] A control device that controls the operation of the work machine based on the detection result of the object,

[0261] The image processing apparatus includes a memory that stores a learned model for detecting the object from an input image,

[0262] The image processing apparatus

[0263] Extract a partial image representing a region of interest determined based on the operating state of the work machine from the image obtained by the image acquisition device.

[0264] Generate the input image based on the partial image.

[0265] Detect the object by inputting the input image into the learned model.

[0266] [Item A2]

[0267] The work machine according to Item A1, wherein

[0268] The image processing device obtains information related to at least one of the moving speed of the work machine, the turning state of the work machine, and the tilting state of the work machine, and changes the region of interest based on the information.

[0269] [Item A3]

[0270] The work machine according to Item A2, wherein

[0271] The image processing device changes the size of the region of interest according to the moving speed of the work machine.

[0272] [Item A4]

[0273] The work machine according to Item A3, wherein

[0274] The higher the moving speed of the work machine, the smaller the region of interest the image processing device makes.

[0275] [Item A5]

[0276] The work machine according to Item A2, wherein

[0277] When the work machine turns right, the image processing device moves the region of interest to the right, and when the work machine turns left, the image processing device moves the region of interest to the left.

[0278] [Item A6]

[0279] The work machine according to Item A5, wherein

[0280] The larger the steering angle during turning of the work machine, the larger the region of interest the image processing device moves.

[0281] [Item A7]

[0282] The work machine according to Item A2, wherein

[0283] It also includes an inclination sensor that measures the inclination of the work machine,

[0284] and the image processing device changes the position of the region of interest according to the inclination.

[0285] [Item A8]

[0286] The work machine according to Item A7, wherein

[0287] the inclination sensor measures the pitch angle of the work machine as the inclination,

[0288] and the image processing device moves the region of interest upward or downward according to the pitch angle.

[0289] [Item A9]

[0290] The work machine according to any one of Items A1 to A8, further comprising:

[0291] a storage device that stores map data including the area of the farmland where the work machine performs operations; and

[0292] a positioning device that obtains the positioning data of the work machine,

[0293] In this work machine, the image processing device determines the region of interest based on the map data, the positioning data, and the operation state of the work machine.

[0294] [Item A10]

[0295] The work machine according to Item A9, wherein

[0296] the image processing device determines, from the image obtained by the image acquisition device, the region corresponding to the operation area where the operation is performed in the farmland based on the map data and the positioning data, and determines the region of interest from the region based on the operation state of the work machine.

[0297] [Item A11]

[0298] The work machine according to any one of Items A1 to A10, wherein

[0299] the image processing device generates the input image by including a process of compressing the number of pixels of the partial image to a preset number of pixels.

[0300] [Item A12]

[0301] The work machine according to any one of Items A1 to A11, wherein

[0302] The work machine is a work vehicle or an unmanned aerial vehicle that performs autonomous driving.

[0303] The control device controls the autonomous driving operation of the work machine.

[0304] [Item A13]

[0305] The work machine according to any one of Items A1 to A12, wherein

[0306] When the object is detected, the control device performs at least one of stopping the work machine, decelerating the work machine, and outputting a warning.

[0307] [Item A14]

[0308] An image processing device that detects a specific object from an image acquired by an image acquisition device mounted on a work machine, the image processing device comprising:

[0309] A memory that stores a learned model for detecting the object from the input image; and

[0310] An arithmetic circuit

[0311] The arithmetic circuit

[0312] Extracts a partial image representing a region of interest determined based on the operation state of the work machine from the image acquired by the image acquisition device,

[0313] Generates the input image based on the partial image,

[0314] Detects the object by inputting the input image into the learned model.

[0315] [Item A15]

[0316] A method executed by an image processing device that detects a specific object from an image acquired by an image acquisition device mounted on a work machine,

[0317] The method includes the following steps:

[0318] Acquires the image from the image acquisition device;

[0319] Acquires information indicating the operation state of the work machine;

[0320] Extracts a partial image representing a region of interest determined based on the operation state of the work machine from the image;

[0321] Generate an input image based on the partial image; and

[0322] Detect the object by inputting the input image into a pre-generated and learned model.

[0323] [Item B1]

[0324] A working machine, comprising:

[0325] A body;

[0326] An image acquisition device; and

[0327] An image processing device that detects a specific object from the image acquired by the image acquisition device,

[0328] The image processing device

[0329] Detect the boundary between the sky and the ground objects other than the sky and the object from the image,

[0330] Estimate the inclination of the body based on the position of the boundary in the image,

[0331] Estimate the position of the object in the coordinate system fixed to the ground based on the position of the object in the image and the estimated inclination.

[0332] [Item B2]

[0333] The working machine according to Item B1, wherein

[0334] The image processing device estimates the distance from the working machine to the object based on the position of the object in the coordinate system fixed to the ground.

[0335] [Item B3]

[0336] The working machine according to Item B1, wherein

[0337] It further comprises an inclination sensor that measures the inclination of the body,

[0338] The image processing device

[0339] Calculates the difference between the inclination estimated based on the position of the boundary in the image and the inclination measured by the inclination sensor,

[0340] When the difference is less than a threshold value, estimate the position of the object in the coordinate system fixed to the ground based on the position of the object in the image and the estimated inclination.

[0341] [Item B4]

[0342] The working machine according to any one of Items B1 to B3, wherein,

[0343] The image processing device estimates the inclination of the machine body based on the position of the boundary in the image and the displacement amount from the position of the boundary detected in the image acquired by the image acquisition device at a past time.

[0344] [Item B5]

[0345] The working machine according to Item B4, wherein,

[0346] The past time is a time when the inclination measured by the inclination sensor is less than a reference value.

[0347] [Item B6]

[0348] The working machine according to Item B4 or B5, wherein,

[0349] The image processing device estimates the pitch angle of the machine body as the inclination based on the displacement amount.

[0350] [Item B7]

[0351] The working machine according to any one of Items B1 to B6, wherein,

[0352] The image processing device converts the position coordinates of the object in the image into position coordinates in the coordinate system fixed to the ground based on the estimated inclination, and estimates the distance to the object based on the converted position coordinates.

[0353] [Item B8]

[0354] The working machine according to any one of Items B1 to B7, wherein,

[0355] It further includes a control device that controls the operation of the working machine based on the position of the object in the coordinate system fixed to the ground.

[0356] [Item B9]

[0357] The working machine according to Item B8, wherein,

[0358] When the distance estimated based on the position of the object in the coordinate system is less than a specified value, the control device performs at least one of stopping the working machine, decelerating the working machine, and outputting a warning.

[0359] [Item B10]

[0360] The working machine as described in item B8 or B9, wherein,

[0361] the working machine is a working vehicle or an unmanned aircraft that performs autonomous driving,

[0362] and the control device controls the autonomous driving operation of the working machine.

[0363] [Item B11]

[0364] An image processing device that detects a specific object from an image acquired by an image acquisition device mounted on a working machine,

[0365] this image processing device

[0366] detects the boundary between the sky and the ground objects other than the sky, and the object, from the image,

[0367] estimates the inclination of the body based on the position of the boundary in the image,

[0368] and estimates the position of the object in a coordinate system fixed to the ground based on the position of the object in the image and the estimated inclination.

[0369] [Item B12]

[0370] A method executed by an image processing device that detects a specific object from an image acquired by an image acquisition device mounted on a working machine,

[0371] this method includes the following steps:

[0372] detecting the boundary between the sky and the ground objects other than the sky, and the object, from the image;

[0373] estimating the inclination of the body based on the position of the boundary in the image; and

[0374] estimating the position of the object in a coordinate system fixed to the ground based on the position of the object in the image and the estimated inclination.

[0375] [Item C1]

[0376] A working machine that performs operations while moving, the working machine comprising:

[0377] an image acquisition device that acquires an image in the moving direction of the working machine;

[0378] an image processing device that detects a specific object from the image; and

[0379] A control device controls the operation of the work machine based on the detection result of the object.

[0380] The image processing device includes a memory that stores a plurality of learned models for detecting the object from an input image.

[0381] The image processing device

[0382] generates the input image based on the image acquired by the image acquisition device.

[0383] The object is detected by inputting the input image into one learned model selected from the plurality of learned models according to the environment around the work machine.

[0384] [Item C2]

[0385] The work machine according to Item C1, wherein

[0386] the plurality of learned models are associated with different luminances of the environment around the work machine.

[0387] The image processing device selects one learned model corresponding to the luminance of the current environment from the plurality of learned models.

[0388] [Item C3]

[0389] The work machine according to Item C2, wherein

[0390] the image processing device determines the luminance of the environment based on at least one of the image, the output from an illuminance sensor, the input from a user, and the lighting state of a lamp.

[0391] [Item C4]

[0392] The work machine according to Item C1, wherein

[0393] the plurality of learned models are associated with different time periods.

[0394] The image processing device selects one learned model corresponding to the current time period from the plurality of learned models.

[0395] [Item C5]

[0396] The work machine according to Item C4, wherein

[0397] the plurality of learned models include a daytime model and a nighttime model.

[0398] The image processing device selects the daytime model during the daytime period and selects the nighttime model during the nighttime period.

[0399] [Item C6]

[0400] The work machine according to Item C1, wherein

[0401] The plurality of learned models include a backlight model and a non-backlight model.

[0402] The image processing device determines whether it is backlit based on at least one of the image, an input from the user, and an output from an illuminance sensor, selects the backlight model in the case of backlighting, and selects the non-backlight model in the case of non-backlighting.

[0403] [Item C7]

[0404] The work machine according to Item C1, wherein

[0405] The plurality of learned models are associated with different types of crops.

[0406] The image processing device selects one learned model corresponding to the type of crop of the work object from the plurality of learned models.

[0407] [Item C8]

[0408] The work machine according to Item C7, wherein

[0409] The image processing device determines the type of crop of the work object based on the image or an input from the user.

[0410] [Item C9]

[0411] The work machine according to Item C1, wherein

[0412] The plurality of learned models are associated with different weathers.

[0413] The image processing device selects one learned model corresponding to the current weather from the plurality of learned models.

[0414] [Item C10]

[0415] The work machine according to Item C9, wherein

[0416] The image processing device determines the current weather based on the image or weather-related information obtained from an external device.

[0417] [Item C11]

[0418] The work machine according to any one of Items C1 to C10, wherein,

[0419] Each of the plurality of learned models has a common model architecture and different weight sets.

[0420] [Item C12]

[0421] The work machine according to any one of Items C1 to C11, wherein,

[0422] The control device moves the work machine along a set path,

[0423] Whenever the work machine changes direction on the path, the image processing device selects the learned model.

[0424] [Item C13]

[0425] The work machine according to any one of Items C1 to C12, wherein,

[0426] During the movement of the work machine, the image processing device selects the learned model at each predetermined time.

[0427] [Item C14]

[0428] The work machine according to any one of Items C1 to C13, wherein,

[0429] When the object is detected, the control device performs at least one of stopping the work machine, decelerating the work machine, and outputting a warning.

[0430] [Item C15]

[0431] An image processing device that detects a specific object from an image obtained by an image acquisition device mounted on a work machine, the image processing device comprising:

[0432] A memory that stores a plurality of learned models for detecting the object from an input image; and

[0433] An arithmetic circuit,

[0434] The arithmetic circuit generates the input image based on the image obtained by the image acquisition device,

[0435] The object is detected by inputting the input image into one learned model selected from the plurality of learned models according to the environment around the work machine.

[0436] [Item C16]

[0437] A method, which is a method executed by an image processing device that detects a specific object from an image obtained by an image acquisition device mounted on a work machine.

[0438] The method includes the following steps:

[0439] Generating the input image based on the image obtained by the image acquisition device; and

[0440] Detecting the object by inputting the input image into one learned model selected from a plurality of learned models according to the environment around the work machine.

[0441] [Item D1]

[0442] A work machine that performs work while moving, and the work machine includes:

[0443] An image acquisition device that acquires an image in the moving direction of the work machine;

[0444] An image processing device that detects a specific object from the image; and

[0445] A control device that controls the operation of the work machine based on the detection result of the object.

[0446] The image processing device includes a memory that stores a learned model for detecting the object from an input image.

[0447] The image processing device

[0448] Generates the input image based on the image obtained by the image acquisition device.

[0449] Detects the object by inputting the input image into the learned model.

[0450] Based on one or more images in which the object is detected among a plurality of images obtained by the image acquisition device during the operation of the work machine, re-learning of the learned model is performed.

[0451] [Item D2]

[0452] The work machine according to Item D1, wherein

[0453] The image processing device

[0454] For each of the one or more images in which the object is detected, based on the operation of the work machine after the detection of the object, it is determined whether the detection result of the object is correct.

[0455] When it is determined that the detection result of the object is correct, the image is used in the relearning of the learned model.

[0456] [Item D3]

[0457] The working machine according to Item D2, wherein

[0458] When the object is detected, the control device stops the working machine and restarts the movement of the working machine in response to an input restart instruction.

[0459] The image processing device determines whether the detection result of the object is correct based on the time from when the working machine stops to when it restarts moving.

[0460] [Item D4]

[0461] The working machine according to Item D1, wherein

[0462] It further includes an object detection sensor.

[0463] The image processing device

[0464] For each of the one or more images in which the object is detected, it determines whether the detection result of the object is correct based on the output from the object detection sensor.

[0465] When it is determined that the detection result of the object is correct, the image is used in the relearning of the learned model.

[0466] [Item D5]

[0467] The working machine according to any one of Items D1 to D4, wherein

[0468] When it is determined that the detection result of the object is incorrect, the image processing device associates and records information indicating that the detection result is incorrect with the image.

[0469] The image processing device determines the images to be used in the relearning of the learned model based on the information.

[0470] [Item D6]

[0471] The working machine according to any one of Items D1 to D4, wherein

[0472] When it is determined that the detection result of the object is correct, the image processing device associates and records information indicating that the detection result is correct with the image.

[0473] The image processing device determines an image to be used in relearning the learned model based on the information.

[0474] [Item D7]

[0475] The working machine according to any one of Items D1 to D6, wherein

[0476] the image processing device

[0477] associates and records the position information of the object in each of the one or more images in which the object is detected with the image.

[0478] Based on the position information, relearning of the learned model is performed.

[0479] [Item D8]

[0480] The working machine according to any one of Items D1 to D7, wherein

[0481] After the relearning is performed, the image processing device updates the learned model with the learned model that has been relearned according to an instruction from a user.

[0482] [Item D9]

[0483] The working machine according to any one of Items D1 to D8, wherein

[0484] the image processing device

[0485] During operation by the working machine, an input image is generated based on the image acquired by the image acquisition device, the object is detected by inputting the input image into the learned model, and the action of recording the input image is repeatedly performed when the object is detected.

[0486] After receiving a signal indicating that the operation has ended, relearning of the learned model is performed based on one or more input images in which the object is detected.

[0487] [Item D10]

[0488] An image processing device that detects a specific object from an image acquired by an image acquisition device mounted on a working machine, the image processing device comprising:

[0489] a memory that stores a learned model for detecting the object from an input image; and

[0490] an arithmetic circuit

[0491] The arithmetic circuit

[0492] Based on the image acquired by the image acquisition device, generate the input image.

[0493] By inputting the input image into the learned model, detect the object.

[0494] Based on one or more images in which the object is detected among a plurality of images acquired by the image acquisition device during the operation of the work machine, perform relearning of the learned model.

[0495] [Item D11]

[0496] A method for detecting a specific object from an image acquired by an image acquisition device mounted on a work machine.

[0497] The method includes the following steps:

[0498] Based on the image acquired by the image acquisition device, generate the input image;

[0499] By inputting the input image into the learned model, detect the object; and

[0500] Based on one or more images in which the object is detected among a plurality of images acquired by the image acquisition device during the operation of the work machine, perform relearning of the learned model.

[0501] Industrial applicability

[0502] The technology of the present disclosure can be applied to various work machines such as agricultural machines such as harvesters, tractors, transplanters, and agricultural drones, construction machines such as backhoes, wheel loaders, and transport vehicles, and snow removal vehicles.

[0503] Explanation of reference numerals

[0504] 10 Shooting device; 20 Image processing device; 22 Processor; 24 Memory; 25 Computer program; 27 Learned model; 30 Control device; 100 Agricultural machinery; 101 Body; 102 Traveling device; 103 Cutting device; 104 Conveyor device; 105 Threshing device; 106 Box; 107 Discharge device; 108 Discharged straw processing device; 109 Reel; 110 Cabin; 111 Engine; 112 Transmission device; 117 Discharge port; 120 GNSS unit; 121 GNSS receiver; 122 RTK receiver; 123 Inertial measurement unit (IMU); 124 Processing circuit; 125 LiDAR sensor; 126 Camera; 127 Obstacle sensor; 131 Terminal monitor; 132 Operation switch group; 133 Buzzer; 140 Driving device; 141 Power transmission mechanism; 142 Lamp; 150 Sensor group; 151 Vehicle speed sensor; 152 Steering angle sensor; 153 Illuminance sensor; 160 Control system; 164 Storage device; 165 - 167 ECU; 190 Communication device; 200 Construction work vehicle.

Claims

1. A working machine having: Body; an image acquisition device; and an image processing device for detecting a specific object from the image acquired by the image acquisition device; The image processing device detecting the boundary between the sky and the ground object other than the sky and the object from the image, estimating the inclination of the body based on the position of the boundary in the image, The position of the object in a coordinate system fixed to the ground is estimated based on the position of the object in the image and the estimated inclination.

2. The working machine according to claim 1, wherein: The image processing device estimates the distance from the working machine to the object based on the position of the object in the coordinate system fixed to the ground.

3. The working machine according to claim 1, wherein: A tilt sensor is also provided, the tilt sensor measures the tilt of the body, The image processing device calculating a difference between the inclination estimated based on the position of the boundary in the image and the inclination measured by the inclination sensor, When the difference is smaller than a threshold value, the position of the object in the coordinate system fixed to the ground is estimated based on the position of the object in the image and the estimated inclination.

4. The working machine according to any one of claims 1 to 3, wherein: The image processing device estimates the inclination of the machine body based on the position of the boundary in the image and the amount of displacement of the position of the boundary detected from the image acquired by the image acquisition device at a past time.

5. The working machine according to claim 4, wherein: The past time is the time when the inclination measured by the inclination sensor becomes smaller than a reference value.

6. The working machine according to claim 4, wherein: The image processing device estimates a pitch angle of the body as the inclination based on the displacement amount.

7. The working machine according to any one of claims 1 to 3, wherein: The image processing device converts the position coordinates of the object in the image into position coordinates in the coordinate system fixed to the ground based on the estimated inclination, and estimates the distance to the object based on the converted position coordinates.

8. The working machine according to any one of claims 1 to 3, wherein: A control device is further provided, the control device controlling the operation of the working machine based on the position of the object in the coordinate system fixed to the ground.

9. The working machine according to claim 8, wherein: When the distance estimated based on the position of the object in the coordinate system is smaller than a predetermined value, the control device executes at least one of stopping the working machine, decelerating the working machine, and outputting a warning.

10. The working machine according to claim 8, wherein: The working machine is an autonomous working vehicle or an unmanned aerial vehicle. The control device controls the operation of the automatic driving of the working machine.

11. An image processing device for detecting a specific object from an image acquired by an image acquisition device mounted on a working machine, The image processing device detecting the boundary between the sky and the ground object other than the sky and the object from the image, estimating the inclination of the body based on the position of the boundary in the image, The position of the object in a coordinate system fixed to the ground is estimated based on the position of the object in the image and the estimated inclination.

12. A method, the method being performed by an image processing device, the image processing device detecting a specific object from an image acquired by an image acquisition device mounted on a working machine, The method includes: Detecting the boundary between the sky and the ground objects other than the sky and the object from the image; estimating the tilt of the body based on the position of the boundary in the image; as well as The position of the object in a coordinate system fixed to the ground is estimated based on the position of the object in the image and the estimated inclination.

Images