Work machine, image processing device, and method for detecting specific object from image

Abstract

This work machine executes work while moving. The work machine comprises: an image acquisition device that acquires an image in a movement direction of the work machine; an image processing device that detects a specific object from the image; and a control device that controls the operation of the work machine on the basis of the result of the detection of the object. The image processing device includes a memory for storing a plurality of trained models for detecting the object from an input image, generates the input image on the basis of the image acquired by the image acquisition device, and detects the object by inputting the input image to one trained model selected from among the plurality of trained models according to an environment around the work machine.

Description

Work machine, image processing device, and method for detecting specific objects from images

[0001] The present disclosure relates to a work machine, an image processing device, and a method for detecting a specific object from an image.

[0002] Research and development is underway on smart agriculture, which utilizes information and communication technology (ICT) and the Internet of Things (IoT), as the next generation of agriculture. Research and development is also underway to automate and unmanned agricultural machinery used in fields, such as tractors, harvesters, transplanters, and agricultural drones. For example, agricultural machinery that performs agricultural work while autonomously moving within fields using positioning systems such as the Global Navigation Satellite System (GNSS), which enables precise positioning, is now being put into practical use.

[0003] Patent Literature 1 discloses an agricultural work machine equipped with an imaging device capable of capturing images of the area ahead of the machine in the direction of travel in a field. This agricultural work machine can detect the presence of an obstacle in the field based on the captured images and identify the type of obstacle. This agricultural work machine is also capable of output control (e.g., decelerating or stopping the machine, issuing a warning about the obstacle, etc.) based on a control pattern selected from multiple control patterns according to the type of obstacle.

[0004] Japanese Patent Application Laid-Open No. 2020-178619

[0005] There is a demand for improved object detection performance in agricultural machinery, construction machinery, and other work machines that detect specific objects based on images (e.g., visible light images, infrared images, or point cloud images) acquired by an image acquisition device such as an imaging device (camera) or LiDAR.

[0006] The present disclosure provides techniques for further improving the performance of detecting an object.

[0007] A method according to one embodiment of the present disclosure is a method executed by an image processing device that detects a specific object from an image acquired by an image acquisition device mounted on a work machine, and includes acquiring the image from the image acquisition device, acquiring information indicating the operating state of the work machine, extracting from the image a partial image indicating a region of interest determined based on the operating state of the work machine, generating an input image based on the partial image, and detecting the object by inputting the input image into a pre-generated trained model.

[0008] A method according to another embodiment of the present disclosure is a method executed by an image processing device that detects a specific object from an image acquired by an image acquisition device mounted on a work machine, the method including: detecting a boundary between sky and a non-sky feature, and the object from the image; estimating the inclination of the aircraft based on the position of the boundary in the image; and estimating the position of the object in a coordinate system fixed to the ground based on the position of the object in the image and the estimated inclination.

[0009] A method according to yet another embodiment of the present disclosure is a method executed by an image processing device that detects a specific object from an image acquired by an image acquisition device mounted on a work machine, and includes generating an input image based on the image acquired by the image acquisition device, and detecting the object by inputting the input image into one trained model selected from a plurality of trained models according to the environment around the work machine.

[0010] A method according to yet another embodiment of the present disclosure is a method for detecting a specific object from an image acquired by an image acquisition device mounted on a work machine, comprising: generating an input image based on the image acquired by the image acquisition device; detecting the object by inputting the input image into the trained model; and retraining the trained model based on one or more images in which the object is detected, among a plurality of images acquired by the image acquisition device during operation of the work machine.

[0011] A general or specific aspect of the present disclosure may be realized by an apparatus, a system, a method, an integrated circuit, a computer program, or a computer-readable non-transitory storage medium, or any combination thereof. The computer-readable storage medium may include a volatile storage medium or a non-volatile storage medium. An apparatus may be composed of multiple devices. When an apparatus is composed of two or more devices, the two or more devices may be located in a single device or may be located separately in two or more separate devices.

[0012] According to an embodiment of the present disclosure, it is possible to improve the object detection performance in a work machine that detects a specific object based on an image acquired by an imaging device.

[0013] FIG. 1 is a block diagram showing a schematic configuration of an agricultural machine, which is an example of a work machine according to an exemplary embodiment of the present disclosure. FIG. 2 is a block diagram showing an example of the configuration of an image processing device. FIG. 3 is a flowchart showing an example of the operation of the image processing device. FIG. 4 is a side view schematically showing an example of the agricultural machine. FIG. 5 is a block diagram showing an example of the configuration of the agricultural machine. FIG. 6 is a diagram showing an example of a route taken by the agricultural machine when traveling while harvesting crops in a field. FIG. 7 is a diagram showing a schematic diagram of detecting an object (person) using a camera mounted on the agricultural machine. FIG. 8 is a diagram showing a schematic diagram of a processing flow executed by an ECU (image processing device). FIG. 9A is a diagram showing an example of a captured image acquired by a camera. FIG. 9B is a diagram showing an example of a region of interest that can be selected when the agricultural machine is turning right. FIG. 9C is a diagram showing a partial image corresponding to the region of interest shown in FIG. 9B. FIG. 9D is a diagram showing an example of a region of interest that can be selected when the agricultural machine is turning left. FIG. 9E is a diagram showing a partial image corresponding to the region of interest shown in FIG. 9D. FIG. 10A is a diagram showing an example of a region of interest that can be selected when the agricultural machine is moving forward at a relatively high speed. FIG. 10B is a diagram showing a partial image corresponding to the region of interest shown in FIG. 10A . FIG. 11 is a diagram showing an example of an agricultural machine traveling on the outermost periphery of a work area in a farm field. FIG. 12 is a perspective view schematically showing the positional relationship between a camera coordinate system Σc, a vehicle coordinate system Σv, a world coordinate system Σw, and a reference plane Re. FIG. 13 is a flowchart showing an example of image processing executed by an image processing device according to a second embodiment. FIG. 14 is a diagram showing an example of a boundary between the sky and a feature detected from a captured image. FIG. 15 is a flowchart showing a modified example of the second embodiment. FIG. 16 is a flowchart showing another modified example of the second embodiment. FIG. 17 is a block diagram showing an example of the configuration of an image processing device according to a third embodiment. FIG. 18 is a table showing an example of correspondence between a plurality of trained models stored in a memory and an environment. FIG. 19 is a flowchart showing an example of the operation of the image processing device according to the third embodiment. FIG. 20 is a flowchart showing an example of the operation of an image processing device in a case where a trained model is selected each time the agricultural machine changes direction.Fig. 21 is a flowchart showing an example of the model selection process in step S320. Fig. 22 is a flowchart showing a specific example of the operation of the image processing device in this embodiment. Fig. 23 is a diagram showing an example of the detection result of an object. Fig. 24 is a diagram showing an example of a construction vehicle.

[0014] (Definition of Terms) In this disclosure, a "work machine" refers to a machine used for a specific purpose such as agriculture or construction. In this disclosure, an "agricultural machine" refers to a work machine used for agricultural purposes. A "construction machine" refers to a work machine used for construction purposes. "Work" includes, for example, farm work, construction work, debris removal, snow removal, and the like. The work machine of this disclosure may be a mobile machine that can perform work while moving. Examples of agricultural machines include tractors, harvesters, rice transplanters, riding cultivators, vegetable transplanters, brush cutters, seed sowing machines, fertilizer applicators, agricultural mobile robots, and agricultural unmanned aerial vehicles (e.g., drones). Examples of construction machines include, for example, backhoes, wheel loaders, carriers, construction mobile robots, and construction unmanned aerial vehicles. An agricultural work vehicle such as a tractor or a combine harvester, or a construction work vehicle may function alone as a "work machine," or the entire work vehicle and an implement attached to or towed by the work vehicle may function as a single "work machine." Agricultural machinery performs agricultural tasks on the ground in fields, such as plowing, sowing seeds, pest control, fertilizing, planting crops, and harvesting. Construction machinery performs tasks such as transporting soil, rubble, and other materials at construction sites. These tasks are sometimes referred to as "ground work" or simply "work." When a vehicle-type work machine travels while performing work, it is sometimes referred to as "work travel."

[0015] "Autonomous driving" refers to controlling the movement of a work machine, such as an agricultural machine, through the action of a control device, without manual operation by a driver. Agricultural machines that perform autonomous driving are sometimes called "autonomous agricultural machines" or "robotic agricultural machines." During autonomous driving, not only the movement of the work machine but also the work operation (e.g., the operation of a work implement attached to the work machine) may be automatically controlled. When the work machine is a vehicle-type machine, the movement of the work machine through autonomous driving is referred to as "autonomous driving." The control device may control at least one of the steering, speed adjustment, and start and stop of movement required for the movement of the work machine. When controlling a work machine equipped with a work implement, the control device may control operations such as raising and lowering the work implement and starting and stopping its operation. Movement through autonomous driving may include not only movement of the work machine toward a destination along a predetermined route, but also movement in which the work machine tracks a tracking target. An autonomously driving work machine may move partially based on user instructions. Furthermore, an autonomously driving work machine may operate in a manual driving mode, in addition to an autonomous driving mode, in which movement is performed by manual operation by the driver. Steering a work machine by the action of a control device, without manual operation, is called "automatic steering." Part or all of the control device may be external to the work machine. Control signals, commands, data, and the like may be communicated between the work machine and a control device external to the work machine. An autonomously driven work machine may move autonomously while sensing the surrounding environment, without a human being being involved in controlling the movement of the work machine. A work machine capable of autonomous movement can travel unmanned within or outside a field (e.g., on a road). During autonomous movement, it may detect obstacles and take action to avoid them.

[0016] In this disclosure, the term "image acquisition device" refers to a device capable of acquiring images or similar information. The image acquisition device may be, for example, an imaging device such as a camera capable of acquiring images such as visible light images, infrared images, and ultraviolet images by photographing, a LiDAR sensor capable of acquiring point cloud image data, or a radar capable of acquiring image-like information using short-wavelength electromagnetic waves such as millimeter waves.

[0017] One example of a "controller" in this disclosure is a computing device that includes at least one processor and at least one memory that stores a computer program (code) that defines a control process executed by the processor. Another example of a "controller" is a computing device that includes a hardware accelerator, such as a field-programmable gate array (FPGA), an application-specific standard product (ASSP), or an application-specific integrated circuit (ASIC), configured to execute the control process.

[0018] Similarly, one example of an "image processing device" in this disclosure is a computing device that includes at least one processor and at least one memory that stores a computer program (code) that defines an image processing process to be performed by the processor. Another example of an "image processing device" is a computing device that includes a hardware accelerator, such as an FPGA or ASIC, configured to perform the image processing process.

[0019] In this disclosure, a "processor" refers to a hardware electronic circuit such as a CPU (Central Processing Unit), GPU (Graphics Processing Unit), DSP (Digital Signal Processor), ISP (Image Signal Processor), or NPU (Neural Network Processing Unit). A "memory" refers to a hardware electronic circuit such as a ROM (Read Only Memory) or RAM (Random Access Memory). Part of the memory may be a storage medium connected to the processor by wiring or a network. These hardware electronic circuits may be implemented by one or more integrated circuits (ICs) or large-scale integrated circuits (LSIs). Each functional unit or block and related components in the electronic circuit may be fabricated individually as a separate integrated circuit chip, or some or all of these functional units or blocks may be combined and fabricated as a single integrated circuit chip.

[0020] The program that defines the operation of the processor is designed to cause the processor to perform one or more functions, operations, steps, or processes in the embodiments of the present invention.

[0021] Hereinafter, embodiments of the present disclosure will be described. However, more detailed descriptions than necessary may be omitted. For example, detailed descriptions of well-known matters and redundant descriptions of substantially identical configurations may be omitted. This is to avoid unnecessary redundancy in the following description and to facilitate understanding by those skilled in the art. Note that the inventors provide the accompanying drawings and the following description to enable those skilled in the art to fully understand the present disclosure, and do not intend for them to limit the subject matter described in the claims. In the following description, components having the same or similar functions are designated by the same reference numerals.

[0022] The following embodiments are examples, and the technology of the present disclosure is not limited to the following embodiments. For example, the numerical values, shapes, materials, steps, step order, display screen layout, etc. shown in the following embodiments are merely examples, and various modifications are possible as long as no technical contradiction occurs. Furthermore, one aspect can be combined with another aspect.

[0023] Below, several embodiments in which the technology of the present disclosure is applied to an agricultural machine, which is an example of a work machine, are described. The various technologies described below for agricultural machines can also be applied to construction machines such as construction vehicles used at construction sites, work vehicles used at disaster sites, snowplows used in areas with heavy snowfall, and unmanned aerial vehicles (drones) that perform tasks such as transporting or monitoring goods.

[0024] (Embodiment 1) Fig. 1 is a block diagram showing a schematic configuration of an agricultural machine 100 according to an exemplary embodiment of the present disclosure. The agricultural machine 100 shown in Fig. 1 includes an imaging device 10, an image processing device 20, and a control device 30. The agricultural machine 100 is a work machine configured to perform agricultural work while moving. Although not shown in Fig. 1, the agricultural machine 100 may include various devices required for movement (e.g., running or flying), such as a power source such as an internal combustion engine or a drive electric motor, and a traveling device such as wheels with tires, or a propeller.

[0025] The imaging device 10 is a device such as a camera that captures images in the direction of movement of the agricultural machine 100. The imaging device 10 is attached to the agricultural machine 100 so as to capture images in the direction of movement of the agricultural machine 100 (for example, forward, backward, rightward, or leftward). The orientation of the imaging device 10 (i.e., the direction of the optical axis of the optical system in the imaging device 10) does not necessarily have to coincide with the direction of movement of the agricultural machine 100, and may be inclined relative to the direction of movement. For example, the imaging device 10 may be disposed facing diagonally downward relative to the front, rear, rightward, or leftward direction of the agricultural machine 100.

[0026] The agricultural machine 100 may be equipped with a plurality of imaging devices 10 mounted facing in different directions. The imaging devices 10 generate image data by capturing images while the agricultural machine 100 is moving. In an embodiment, the agricultural machine 100 may be configured to generate moving image data at a predetermined frame rate, such as 30 fps or 60 fps.

[0027] In the example of FIG. 1 , an imaging device 10 is used as an example of an image acquisition device. Instead of or in addition to the imaging device 10, a device capable of acquiring point cloud data similar to an image, such as a LiDAR sensor, may be used. Alternatively, a radar capable of acquiring distribution information of surrounding objects similar to an image may be used. Such a device capable of acquiring images or similar data may be used as an "image acquisition device." In this specification, the generation of moving image or still image data, or image-like data, by an image acquisition device is referred to as "acquiring an image."

[0028] The image processing device 20 is a computing device that processes images acquired by the imaging device 10. The image processing device 20 may include one or more processors and one or more memories. The image processing device 20 may be configured or programmed to perform processing to detect specific objects from images acquired by the imaging device 10 (hereinafter, sometimes referred to as "captured images"). The specific objects may be, for example, people, animals, other agricultural machinery, vehicles, crops, obstacles such as stones or rocks, or any combination of these obstacles. In one embodiment, the specific objects are people, and the image processing device 20 is configured or programmed to perform processing to detect people from captured images.

[0029] The control device 30 is a device that controls the operation of the agricultural machine 100. The control device 30 may be, for example, a computing device such as an electronic control unit (ECU). In this embodiment, the control device 30 controls the operation of the agricultural machine 100 based on the object detection result by the image processing device 20. For example, the control device 30 may be configured to stop the movement of the agricultural machine 100 or to cause an audio output device such as a buzzer to emit a warning sound when a specific object is detected by the image processing device 20. Such control makes it possible to avoid a collision between the agricultural machine 100 and the object or to alert the object (e.g., a person). If the agricultural machine 100 has an automatic driving function, the control device 30 may also be configured to control the automatic driving.

[0030] FIG. 2 is a block diagram showing an example of the configuration of an image processing device 20. The image processing device 20 shown in FIG. 2 includes one or more processors 22 and one or more memories 24. The processor 22 is an electronic circuit (arithmetic circuit) that performs arithmetic processing, such as a CPU, a GPU, or an NPU. The image processing device 20 may include multiple processors. The multiple processors may collaboratively perform the image processing described below. The memory 24 may be, for example, a ROM such as an EPROM (Erasable Programmable Read-Only Memory) or an EEPROM (Electrically Erasable Programmable Read-Only Memory), or a RAM such as a DRAM (Dynamic Random Access Memory) or an SRAM (Static Random Access Memory). The memory 24 stores a computer program 25 executed by the processor 22 and a trained model 27 for detecting objects from an input image. The program 25 and the trained model 27 may be stored in a distributed manner across multiple memories. The processor 22 executes a computer program 25 stored in the memory 24 to perform processing for detecting a specific object from a captured image. Note that instead of the processor 22 such as a CPU or GPU, a hardware accelerator such as an FPGA or ASIC configured to perform the image processing of this embodiment may be provided in the image processing device 20. Such a hardware accelerator can replace the processing performed by the processor in the following description.

[0031] The trained model 27 may be a machine learning model (hereinafter also referred to as an "AI model") trained using an algorithm based on machine learning or artificial intelligence (AI) technology, such as a convolutional neural network (CNN) or a vision transformer (ViT). The trained model 27 may be a model that detects objects from an image based on an object detection algorithm, such as SSD (Single Shot Multibox Detector), YOLO (You Only Look Once), R-CNN (Regions with Convolutional Neural Network), Fast R-CNN, Faster R-CNN, or RetinaNet.

[0032] The image processing device 20 performs necessary preprocessing on the captured image generated by the imaging device 10 to generate an input image to be input to the model 27. The image processing device 20 inputs the input image to the model 27, thereby detecting a specific object (e.g., a person) from the image.

[0033] The image processing device 20 in this embodiment does not detect an object from the entire captured image acquired by the imaging device 10, but detects the object from a region of interest, which is a part of the captured image. Specifically, the image processing device 20 extracts a partial image indicating the region of interest from the captured image and performs predetermined preprocessing on the partial image to generate an input image for the trained model 27. At this time, the image processing device 20 dynamically changes at least one of the position and size of the region of interest depending on the operating state of the agricultural machine 100. For example, the image processing device 20 shifts the region of interest to the right when the agricultural machine 100 is turning right, and shifts the region of interest to the left when the agricultural machine 100 is turning left. Alternatively, the image processing device 20 may shift the region of interest downward when the agricultural machine 100 is tilted upward, and shift the region of interest upward when the agricultural machine 100 is tilted downward. The image processing device 20 may also change the size of the region of interest depending on the moving speed of the agricultural machine 100. For example, the region of interest may be made smaller as the moving speed of the agricultural machine 100 increases. This type of processing enables more accurate detection of specific objects such as people.

[0034] Generally, when a machine learning model is used to detect a specific object from an image, the image is resized to a predetermined number of pixels (e.g., 300 x 300 pixels) that matches the model being used, and then input to the model. The number of pixels in the resized input image is generally smaller than the number of pixels in the original image. Therefore, if the original image is large, the loss of resolution when converted to the input image becomes significant, which can significantly degrade the object detection performance.

[0035] Therefore, the image processing device 20 in this embodiment extracts a partial image indicating a region of interest determined based on the operating state of the agricultural machine 100 from the captured image, and generates an input image based on the partial image. As the region of interest, an area in which an object such as a person may obstruct the movement of the agricultural machine 100 if present is adaptively selected according to the operating state of the agricultural machine 100. When the partial image is resized to a predetermined number of pixels that matches the model, the reduction in resolution of the region of interest in which an object may exist is suppressed compared to when the original captured image is resized. This makes it possible to more accurately detect objects that may obstruct the movement of the agricultural machine 100.

[0036] Fig. 3 is a flowchart showing an example of the operation of the image processing device 20. In the example shown in Fig. 3, the image processing device 20 executes the operations of steps S110 to S150 while the agricultural machine 100 is in operation, thereby detecting a specific object from a captured image and transmitting the detection result to the control device 30. The operations of steps S110 to S150 are executed repeatedly while the agricultural machine 100 is in operation.

[0037] In step S110, the image processing device 20 acquires the image captured by the imaging device 10 and information indicating the operating state of the agricultural machine 100. The information indicating the operating state of the agricultural machine 100 may be, for example, information regarding at least one of the travel speed, turning state, and tilt state of the agricultural machine 100. The image processing device 20 may be configured to acquire the information indicating the operating state of the agricultural machine 100, for example, from the control device 30. The control device 30 may be configured to generate the information indicating the operating state of the agricultural machine 100 based on signals from one or more sensors mounted on the agricultural machine 100.

[0038] In step S120, the image processing device 20 determines a region of interest from the image acquired by the imaging device 10 based on the operating state of the agricultural machine 100, and extracts a partial image showing the region of interest. The region of interest is an area in the image that is the target of object detection processing. The image processing device 20 may be configured to acquire information regarding at least one of the traveling speed, turning state, and tilt state of the agricultural machine 100, and change the region of interest based on the information. For example, the image processing device 20 may make the region of interest smaller as the traveling speed of the agricultural machine 100 increases. Alternatively, the image processing device 20 may shift the region of interest to the right when the agricultural machine 100 turns right, and shift the region of interest to the left when the agricultural machine 100 turns left. In this case, the image processing device 20 may shift the region of interest more as the steering angle of the agricultural machine 100 during turning increases. The agricultural machine 100 may also be equipped with an inclination sensor that measures the amount of tilt of the agricultural machine 100. In this case, the image processing device 20 may change the position of the region of interest in accordance with the measured amount of tilt. In one embodiment, the tilt sensor measures the pitch angle of the agricultural machine 100 as the amount of tilt. In this case, the image processing device 20 may shift the region of interest up or down in accordance with the measured pitch angle. For example, when the agricultural machine 100 is tilted upward, such as when the agricultural machine 100 is traveling uphill, much of the upper side of the captured image may be an area corresponding to the sky. In this case, the object to be detected may be located at the bottom of the image, so the image processing device 20 may shift the region of interest downward. Conversely, when the agricultural machine 100 is tilted downward, the image processing device 20 may shift the region of interest upward.

[0039] In step S130, the image processing device 20 generates an input image to be input to the trained model 27 based on the extracted partial image. The image processing device 20 generates the input image by preprocessing, which includes compressing (resizing) the number of pixels of the partial image to a predetermined number of pixels. In addition to image resizing, the preprocessing may include various processes such as normalization, color space conversion, and noise removal. As a result, an input image is generated that shows a region of interest that is appropriately selected depending on the operating state of the agricultural machine 100.

[0040] In step S140, the image processing device 20 detects a specific object (e.g., a person) by inputting the input image into the trained model 27 stored in the memory 24. For example, if a specific object is present in the image, the image processing device 20 may be configured to output, as a detection result, coordinate information indicating the position of a rectangular frame (called a "bounding box") surrounding the area in which the object is present, and information on the width and height of the bounding box. Alternatively, the image processing device 20 may output, as a detection result, a signal indicating whether or not a specific object is present in the input image. The image processing device 20 may determine that an object is present in the input image if the distance from the imaging device 10 to the object, calculated based on the position of the bounding box in the input image, is less than a threshold.

[0041] In step S150, the image processing device 20 transmits the object detection result to the control device 30. For example, the image processing device 20 may transmit a signal indicating whether or not a specific object is present in the input image to the control device 30. Alternatively, the image processing device 20 may transmit a signal indicating that a specific object is detected to the control device 30 only when the specific object is detected from the input image. Furthermore, the image processing device 20 may not only detect a specific object from the input image, but also estimate the distance from the imaging device to the object based on the input image, and transmit a signal indicating the presence of the object to the control device 30 only when the distance is less than a threshold.

[0042] When the control device 30 receives a signal indicating the detection result from the image processing device 20, it controls the operation of the agricultural machine 100 in accordance with the detection result. For example, when the control device 30 receives a signal indicating that an object has been detected, it can be configured to stop the movement of the agricultural machine 100 or cause a buzzer or speaker to emit a warning sound. Such operations can avoid a collision between the agricultural machine 100 and the object, or alert the object (e.g., a person) or the rider of the agricultural machine 100.

[0043] Below, an embodiment in which the technology of the present disclosure is applied to a harvester, which is an example of an agricultural machine 100, will be described. The technology of the present disclosure is not limited to harvesters, but can also be applied to other types of agricultural machines, such as tractors, transplanters, or agricultural drones. The technology of the present disclosure can also be applied to work machines used for purposes other than agriculture (for example, construction vehicles, snowplows, or mobile work robots). In the following description, an imaging device (camera) is used as an example of an image acquisition device, but other types of image acquisition devices, such as a LiDAR sensor capable of acquiring point cloud data similar to images, or a radar capable of acquiring distance distribution information of surrounding objects similar to images, may also be used.

[0044] [1. Configuration] Fig. 4 is a side view schematically illustrating an example of an agricultural machine 100. The agricultural machine 100 in this embodiment is a harvester such as a combine harvester. The agricultural machine 100 performs tasks such as harvesting crops in a field, threshing the harvested crops, and discharging the threshed crops. The crops are plants from which grains such as rice, wheat, corn, and soybeans can be harvested. The symbols F, B, U, and D shown in Fig. 4 represent front, rear, top, and bottom, respectively.

[0045] The agricultural machine 100 includes a machine body 101 and a traveling device 102. The traveling device 102 shown in Fig. 4 includes a plurality of wheels (crawlers) equipped with tracks. The traveling device 102 may include wheels with tires instead of crawlers. A cabin 110 is provided above the machine body 101.

[0046] A reaping device 103 that reaps crops is provided in front of the traveling device 102, and its height is adjustable. A reel 109 that raises the stalks of the crops is provided above the reaping device 103, and its height is adjustable. A threshing device 105 and a tank 106 that stores the harvested crops are arranged side by side in the left-right direction behind the cabin 110. The threshing device 105 threshes the harvested crops. The tank 106 stores the harvested crops, such as grains, obtained by threshing. A straw waste processing device 108 is provided behind the threshing device 105. The straw waste processing device 108 finely cuts the stalks and other parts of the harvested crops after the grains and other harvested crops have been removed, and discharges them outside.

[0047] A conveying device 104 for conveying the harvested crops is provided between the reaping device 103 and the threshing device 105. A discharge device 107 for discharging the harvested crops from the tank 106 is provided in the tank 106. The harvested crops are discharged to the outside from a discharge port 117 at the tip of the cylindrical discharge device 107. The discharge device 107 is capable of raising and lowering and rotating, and the position of the discharge port 117 can be changed. The configurations and operations of the various devices that perform the harvesting operation, such as the reaping device 103, conveying device 104, threshing device 105, discharge device 107, straw waste treatment device 108, and reel 109, are well known, so detailed explanations thereof will be omitted here.

[0048] The agricultural machine 100 in this embodiment can operate in both a manual operation mode and an automatic operation mode. In the automatic operation mode, the agricultural machine 100 can travel unmanned while performing an operation to harvest crops in a field.

[0049] 4, the agricultural machine 100 includes a prime mover (engine) 111 and a transmission 112. Inside the cabin 110, a driver's seat, control levers, an operation terminal (terminal monitor), and a group of switches for operation are provided.

[0050] The agricultural machine 100 is equipped with a plurality of sensing devices for sensing the environment around the agricultural machine 100. In the example shown in Fig. 4 , the plurality of sensing devices include a laser sensor 125, a plurality of cameras 126, and a plurality of millimeter-wave radars 127.

[0051] The laser sensor 125 is a distance measuring device that can measure the distance to a reflection point by emitting laser light and detecting the reflected light, and is also referred to as a LiDAR sensor. The laser sensor 125 can obtain information on the distance distribution to surrounding features by changing the emission direction of the laser light. The laser sensor 125 illustrated in FIG. 4 is disposed at the front of the agricultural machine 100. The laser sensor 125 may also be provided at the side or rear of the agricultural machine 100. The laser sensor 125 may include a light source that generates laser light, a detector that detects reflected light, and a processing circuit that processes a signal of the detected reflected light. The laser sensor 125 may include a beam scanner that changes the direction of the emitted light beam. The laser sensor 125 may be configured to generate sensor data such as point cloud data that indicates the distance and direction to each measurement point of an object present in the environment surrounding the agricultural machine 100, or three-dimensional or two-dimensional coordinate values ​​of each measurement point. The sensor data output from the laser sensor 125 is processed by a control device of the agricultural machine 100. The control device can measure the height or degree of lodging of crops present around the agricultural machine 100 based on the sensor data, and adjust the height or speed of the reaping device 103 according to the height or degree of lodging of the crops. It is also possible to use the point cloud data output from the laser sensor 125 to detect objects.

[0052] The camera 126 is an example of an imaging device that captures images of the environment around the agricultural machine 100 and generates image data. The cameras 126 may be installed, for example, on the front, rear, left, and right sides of the agricultural machine 100. The images captured by the cameras 126 are sent to a control device mounted on the agricultural machine 100. The images are used to detect obstacles, such as people, present around the agricultural machine 100 by image processing during autonomous driving.

[0053] The millimeter-wave radar 127 is a sensor for detecting metal objects such as vehicles present in the vicinity of the agricultural machine 100. In the example shown in Fig. 4, two millimeter-wave radars 127 are provided at the front and rear of the agricultural machine 100. The millimeter-wave radars 127 may be arranged at other locations, such as at the sides of the agricultural machine 100.

[0054] The agricultural machine 100 further includes a GNSS unit 120. The GNSS unit 120 includes a GNSS receiver and functions as a positioning device that acquires positioning data of the agricultural machine 100. The GNSS receiver may include an antenna that receives signals from GNSS satellites and a processor that calculates the position of the agricultural machine 100 based on the signals received by the antenna. The GNSS unit 120 receives satellite signals transmitted from a plurality of GNSS satellites and performs positioning based on the satellite signals. GNSS is a general term for satellite positioning systems such as GPS (Global Positioning System), QZSS (Quasi-Zenith Satellite System, for example, Michibiki), GLONASS, Galileo, and BeiDou. In this embodiment, the GNSS unit 120 is provided on top of the cabin 110, but may be provided in another location.

[0055] The GNSS unit 120 may include an inertial measurement unit (IMU). Signals from the IMU can be used to complement the position data. The IMU can measure the tilt and minute movements of the agricultural machine 100. By complementing the position data based on satellite signals with data acquired by the IMU, the positioning performance can be improved. The IMU may be provided at a location different from that of the GNSS unit 120.

[0056] The prime mover 111 may be, for example, a diesel engine. An electric motor may be used instead of a diesel engine. The transmission 112 can change the propulsive force and travel speed of the agricultural machine 100 by changing the speed. The transmission 112 can also switch the agricultural machine 100 between forward and reverse travel.

[0057] In a configuration in which the agricultural machine 100 is equipped with a crawler-type traveling device 102, the traveling direction of the agricultural machine 100 can be changed by varying the rotational speeds of the left and right wheels equipped with caterpillars or by varying the rotational directions of the left and right wheels. In a configuration in which the agricultural machine 100 is equipped with traveling devices including tires, the control device of the agricultural machine 100 can change the traveling direction of the agricultural machine 100 by controlling the power steering device to change the turning angle (steering angle) of the steering wheels.

[0058] The agricultural machine 100 shown in Fig. 4 is capable of being driven by a driver, but may be capable of being driven only in an unmanned manner. In that case, components that are necessary only for driven operation, such as the cabin 110, a steering device, and a driver's seat, may not be provided in the agricultural machine 100. The unmanned agricultural machine 100 can travel autonomously or by being remotely controlled by a user.

[0059] Fig. 5 is a block diagram showing an example configuration of the agricultural machine 100. The agricultural machine 100 illustrated in Fig. 5 includes a GNSS unit 120, a laser sensor 125, a camera 126, a millimeter-wave radar 127, a terminal monitor 131, an operation switch group 132, a buzzer 133, a drive device 140, a power transmission mechanism 141, a light 142, a sensor group 150, a control system 160, and a communication device 190. These components are connected to each other via a bus so that they can communicate with each other.

[0060] The GNSS unit 120 includes a GNSS receiver 121, an RTK receiver 122, an inertial measurement unit (IMU) 123, and a processing circuit 124. The sensor group 150 includes various sensors such as a vehicle speed sensor 151, a steering angle sensor 152, and an illuminance sensor 153. The control system 160 includes a storage device 164 and electronic control units (ECUs) 165, 166, and 167. Fig. 5 shows components that are relatively highly related to the autonomous driving operation of the agricultural machine 100, and does not show other components.

[0061] The GNSS receiver 121 included in the GNSS unit 120 receives satellite signals transmitted from multiple GNSS satellites and generates GNSS data based on the satellite signals. The GNSS data is generated in a predetermined format, such as the NMEA-0183 format. The GNSS data may include, for example, values ​​indicating the identification number, elevation angle, azimuth angle, and reception strength of each satellite from which the satellite signal is received.

[0062] The GNSS unit 120 illustrated in FIG. 5 is capable of positioning the agricultural machine 100 using RTK (Real Time Kinematic)-GNSS. Positioning using RTK-GNSS utilizes correction signals transmitted from a reference station in addition to satellite signals transmitted from multiple GNSS satellites. The reference station may be installed near the field where the agricultural machine 100 performs work (for example, within 10 km of the agricultural machine 100). The reference station generates correction signals, for example, in RTCM format, based on satellite signals received from multiple GNSS satellites and transmits them to the GNSS unit 120. The RTK receiver 122 includes an antenna and a modem and receives the correction signals transmitted from the reference station. The processing circuit 124 of the GNSS unit 120 corrects the positioning results obtained by the GNSS receiver 121 based on the correction signals. Using RTK-GNSS makes it possible to perform positioning with an accuracy of, for example, a few centimeters. Position data including information on latitude, longitude, and altitude is acquired by highly accurate positioning using the RTK-GNSS. The GNSS unit 120 calculates the position of the agricultural machine 100, for example, at a frequency of approximately 1 to 10 times per second.

[0063] The positioning method is not limited to RTK-GNSS, and any positioning method (such as interferometric positioning or relative positioning) that can obtain position data with the required accuracy can be used. For example, positioning may be performed using a virtual reference station (VRS) or a differential global positioning system (DGPS). If position data with the required accuracy can be obtained without using a correction signal transmitted from a reference station, the position data may be generated without using a correction signal. In this case, the GNSS unit 120 does not need to be equipped with an RTK receiver 122.

[0064] The IMU 123 may include a three-axis acceleration sensor and a three-axis gyroscope. The IMU 123 may also include an orientation sensor such as a three-axis geomagnetic sensor. The IMU 123 functions as a motion sensor and can output signals indicating various quantities such as the acceleration, velocity, displacement, and attitude of the agricultural machine 100. The processing circuit 124 can estimate the position and orientation of the agricultural machine 100 with higher accuracy based on the signals output from the IMU 123 in addition to the satellite signals and correction signals. The signals output from the IMU 123 can be used to correct or complement the position calculated based on the satellite signals and correction signals. The IMU 123 outputs signals at a higher frequency than the GNSS receiver 121. Using the high-frequency signals, the processing circuit 124 can measure the position and orientation of the agricultural machine 100 at a higher frequency (e.g., 10 Hz or higher). A three-axis acceleration sensor and a three-axis gyroscope may be provided separately instead of the IMU 123. Furthermore, the IMU 123 may be provided as a device separate from the GNSS unit 120. The IMU 123 serves as an inclination sensor that measures the amount of inclination (for example, pitch angle, roll angle, yaw angle) of the agricultural machine 100 with respect to a reference attitude.

[0065] The camera 126 is an example of an imaging device that captures images of the environment around the agricultural machine 100. The camera 126 includes an image sensor, such as a charge coupled device (CCD) or a complementary metal oxide semiconductor (CMOS). The camera 126 may also include an optical system including one or more lenses and a signal processing circuit. The camera 126 captures images of the environment around the agricultural machine 100 while the agricultural machine 100 is traveling and generates image (e.g., video) data. The camera 126 can capture video at a frame rate of, for example, 3 frames per second (fps) or more. The images generated by the camera 126 are used to detect obstacles such as people. The images generated by the camera 126 may be used for positioning or remote monitoring. Multiple cameras 126 may be provided at different positions on the agricultural machine 100, or a single camera may be provided. A visible camera that generates visible light images and an infrared camera that generates infrared images may be separately provided. Both a visible camera and an infrared camera may be provided. An infrared camera can be used to detect obstacles at night.

[0066] The millimeter-wave radar 127 is provided to detect obstacles containing metal, such as vehicles, that are present in the vicinity of the agricultural machine 100. The millimeter-wave radar 127 outputs a signal indicating the presence of an obstacle when an object is present closer than a predetermined distance from the millimeter-wave radar 127. As shown in FIG. 4 , multiple millimeter-wave radars 127 may be provided at different positions on the agricultural machine 100. By providing multiple millimeter-wave radars 127, it is possible to reduce blind spots in monitoring obstacles in the vicinity of the agricultural machine 100.

[0067] The buzzer 133 is an audio output device that emits a warning sound to notify of an abnormality. For example, the buzzer 133 emits the warning sound when an obstacle is detected during autonomous driving. The buzzer 133 is controlled by the control system 160.

[0068] The drive device 140 includes various devices necessary for driving the agricultural machine 100 to travel, such as the prime mover 111 and the transmission 112. The prime mover 111 may be equipped with an internal combustion engine such as a diesel engine. The drive device 140 may be equipped with an electric motor for traction instead of or in addition to the internal combustion engine.

[0069] The power transmission mechanism 141 transmits the power generated by the prime mover 111 to various devices that perform harvesting operations. The devices that perform harvesting operations include the reaping device 103, the transport device 104, the threshing device 105, the discharge device 107, the straw waste treatment device 108, and the reel 109. The agricultural machine 100 may be provided with a power source (such as an electric motor) that supplies power to at least one of these devices that perform harvesting operations, separate from the prime mover 111.

[0070] The light 142 is a device, such as a headlight or a work light, that illuminates the surroundings of the agricultural machine 100. A plurality of lights 142 may be mounted on the agricultural machine 100. The light 142 includes one or more light sources. Each light source may be, for example, a light-emitting diode (LED), a halogen lamp, or a xenon lamp.

[0071] The vehicle speed sensor 151 is a sensor that measures the traveling speed of the agricultural machine 100. The vehicle speed sensor 151 measures, for example, the rotational speed of a wheel or an axle, and calculates the vehicle speed based on the measurement value. The steering angle sensor 152 is a sensor that measures the steering angle of the steering wheel. The illuminance sensor 153 is a sensor that measures the illuminance of the surrounding environment, and may be disposed outside or inside the cabin of the agricultural machine 100.

[0072] The storage device 164 includes one or more storage media, such as a flash memory or a magnetic disk. The storage device 164 stores various types of data generated by the GNSS unit 120, the laser sensor 125, the camera 126, the millimeter-wave radar 127, the sensor group 150, and the ECUs 165, 166, and 167. The data stored in the storage device 164 may include map data of an area including a field where agricultural work is performed by the agricultural machine 100, and data of a target route for autonomous driving.

[0073] The ECU 165 controls the overall operation of the agricultural machine 100. The ECU 165 controls the operation of the agricultural machine 100 by controlling the prime mover 111, the transmission 112, the travel device 102, the power transmission mechanism 141, etc., which are included in the drive device 140.

[0074] The ECU 166 performs calculations and control to achieve autonomous driving based on data output from the GNSS unit 120, the laser sensor 125, the camera 126, the millimeter-wave radar 127, and the sensor group 150. For example, the ECU 166 determines the position and orientation of the agricultural machine 100 based on data output from the GNSS unit 120. During autonomous driving, the ECU 166 performs calculations necessary for the agricultural machine 100 to travel along a set target route based on the position and orientation of the agricultural machine 100. The ECU 166 may execute processing to generate a target route from the start point of movement of the agricultural machine 100 to the destination point.

[0075] The ECU 167 performs processing to detect a specific object (e.g., a person) located in the vicinity of the agricultural machine 100 based on an image acquired by the camera 126. The ECU 167 in this embodiment is an edge computing device mounted on the agricultural machine 100, but at least some of the functions of the ECU 167 may be executed by a computer external to the agricultural machine 100 (e.g., a server computer on the cloud).

[0076] Through the operation of these ECUs, the control system 160 realizes automatic driving and crop harvesting operations. During automatic driving, the control system 160 controls the drive unit 140 based on the measured position and orientation of the agricultural machine 100 and the target route. In this way, the control system 160 can cause the agricultural machine 100 to travel along the target route.

[0077] The multiple ECUs included in the control system 160 can communicate with each other according to a vehicle bus standard such as CAN (Controller Area Network). Instead of CAN, a faster communication method such as in-vehicle Ethernet (registered trademark) may be used. While each of the ECUs 165, 166, and 167 is shown as an individual block in FIG. 3 , the functions of each of these ECUs may be realized by multiple ECUs. An in-vehicle computer that integrates at least some of the functions of the ECUs 165, 166, and 167 may also be provided. The control system 160 may include ECUs other than the ECUs 165, 166, and 167, and any number of ECUs may be provided depending on the functions. Each ECU includes a processing circuit including one or more processors.

[0078] In the example shown in Fig. 5, the camera 126 functions as the imaging device 10 shown in Fig. 1. The ECU 167 functions as the image processing device 20 shown in Fig. 1. The combination of the ECUs 165 and 166 functions as the control device 30 shown in Fig. 1.

[0079] The communication device 190 is a device that includes a circuit for communicating with an external device. The communication device 190 includes a circuit for wireless communication. The communication device 190 may include an antenna and a communication circuit for transmitting and receiving signals via a network, for example, to and from an external terminal device or an external server computer. The network may include, for example, a cellular mobile communication network such as 3G, 4G, or 5G, and the Internet. The communication device 190 may have a function for communicating with a mobile terminal used by a monitor who is located near the agricultural machine 100. Communication with such a mobile terminal may be performed in accordance with any wireless communication standard, such as Wi-Fi (registered trademark), cellular mobile communication such as 3G, 4G, or 5G, or Bluetooth (registered trademark).

[0080] The terminal monitor 131 is a terminal through which a user performs operations related to the traveling and work of the agricultural machine 100, and is also referred to as a virtual terminal (VT). The terminal monitor 131 may include a display device such as a touch screen and / or one or more buttons. The display device may be, for example, a liquid crystal display or an organic light-emitting diode (OLED) display. By operating the terminal monitor 131, a user can perform various operations, such as switching the autonomous driving mode on / off, recording or editing map data of the field, setting a target route, setting the type of crop, and setting the type of work. At least some of these operations can also be achieved by operating the operation switch group 132. The terminal monitor 131 may be configured to be detachable from the agricultural machine 100. A user located remote from the agricultural machine 100 may operate the detached terminal monitor 131 to control the operation of the agricultural machine 100. The user may control the operation of the agricultural machine 100 by operating a computer on which necessary application software is installed, instead of the terminal monitor 131.

[0081] The operation switch group 132 includes a plurality of switches for operating the agricultural machine 100. In this specification, the term "switch" broadly refers to devices used by the driver for operation, such as levers, pedals, and buttons. The operation switch group 132 may include, for example, a switch for switching between automatic driving mode and manual driving mode, a switch for switching between forward and reverse, an accelerator pedal, a brake pedal, a lever for changing gears, a switch for turning lights on and off, and the like.

[0082] 2. Operation Next, the operation of the agricultural machine 100 will be described.

[0083] 6 is a diagram showing an example of a route that the agricultural machine 100 travels while harvesting crops in a field 70. The agricultural machine 100 of this embodiment harvests crops while traveling in the field 70 in an automatic driving manner. In the field 70, the agricultural machine 100 performs an operation of harvesting crops while traveling along a set target route 74. The automatic driving control ECU 166 performs steering control of the agricultural machine 100 so as to eliminate deviations between the position and orientation of the agricultural machine 100 determined based on data output from the GNSS unit 120 and the position and orientation of the target route 74. This makes it possible to cause the agricultural machine 100 to travel along the target route 74.

[0084] In the example shown in FIG. 6 , the field 70 includes a work area 71 where the agricultural machine 100 harvests crops, and a headland 72 located near the outer periphery of the field 70. Map data (also referred to as a "field map") indicating which areas of the field 70 correspond to the work area 71 and which areas correspond to the headland 72, and a target route 74, can be determined by the ECU 166 for automatic driving control. For example, when the agricultural machine 100 performs harvesting while manually traveling along a route 73 on the outermost periphery of the work area 71, the ECU 166 generates a field map based on the trajectory of the route 73 and generates a target route 74 for automatic traveling inside the route 73. As a result, from the second lap onwards, the agricultural machine 100 can perform work traveling automatically (e.g., unmanned) under automatic driving control by the ECU 166. The agricultural machine 100 automatically travels along the target route 74 as shown in FIG. 6 . 6 is merely an example, and the method for determining the target route 74 is arbitrary. Furthermore, the method for generating the field map and the target route 74 is not limited to the above-described method. For example, the user may set the field map and the target route 74 by operating the terminal monitor 131.

[0085] In this embodiment, the agricultural machine 100 performs an operation of detecting obstacles using the camera 126 and the millimeter-wave radar 127 while traveling for work. The camera 126 is used mainly to detect specific objects such as people. The millimeter-wave radar 127 is used mainly to detect metal objects such as other vehicles. In this embodiment, the ECU 167 performs image processing based on the captured image acquired by the camera 126, thereby enabling highly accurate detection of people in the field 70 where crops exist. The control system 160 in this embodiment is configured to detect people and vehicles but not to react to obstacles such as birds that have little impact on traveling for work.

[0086] FIG. 7 is a diagram schematically illustrating the detection of an object 76 (a person in this example) using the camera 126 mounted on the agricultural machine 100. The symbols F, B, R, and L shown in FIG. 7 represent the front, rear, right, and left, respectively. The dashed lines in FIG. 7 represent an example of the range captured by the camera 126. As shown in FIG. 7 , when the object 76 is present in the traveling direction of the agricultural machine 100, the ECU 167 detects the object 76 from the image captured by the camera 126 and transmits a signal indicating the presence of the object 76 to the ECU 165. For example, the ECU 167 may calculate the position of the object 76 in a coordinate system fixed to the ground based on the position of the object 76 in the image and information about the position and orientation of the camera 126 on the agricultural machine 100, and transmit a signal indicating the presence of the object 76 to the ECU 165 when the distance between that position and the agricultural machine 100 or the camera 126 is less than a threshold. Note that measurement values ​​from another distance measuring sensor, such as the laser sensor 125, may be used to determine the distance. However, in the agricultural machine 100 that can be used to harvest tall crops (e.g., rice) as in this embodiment, there are cases where an object 76, such as a person, that is among the crops cannot be accurately detected using a method that uses another distance measuring sensor, such as the laser sensor 125. Even in such cases, the distance to the object 76 can be more accurately estimated by using the captured image.

[0087] When the ECU 165 receives a signal indicating the presence of an object, it controls the drive device 140 to stop the traveling of the agricultural machine 100 and causes the buzzer 133 to emit a warning sound. Note that if an obstacle such as a vehicle is detected within a predetermined distance from the agricultural machine 100 based on the signal output from the millimeter wave radar 127, the ECU 165 also stops the traveling of the agricultural machine 100 and causes the buzzer 133 to emit a warning sound. Through such control, it is possible to avoid a collision between the agricultural machine 100 and the obstacle.

[0088] The ECU 167 in this embodiment corresponds to the image processing device 20 shown in Fig. 1. That is, the ECU 167 has the configuration shown in Fig. 2 and executes the operation shown in Fig. 3. The image processing device 20 can detect an object 76, such as a person, with high accuracy by appropriately extracting a partial image to be input to the trained model 27 from the image captured by the camera 126, which has a wide field of view, in accordance with the operating state of the agricultural machine 100. This processing will be described in more detail below.

[0089] FIG. 8 is a diagram schematically illustrating the flow of processing executed by the ECU 167 (i.e., the image processing device). In the example illustrated in FIG. 8 , the ECU 167 first acquires a captured image captured by the camera 126 (step S801). Next, the ECU 167 determines a region of interest within the captured image that is to be the target of object detection processing (step S802). The ECU 167 determines the region of interest by referring to various information indicating the operating state of the agricultural machine 100. In the example illustrated in FIG. 8 , the ECU 167 also determines the region of interest by referring to map data (field map) of an area including the field where agricultural work is performed by the agricultural machine 100. The information indicating the operating state of the agricultural machine 100 may include, for example, information indicating the steering angle or steering operation value measured by the steering angle sensor 152, information indicating the speed measured by the vehicle speed sensor 151, and information indicating the amount of tilt of the agricultural machine 100 measured by the IMU 123 (i.e., the tilt sensor). The ECU 167 determines the position and size of the region of interest in the captured image based on at least part of this information.

[0090] The ECU 167 cuts out a partial image corresponding to the determined region of interest from the captured image (step S803). The ECU 167 generates an input image by performing predetermined preprocessing (e.g., image resizing, normalization, noise removal, etc.) on the partial image. The ECU 167 inputs the input image to an AI model (i.e., a trained model) to execute processing to detect a specific object, and outputs a signal indicating the detection result to the ECU 165.

[0091] Below, some specific examples of the process shown in FIG. 8 will be described.

[0092] Fig. 9A shows an example of a captured image acquired by the camera 126. When the agricultural machine 100 is traveling to perform work, the camera 126 captures images at a predetermined frame rate, thereby repeatedly acquiring captured images such as those shown in Fig. 9A. The captured image shown in Fig. 9A shows an object 76 (a person in this example). The ECU 167 extracts a partial image indicating a region of interest determined based on the operating state of the agricultural machine 100 from such a captured image.

[0093] FIG. 9B shows an example of a region of interest 77 that can be selected when the agricultural machine 100 is turning right. FIG. 9C shows a partial image corresponding to the region of interest 77 shown in FIG. 9B. When the agricultural machine 100 is turning right, the ECU 167 determines a region on the right side of the captured image as the region of interest 77, as shown in FIG. 9B, and extracts a partial image showing the region of interest 77, as shown in FIG. 9C. The ECU 167 performs preprocessing, such as pixel reduction (resizing), on the partial image shown in FIG. 9C to generate an input image to be input to the trained model 27. The ECU 167 inputs the generated input image into the trained model 27, thereby detecting an object 76. As described above, any object detection algorithm, such as SSD, YOLO, or R-CNN, can be used for this object detection processing.

[0094] Fig. 9D shows an example of a region of interest 77 that can be selected when the agricultural machine 100 is turning left. Fig. 9E shows a partial image corresponding to the region of interest 77 shown in Fig. 9D. Contrary to the above example, when the agricultural machine 100 is turning left, the ECU 167 extracts a partial image that defines a region closer to the left in the captured image as the region of interest 77.

[0095] 9B to 9E, the ECU 167 shifts the region of interest 77 to the right when the agricultural machine 100 turns right, and shifts the region of interest 77 to the left when the agricultural machine 100 turns left. This processing makes it possible to more accurately detect the object 76 that may affect the traveling of the agricultural machine 100 when turning right or left.

[0096] In the conventional technology, a relatively small input image (e.g., 300 × 300 pixels) generated by preprocessing a relatively large captured image (e.g., 1280 × 960 pixels) as shown in Fig. 9A is input to an AI model. In this case, the image resolution is significantly reduced, which can cause a problem of reduced accuracy in detecting objects, especially at the far end or at the edge of the image.

[0097] In contrast, in this embodiment, a partial image corresponding to a region of interest 77 where an object that may affect the work travel of the agricultural machine 100 may exist is first extracted from the captured image, and the partial image is preprocessed to generate an input image, which is then input to the trained model. The number of pixels in the input image is the same as in the conventional example (e.g., 300 x 300 pixels), but because the range captured in the input image is narrower, a decrease in image resolution is suppressed. This makes it possible to improve the accuracy of object detection.

[0098] The number of pixels in the area cut out as a partial image depends on the operating state of the agricultural machine 100, but may be, for example, 2 / 3 or less, 1 / 2 or less, 1 / 3 or less, or 1 / 4 or less of the number of pixels in the original captured image.

[0099] When performing the above processing, the ECU 167 may shift the region of interest 77 more significantly as the steering angle during turning of the agricultural machine 100 increases. In other words, the region of interest 77 may be shifted more to the right as the magnitude of the steering angle during a right turn increases, and the region of interest 77 may be shifted more to the left as the magnitude of the steering angle during a right turn increases. The ECU 167 can acquire information about the steering angle at that time from the measurement value of the steering angle sensor 152. Data such as a table indicating the relationship between the magnitude of the steering angle during a right turn or a left turn and the shift amount of the region of interest 77 within the image may be stored in advance in the memory 24 or the storage device 164. The ECU 167 can determine the shift amount based on the data and the measured steering angle. In the examples shown in FIGS. 9B to 9E , the vertical size of the cropped partial image is the same as the vertical size of the original captured image, and the horizontal size of the partial image is smaller than the horizontal size of the captured image. The cutting method is not limited to this, and an area smaller than the captured image in the vertical direction may also be cut out as a partial image.

[0100] The region of interest 77 may be determined based on other conditions as well as the turning state of the agricultural machine 100. For example, the region of interest 77 may be determined based on the traveling speed of the agricultural machine 100.

[0101] FIG. 10A shows an example of a region of interest 77 that can be selected when the agricultural machine 100 is moving forward at a relatively high speed. FIG. 10B shows a partial image corresponding to the region of interest 77 shown in FIG. 10A . In this example, the ECU 167 changes the size of the region of interest 77 in accordance with the traveling speed of the agricultural machine 100. Specifically, the ECU 167 makes the region of interest 77 smaller the higher the traveling speed of the agricultural machine 100, and makes the region of interest 77 larger the lower the traveling speed. When the agricultural machine 100 is moving forward at high speed, it is necessary to accurately detect objects 76 that are further away. Therefore, the ECU 167 makes the region of interest 77 smaller the higher the traveling speed, thereby improving the detection accuracy of the distant objects 76. In addition to changing the size of the region of interest 77 in accordance with the traveling speed of the agricultural machine 100, the position of the region of interest 77 may also be changed in accordance with the traveling speed. For example, the position of the region of interest 77 may be shifted upward as the traveling speed increases. Such processing makes it easier to detect distant objects when the agricultural machine 100 is traveling at high speed. ECU 167 obtains information about the current traveling speed from the measurement value of vehicle speed sensor 151. Data such as a table showing the relationship between the traveling speed and the size and / or position of region of interest 77 within the image may be stored in advance in memory 24 or storage device 164. ECU 167 can determine the size and / or position of region of interest 77 based on the data and the measured traveling speed.

[0102] The region of interest 77 may be determined based on the amount of tilt of the agricultural machine 100. The amount of tilt of the agricultural machine 100 refers to the magnitude of the tilt angle relative to a reference attitude, such as the pitch angle (i.e., the angle of rotation about a left-right axis) or the roll angle (i.e., the angle of rotation about a front-to-rear axis). Here, the reference attitude refers to the attitude of the agricultural machine 100 when it is on level ground. The ECU 167 may shift the region of interest up or down in accordance with the amount of tilt, such as the pitch angle, measured by the IMU 123. The pitch angle takes a positive value when the agricultural machine 100 is tilted upward with respect to a horizontal plane, such as when the agricultural machine 100 is traveling uphill, and takes a negative value when the agricultural machine 100 is tilted downward with respect to a horizontal plane, such as when the agricultural machine 100 is traveling downhill. When the agricultural machine 100 is tilted upward (i.e., the pitch angle takes a positive value), the region of interest in the captured image may be shifted downward. Conversely, when the agricultural machine 100 is tilted downward (i.e., the pitch angle has a negative value), the region of interest may be shifted upward. This prevents much of the sky or ground from being included in the region of interest, improving the object detection performance. The ECU 167 may increase the amount of shift of the region of interest as the absolute value of the pitch angle increases. In this case, data indicating the relationship between the absolute value of the pitch angle and the amount of shift is stored in the memory 24 or the storage device 164. The ECU 167 can determine the amount of shift of the region of interest based on this data and the measurement value of the IMU 123.

[0103] The ECU 167 may further determine the region of interest based on map data of the field. For example, the ECU 167 may determine the region of interest based on the map data, the positioning data output from the GNSS unit 120, and the operating state of the agricultural machine 100.

[0104] 11 is a diagram showing an example of the agricultural machine 100 traveling along the outermost periphery of a work area 71 in a field. In this example, a person, which is an object 76, is present outside the work area 71 in the field. Because the agricultural machine 100 does not travel outside the work area 71 while traveling to perform work, such an object 76 does not need to be detected as an obstacle. Therefore, the ECU 167 may identify an area corresponding to the work area 71 in the field where agricultural work is performed from the captured image acquired by the camera 126 based on the map data and positioning data, and may determine a region of interest from that area based on the operating state of the agricultural machine 100.

[0105] Here, an example of a method for identifying an area corresponding to the work area 71 from a captured image will be described.

[0106] FIG. 12 is a perspective view schematically showing the positional relationship between a camera coordinate system Σc fixed to the camera 126, a vehicle coordinate system Σv fixed to the agricultural machine 100, a world coordinate system Σw fixed to the ground, and a reference plane Re extending parallel to the horizontal plane. The camera coordinate system Σc has an Xc-axis, a Yc-axis, and a Zc-axis that are perpendicular to each other. The vehicle coordinate system Σv has an Xv-axis, a Yv-axis, and a Zv-axis that are perpendicular to each other. The world coordinate system Σw has an Xw-axis, a Yw-axis, and a Zw-axis that are perpendicular to each other. In the example of FIG. 12 , the Xw-axis and Yw-axis of the world coordinate system Σw are on the reference plane Re. The camera 126 is fixed to the agricultural machine 100. Therefore, the position and orientation of the camera coordinate system Σc with respect to the vehicle coordinate system Σv are fixed to a known state. The camera coordinate system Σc is inclined so that its Zc-axis obliquely intersects with the reference plane Re. When the agricultural machine 100 does not rotate in the pitch and roll directions, the plane including the Xv-axis and Yv-axis of the vehicle coordinate system Σv is parallel to the reference plane Re.

[0107] A virtual image plane Im exists at a position away from the origin O of the camera coordinate system Σc in the Zc-axis direction by the focal length of the camera 126. The image plane Im is orthogonal to the Zc-axis and the optical axis λ of the camera 126. Pixel positions on the image plane Im are defined by an image coordinate system having mutually orthogonal u- and v-axes. For example, assume that the coordinates of points P1 and P2 located on the reference plane Re in the world coordinate system Σw are (X1, Y1, Z1) and (X2, Y2, Z2), respectively. In the example of FIG. 12, the Xw- and Yw-axes of the world coordinate system Σw are on the reference plane Re. Therefore, Z1 = Z2 = 0.

[0108] Points P1 and P2 on the reference plane Re are transformed into points p1 and p2, respectively, on the image plane Im of the camera 126 by perspective projection of the pinhole camera model. In the image plane Im, points p1 and p2 are located at pixel positions indicated by the coordinates (u1, v1) and (u2, v2), respectively.

[0109] Given the positional relationship of the camera coordinate system Σc with respect to the reference plane Re in the world coordinate system Σw, a corresponding point (X, Y, 0) on the reference plane Re can be obtained from any point (u, v) on the image plane Im by homography transformation. Such homography transformation is defined by a 3-row by 3-column transformation matrix H when the coordinates of the point are expressed in a simultaneous coordinate system. When the coordinates of a point on the image plane Im of the camera 126 are (u, v, 0), the coordinates (X, Y, 0) of the corresponding point on the reference plane Re are associated with the point (u, v, 0) by the homography transformation matrix H, as shown in the following equation (1).

[0110] The contents of the transformation matrix H depend on the positional relationship of the camera coordinate system Σc with respect to the reference plane Re in the world coordinate system Σw. When the position of the reference plane Re changes, the contents of the transformation matrix H also change. The reference plane Re may be set to be in contact with the ground, or may be set at a height of a predetermined distance from the ground. When the object to be detected is a person, the reference plane Re may be set at a height of, for example, 1 meter to 2 meters from the ground.

[0111] By using such a homography transformation, the coordinates of any point on the image plane Im of the camera 126 can be associated with the coordinates of a point on the reference plane Re.

[0112] During operation of the agricultural machine 100, the ECU 167 can determine the position and attitude of the agricultural machine 100 based on the positioning data output from the GNSS unit 120. Furthermore, the positional relationship between the vehicle coordinate system Σv and the camera coordinate system Σc is known. Therefore, the ECU 167 can determine the positional relationship between the world coordinate system Σw and the camera coordinate system Σc based on the positioning data, and can determine the transformation matrix H based on this positional relationship.

[0113] The ECU 167 can identify which pixel area in the captured image corresponds to the work area 71 based on the result of the calculation of equation (1) and the position information of the work area included in the map data of the field. The ECU 167 can be configured to exclude pixel areas in the captured image that correspond to outside the work area 71 from the targets of object detection, and to determine a region of interest from within the pixel area that corresponds to the work area 71. This makes it possible to avoid the agricultural machine 100 being stopped due to the detection of an object 76 that does not affect the work travel of the agricultural machine 100. Note that, instead of determining a region of interest from within the pixel area that corresponds to the work area 71 in the captured image, the ECU 167 may determine a region of interest from within the pixel area that corresponds to an unworked area of ​​the work area 71. This is because the presence of an object 76 in an area where work has been completed does not interfere with the travel of the agricultural machine 100, and therefore the area where work has been completed may be excluded from the targets of detection.

[0114] Second Embodiment Next, an agricultural machine according to a second exemplary embodiment of the present disclosure will be described.

[0115] Similar to the agricultural machine of embodiment 1, the agricultural machine of this embodiment includes a machine body, an imaging device, and an image processing device that detects a specific object based on an image acquired by the imaging device (image acquisition device). The image processing device executes the following steps (S21) to (S23): (S21) Detects the boundary between the sky and features other than the sky, and the object from the image; (S22) Estimates the inclination of the machine body based on the position of the boundary between the sky and features other than the sky in the image; (S23) Estimates the position of the object in a coordinate system fixed to the ground based on the position of the object in the image and the estimated inclination of the machine body.

[0116] By performing such an operation, the tilt of the machine body can be estimated from the image without using a sensor for measuring the tilt of the machine body, and the position of the target object can be estimated. The agricultural machine according to this embodiment will now be described in detail.

[0117] The agricultural machine in this embodiment has a configuration similar to that of the agricultural machine 100 shown in FIG. 1 or FIG. 5 . In this embodiment, the image processing device 20 (for example, the ECU 167 shown in FIG. 5 ) also has the configuration shown in FIG. 2 . When the image processing device 20 in this embodiment detects a specific object based on an image acquired by the imaging device 10, it estimates the position of the object in a coordinate system fixed to the ground (the above-mentioned world coordinate system Σw) and the distance from the agricultural machine 100 to the object based on the position of the object in the image. This estimation process is performed by coordinate transformation from the image coordinate system to the world coordinate system Σw, as described with reference to FIG. 12 . The coordinate transformation requires information on the positional relationship of the camera coordinate system Σc with respect to the reference plane Re in the world coordinate system Σw. Therefore, while the agricultural machine 100 is traveling, it is necessary to sequentially estimate the orientation of the imaging device 10 in the agricultural machine 100 using a sensor (for example, the IMU 123 shown in FIG. 5 ) capable of measuring the inclination of the agricultural machine 100, and to determine the transformation matrix H in the above equation (1) using the information on this orientation.

[0118] However, when using the measurement values ​​of a sensor such as the IMU 123, there is a problem in that it is difficult to achieve time synchronization between the measurement values ​​of the sensor and the images (i.e., individual frames in the video) generated by the imaging device 10. Due to a time lag, there are cases where the posture of the agricultural machine 100 corresponding to each frame cannot be accurately identified.

[0119] To solve this problem, the image processing device 20 in this embodiment estimates the inclination (tilt angle) of the agricultural machine 100 based on the captured image acquired by the imaging device 10. Specifically, the image processing device 20 detects the boundary between the sky and features other than the sky from the captured image, and estimates the inclination angle (for example, pitch angle) of the agricultural machine 100 based on the temporal fluctuation of the position of that boundary. This makes it possible to more accurately identify the attitude of the agricultural machine 100 corresponding to each frame of the captured image, and improve the accuracy of estimating the position or distance of an object.

[0120] In this embodiment, the agricultural machine 100 may also be equipped with an inclination sensor (e.g., the IMU 123 shown in FIG. 5 ) that measures the inclination of the machine body. The image processing device 20 may calculate the difference between the inclination of the agricultural machine 100 estimated based on the position of the boundary between the sky and other features in the image and the inclination measured by the inclination sensor, and if the difference is less than a threshold, estimate the position of the object in a coordinate system fixed to the ground based on the position of the object in the image and the estimated inclination. In this case, if the difference between the inclination of the agricultural machine 100 estimated based on the position of the boundary between the sky and other features in the image and the inclination measured by the inclination sensor is equal to or greater than a threshold, the image processing device 20 may be configured to reject the estimation result of the former inclination and not perform processing based on the inclination. In this way, the measurement result of the inclination sensor may be used to evaluate the reliability of the inclination estimated based on the image.

[0121] A more specific example of the operation of the image processing device 20 in this embodiment will be described below with reference to FIG.

[0122] 13 is a flowchart showing an example of image processing executed by the image processing device 20 in this embodiment. The image processing device 20 repeatedly executes the processing shown in steps S210 to S270 while the agricultural machine 100 is in operation.

[0123] In step S210, the image processing device 20 acquires the image generated by the imaging device 10.

[0124] In step S220, the image processing device 20 executes a process for detecting a boundary between the sky and a feature other than the sky, and a specific object (e.g., a person) from the acquired image. The boundary between the sky and a feature other than the sky may be, for example, the boundary between the sky and the ground (horizon) or the boundary between the sky and a mountain. The boundary between the sky and a feature may be detected by various methods, such as image hue analysis, edge detection, and / or texture analysis. The process for detecting a specific object may be executed, for example, by a method similar to the processes of steps S120 to S140 shown in FIG. 3. Note that the boundary between the sky and a feature and the specific object may be detected simultaneously using a trained model for detecting both the boundary between the sky and a feature and the specific object from an image. In addition, in this embodiment, the partial image extraction process in step S120 may be omitted. That is, the image processing device 20 may detect a specific object and / or the boundary between the sky and a feature by inputting an input image generated by performing predetermined preprocessing on the captured image into the trained model.

[0125] 14 is a diagram showing an example of a boundary between the sky and a feature detected from a captured image. In this example, a boundary 78 between the sky and a mountain is illustrated by a thick curve. The image processing device 20 can be configured to perform a process of detecting such a boundary 78 for each frame of an image (video) generated by the imaging device 10.

[0126] In the next step S230, the image processing device 20 determines whether or not an object has been detected from the image. If an object has been detected, the process proceeds to step S240. If an object has not been detected, the process returns to step S210.

[0127] In step S240, the image processing device 20 estimates the inclination of the body of the agricultural machine 100 based on the position of the boundary 78 between the sky and the ground object in the image. For example, the image processing device 20 estimates the inclination of the body based on the amount of displacement between the position of the boundary 78 in the image and the position of the boundary 78 detected from an image acquired by the imaging device 10 at a past time. The past time may be a time when the inclination measured by the inclination sensor was less than a reference value. The image processing device 20 estimates, for example, the pitch angle of the body as the inclination of the body based on the amount of displacement. More specifically, the image processing device 20 may estimate the inclination angle of the agricultural machine 100 from a reference attitude based on temporal fluctuations in the position of the boundary 78 detected for each frame of the video generated by the imaging device 10. The reference attitude may be the attitude of the agricultural machine 100 when the agricultural machine 100 is on flat ground. The image processing device 20 uses the position of the boundary 78 in an image frame acquired when the agricultural machine 100 is in the reference posture as a reference and can estimate the pitch angle of the agricultural machine 100 based on how many pixels the boundary 78 has displaced up or down in the image from that position. For example, the image processing device 20 can estimate the pitch angle in the following manner. (1) From the current frame, corresponding points are identified for each of a plurality of pixels (or a portion of pixels, such as a central portion) aligned along the boundary 78 in the reference frame acquired when the agricultural machine 100 is in the reference posture. (2) The amount of displacement from the position in the reference frame is calculated for each corresponding point. (3) The average value of the displacement amounts of the plurality of corresponding points is calculated as the amount of displacement of the boundary 78. (4) The pitch angle is calculated based on the amount of displacement of the boundary 78.

[0128] The correspondence between the pitch angle and the amount of vertical displacement of the boundary 78 in the image can be stored in advance in, for example, the memory 24 shown in FIG. 2 . The image processing device 20 can estimate the pitch angle based on this correspondence. Note that the image processing device 20 can also estimate the roll angle by determining the rotation angle of the boundary 78 between frames based on the change in the position of the boundary 78 in the image over time. Estimating the roll angle in addition to the pitch angle allows for more accurate estimation of the attitude of the image capture device 10.

[0129] In step S250, the image processing device 20 estimates the position of the object in a coordinate system fixed to the ground (world coordinate system) based on the position of the object in the image and the estimated inclination of the airframe. Specifically, the image processing device 20 determines the transformation matrix H in the above equation (1) based on the estimated inclination of the airframe, the known positional relationship of the imaging device 10 with respect to the airframe, and the position and orientation of the airframe acquired from a positioning device (e.g., the GNSS unit 120 in FIG. 5). By performing the operation of equation (1) on the position of the object in the image coordinate system (e.g., the position of the bounding box), the position of the object in the world coordinate system can be calculated.

[0130] In step S260, the image processing device 20 estimates the distance from the agricultural machine 100 to the object based on the position of the object in the world coordinate system. The position of the agricultural machine 100 in the world coordinate system is measured by a positioning device such as the GNSS unit 120. The image processing device 20 calculates the distance between the position of the agricultural machine 100 in the world coordinate system and the position of the object. Here, the "position" of the agricultural machine 100 is the position of a predetermined specific part of the agricultural machine 100, such as the position of the GNSS unit 120.

[0131] In step S270, the image processing device 20 transmits information indicating the distance estimated in step S260 to the control device 30. The control device 30 receives the information and determines whether the distance is less than a predetermined value, and if the distance is less than the predetermined value, executes control such as stopping the movement of the agricultural machine 100 or causing a buzzer to emit a warning sound.

[0132] In this way, the image processing device 20 detects the boundary between the sky and non-sky features from the image acquired by the imaging device 10, and estimates the inclination of the machine based on temporal fluctuations in the position of that boundary. Based on the estimated inclination, the image processing device 20 converts the position coordinates of the object in the image into position coordinates in a coordinate system fixed to the ground, and estimates the distance to the object based on the converted position coordinates. The control device 30 controls the operation of the agricultural machine 100 based on the estimated distance.

[0133] 13 may be performed by the control device 30 instead of the image processing device 20. In that case, the image processing device 20 may be configured to transmit information indicating the position coordinates of the object in the world coordinate system to the control device 30 after step S250. In that case, the control device 30 controls the operation of the agricultural machine based on the position of the object in the world coordinate system. The control device 30 may be configured to execute at least one of stopping the agricultural machine 100, decelerating the agricultural machine 100, and outputting a warning when the distance estimated based on the position of the object in the world coordinate system is smaller than a predetermined value.

[0134] FIG. 15 is a flowchart showing a modified example of this embodiment. In the example of FIG. 15, the image processing device 20, rather than the control device 30, determines whether the distance from the agricultural machine 100 to the object is less than a predetermined value. The example of FIG. 15 differs from the example of FIG. 14 in that step S265 is added after step S260, and step S270 is replaced with step S275. In step S265, the image processing device 20 determines whether the distance from the agricultural machine 100 to the object is less than a predetermined value. If the determination result is Yes, the process proceeds to step S275. If the determination result is No, the process returns to step S210. In step S275, the image processing device 20 transmits a signal indicating the presence of an object to the control device 30. Upon receiving this signal, the control device 30 executes control such as stopping the movement of the agricultural machine 100 or causing a buzzer to emit a warning sound.

[0135] Fig. 16 is a flowchart showing another modified example of this embodiment. In the example shown in Fig. 16, an inclination (e.g., a pitch angle) measured by an inclination sensor (e.g., the IMU 123 shown in Fig. 5) is also used. The example in Fig. 16 differs from the example in Fig. 14 in that step S245 is added after step S240, and the process proceeds to step S255 if a negative determination is made in step S245.

[0136] In step S245, the image processing device 20 determines whether the difference between the aircraft inclination estimated in step S240 and the inclination measured by the inclination sensor is less than a threshold value. The inclination compared here may be, for example, a pitch angle or a roll angle. The inclination measured by the inclination sensor may be averaged over a predetermined time period (e.g., 0.1 seconds to several seconds) and compared with the aircraft inclination estimated in step S240. If the difference between the two inclinations is less than the threshold value, the process proceeds to step S250, and the same processing as in the example of FIG. 14 is performed. If the difference between the two inclinations is greater than or equal to the threshold value, the process proceeds to step S255. A difference between the two inclinations greater than or equal to the threshold value suggests that the aircraft inclination estimated based on the image may be unreliable. Therefore, in such a case, the image processing device 20 estimates the position of the object in a coordinate system fixed on the ground based on the inclination measured by the inclination sensor, rather than the inclination estimated based on the image. After step S255, the process proceeds to step S260.

[0137] Note that step S270 in the example of Fig. 16 may be replaced with steps S265 and S275 in Fig. 15. In that case, the image processing device 20, rather than the control device 30, determines whether the distance from the agricultural machine 100 to the target object is less than a predetermined value.

[0138] (Third embodiment) Next, an agricultural machine according to a third exemplary embodiment of the present disclosure will be described.

[0139] Similar to the agricultural machine of the first embodiment, the agricultural machine of this embodiment includes an imaging device, an image processing device that detects specific objects from images acquired by the imaging device, and a control device that controls the operation of the agricultural machine based on the object detection results. This embodiment differs from the first embodiment in that the image processing device includes a memory that stores multiple trained models according to the environment around the agricultural machine, and switches between these models depending on the environment. The image processing device executes the following steps (S31) to (S33): (S31) Generate an input image based on an image acquired by the imaging device. (S32) Select one trained model from the multiple trained models according to the environment around the agricultural machine. (S33) Detect the object by inputting the input image into the selected trained model.

[0140] This operation enables the detection of objects from captured images with higher accuracy using an appropriate trained model that is suited to the environment surrounding the agricultural machine.

[0141] The environment surrounding agricultural machinery is diverse, and when environmental conditions such as time of day, weather, type of crop being worked on, and type of field change, the characteristics of the image captured by the imaging device (e.g., brightness, saturation, hue, etc. of each pixel) change. For this reason, when a single trained model is used in various environments, the object detection performance may be reduced depending on the environment. To maintain high detection performance in various environments, it is necessary to prepare a large amount of training data and train the model, but even so, it may be difficult to adapt to diverse environments with just one model.

[0142] Therefore, in this embodiment, multiple trained models corresponding to multiple environments are prepared in advance, and the image processing device is configured to use these models appropriately depending on the situation. More specifically, the image processing device generates an input image based on an image acquired by an imaging device, and detects objects by inputting the input image into one trained model selected from the multiple trained models according to the environment around the agricultural machine. This operation makes it possible to detect objects from images with high accuracy using an appropriate trained model according to the environment, even if changes occur in the environment around the agricultural machine.

[0143] The agricultural machine in this embodiment has a configuration similar to that of the agricultural machine 100 shown in Fig. 1 or Fig. 5. In this embodiment, the image processing device 20 (for example, the ECU 167 shown in Fig. 5) also has a hardware configuration similar to that in the example shown in Fig. 2. However, this embodiment differs from the example in Fig. 2 in that the memory 24 stores a plurality of trained models.

[0144] 17 is a block diagram showing an example configuration of an image processing device 20 according to this embodiment. The memory 24 of the image processing device 20 in this example stores a computer program 25 executed by the processor 22 and multiple trained models 27 for detecting objects from an input image. The multiple trained models 27 are stored in association with various environmental conditions around the agricultural machine 100. For example, multiple trained models 27 associated with multiple different brightness levels and / or multiple different time periods in the environment around the agricultural machine 100 may be stored in the memory 24. The image processing device 20 selects and uses one trained model from the multiple trained models 27 that corresponds to the current brightness level of the environment or the current time period. The image processing device 20 can determine the brightness of the current environment based on at least one of an image acquired by the imaging device 10 (e.g., a histogram or average brightness of pixel values ​​in the image), an output from an illuminance sensor (e.g., illuminance sensor 153 shown in FIG. 5 ) provided in the agricultural machine 100, an input from a user (e.g., the type of crop and / or the type of work land input via the terminal monitor 131 shown in FIG. 5 ), and the lighting status of the lights of the agricultural machine 100. The illuminance sensor may be attached to the body of the agricultural machine 100 (e.g., near the imaging device 10). In addition, the image processing device 20 can acquire information about the current time zone from, for example, a clock function included in the processor 22 or a timing circuit such as a real-time clock that may be provided separately from the processor 22.

[0145] In the example shown in FIG. 17 , the multiple trained models 27 include daytime models 27A and 27B and a nighttime model 27C. In this case, the image processing device 20 selects the daytime model 27A or 27B during the daytime, and selects the nighttime model 27C during the nighttime. In the example of FIG. 17 , two types of daytime models are prepared: a non-backlit model 27A and a backlit model 27B. In this case, the image processing device 20 may be configured to determine whether or not the image is backlit during the daytime based on, for example, at least one of an image acquired by the imaging device 10, an input from the user, and an output from the illuminance sensor, and to select the backlit model 27B if the image is backlit, and to select the non-backlit model 27A if the image is not backlit.

[0146] The model example is not limited to the example shown in FIG. 17 . For example, multiple trained models associated with different weather conditions, such as a model for sunny weather, a model for cloudy weather, and a model for rainy weather, may be stored in the memory 24. In this case, the image processing device 20 selects and uses one trained model corresponding to the current weather from the multiple trained models. The image processing device 20 can identify the current weather based on images captured by the imaging device 10 or weather information acquired from an external device (e.g., the communication device 190 shown in FIG. 5 and a server computer connected via a network). Alternatively, multiple trained models associated with different types of crops may be prepared. In this case, the image processing device 20 selects one trained model corresponding to the type of crop to be worked on from the multiple trained models 27. The multiple models corresponding to the type of crop to be worked on, such as a rice model, a wheat model, and a soybean model, may include various models corresponding to the type of crop to be worked on, such as harvesting. In addition, multiple trained models corresponding to the type of field or work area, such as a rice paddy model, a vegetable field model, and a pasture model, may be prepared. Because the color and texture of images captured by the imaging device 10 can vary significantly depending on the type of crop or type of work area, environmental adaptability can be improved by preparing multiple models according to the type of crop or type of work area being worked on. The image processing device 20 can identify the type of crop or type of work area being worked on based on images captured by the imaging device 10 or input from the user. Furthermore, different trained models may be created in advance for any two or more combinations selected from the brightness around the agricultural machine, time of day, weather, type of crop, and type of work area, and these models may be switched depending on environmental conditions. Switching between such multiple models according to the environment enables even more accurate object detection.

[0147] FIG. 18 is a table showing an example of the correspondence between multiple trained models stored in memory 24 and environments. In this example, multiple trained models A to L, which differ depending on the combination of three factors: whether the time of day is daytime or nighttime; whether the weather is sunny or cloudy; whether the weather is backlit if sunny; and the type of crop, are stored in memory 24. Each model is pre-trained using a large amount of learning data corresponding to the corresponding environment and stored in memory 24. The correspondence between models and environments is not limited to the example shown in FIG. 18 and can be modified in various ways. For example, a model corresponding to not only daytime and nighttime but also evening may be created. Alternatively, a model corresponding to other weather conditions such as rain or snow may be created. In the example of FIG. 18, there are three types of crops, A, B, and C, but the number of types may be one, two, or four or more. Furthermore, a model may be created for each type of work area, such as a rice paddy, a vegetable field, or a pasture.

[0148] Each of the multiple trained models 27 may be a machine learning model trained using a deep learning-based algorithm, such as a CNN or a vision transformer. Each model has a common model architecture and a set of weights that differ from each other. By determining the combination of weights for each model using appropriate training data and training software, multiple trained models 27 can be created that are suited to various environments. The multiple trained models 27 may be generated by the image processing device 20 itself, or may be generated by an external device such as a server computer. The multiple trained models 27 are stored in the memory 24 before the agricultural machine 100 starts operating or before the model is used.

[0149] Fig. 19 is a flowchart showing an example of the operation of the image processing device 20 in this embodiment. In the example of Fig. 19, the image processing device 20 repeatedly executes the operations of steps S310 to S340 while the agricultural machine 100 is in operation.

[0150] In step S310, the image processing device 20 generates an input image to be input to the trained model based on the captured image generated by the imaging device 10. The image processing device 20 generates the input image, for example, by performing predetermined preprocessing on the captured image (e.g., image resizing, normalization, noise removal, etc.). Alternatively, the image processing device 20 may generate the input image by performing the processes of steps S110 to S130 shown in Fig. 3. By performing the processes of steps S110 to S130, it is possible to narrow down the area to be detected depending on the operating state of the agricultural machine 100, thereby improving the accuracy of detecting the object.

[0151] In step S320, the image processing device 20 selects one trained model that corresponds to the surrounding environment of the agricultural machine 100 from the plurality of trained models 27 stored in the memory 24. A specific example of the processing in step S320 will be described later.

[0152] In step S330, the image processing device 20 detects a specific object (e.g., a person) by inputting the input image into the selected trained model. This step is similar to the processing of step S140 in FIG. 3 . The image processing device 20 detects an object, such as a person, by inputting the input image into the trained model 27 stored in the memory 24. For example, if a specific object is present in the image, the image processing device 20 may be configured to output, as a detection result, coordinate information indicating the position of a rectangular frame (bounding box) surrounding the area where the object is present, and information on the width and height of the bounding box. Alternatively, the image processing device 20 may output, as a detection result, a signal indicating whether a specific object is present in the input image. The image processing device 20 may determine that an object is present in the input image if the distance from the imaging device 10 to the object, calculated based on the position of the bounding box in the input image, is less than a threshold.

[0153] In step S340, the image processing device 20 transmits the detection result to the control device 30. This step is similar to the processing of step S150 in FIG. 3. The image processing device 20 transmits the detection result of the object to the control device 30. For example, the image processing device 20 may transmit a signal to the control device 30 indicating whether or not a specific object is present in the input image. Alternatively, the image processing device 20 may transmit a signal to the control device 30 indicating that a specific object is detected from the input image only when the distance to the detected object is less than a predetermined value. The distance to the object may be estimated using, for example, the method shown in FIG. 15 or FIG. 16.

[0154] When the control device 30 receives a signal indicating the detection result from the image processing device 20, it controls the operation of the agricultural machine 100 in accordance with the detection result. For example, when the control device 30 receives a signal indicating that an object has been detected, the control device 30 can be configured to perform at least one of stopping the agricultural machine 100, slowing down the agricultural machine 100, and outputting a warning to a device such as a buzzer or a display. Such actions can avoid a collision between the agricultural machine 100 and the object, or alert the object (e.g., a person) or the occupant of the agricultural machine 100.

[0155] The processing of steps S310 to S340 may be performed periodically at the frame rate of image capture by the imaging device 10 (e.g., the camera 126 shown in FIG. 5 ). Alternatively, the processing of steps S310 to S340 may be performed every predetermined number of frames (e.g., 10 frames, 30 frames, 60 frames, etc.) or every predetermined time (e.g., 0.5 seconds, 1 second, 2 seconds, 3 seconds, etc.). The processing of step S320 may be performed before step S310. Furthermore, the processing of step S320 may be performed less frequently than the processing of steps S310, S330, and S340. For example, the processing of step S320 may be performed every predetermined time (e.g., every few seconds, every tens of seconds, every few minutes, etc.) or every time the agricultural machine 100 changes direction. The environment around the agricultural machine 100 does not change over a short period of time, and whether or not the agricultural machine 100 is backlit may change depending on the orientation of the agricultural machine 100. Therefore, model selection does not necessarily have to be performed with the same frequency as object detection, and model selection may be performed at the timing of a change of direction such as a turn.

[0156] The control device 30 of the agricultural machine 100 (for example, the ECU 166 shown in FIG. 5 ) may be configured to move the agricultural machine 100 along a set target route 74 as shown in FIG. 6 . The target route 74 is not limited to the example shown in FIG. 6 and may be, for example, a route including a round-trip section. The image processing device 20 may select a trained model each time the agricultural machine 100 changes direction or each time it reverses its direction of travel in a round-trip section. The image processing device 20 can acquire, from the control device 30, a signal indicating that the agricultural machine 100 has changed direction. The image processing device 20 can select an appropriate model by performing the processing of step S320 in response to the signal. In this case, the same model is used for object detection until the next turning is performed.

[0157] FIG. 20 is a flowchart showing an example of the operation of the image processing device 20 in a case where a trained model is selected each time the agricultural machine 100 changes direction. In the example of FIG. 20 , the image processing device 20 first selects one trained model from a plurality of trained models in step S320 that corresponds to the surrounding environment of the agricultural machine. Thereafter, the image processing device 20 executes the processes of steps S310, S330, and S340. After step S340, the process proceeds to step S350, where the image processing device 20 determines whether the agricultural machine 100 has changed direction. When the image processing device 20 receives a signal indicating that the agricultural machine 100 has changed direction from the control device 30, the image processing device 20 determines that the agricultural machine 100 has changed direction. When a direction change has occurred, the process returns to step S320, where a trained model is reselected. When a direction change has not occurred, the process returns to step S310, where the previously selected trained model is used to perform object detection processing again based on the captured image.

[0158] Next, the model selection process in step S320 will be described in more detail with reference to FIG.

[0159] 21 is a flowchart showing an example of the model selection process in step S320. In this example, step S320 includes steps S321 to S326.

[0160] In step S321, the image processing device 20 acquires information about the type of work area and the type of crop set by the user. This information can be set by the user operating an input device such as the terminal monitor 131 shown in FIG. 5 .

[0161] In step S322, the image processing device 20 acquires information about the date and time. For example, the image processing device 20 can acquire the information about the date and time from a clock function of the processor 22 of the image processing device 20 or a real-time clock that may be provided separately from the processor 22.

[0162] In step S323, the image processing device 20 acquires information regarding whether the lights of the agricultural machine 100 are on or off. Information regarding whether the lights are on or off can be acquired from the control device 30. The user can switch the lights 142 on and off by operating, for example, the changeover switch for the lights 142 included in the operation switch group 132 shown in FIG. 5. A signal indicating whether the lights 142 are on or off can be transmitted from the control device 30 (for example, the ECU 165 shown in FIG. 5) to the image processing device 20.

[0163] In step S324, the image processing device 20 acquires a measurement value of an illuminance sensor (for example, the illuminance sensor 153 shown in FIG. 5). The measurement value of the illuminance sensor indicates the degree of brightness of the environment around the agricultural machine 100.

[0164] In step S325, the image processing device 20 generates a histogram of pixel values ​​in the image. For example, if the pixel value of each pixel is expressed in 256 gradations (8 bits) from 0 to 255 for each color of red, green, and blue, a histogram representing the frequency of each pixel value from 0 to 255 for each color can be generated. The image processing device 20 can evaluate the lightness, hue, saturation, etc. of the entire image based on the histogram. In this step, instead of generating a histogram from the entire image, the image processing device 20 may detect a region corresponding to the sky from the image using, for example, the method in embodiment 2, and generate a histogram for the region corresponding to the sky.

[0165] In step S326, the image processing device 20 selects a trained model to use based on the type of work area, the type of crop, the date and time, whether the lights are on or off, the measurement value of the illuminance sensor, and the image histogram. Based on this information or signal, the image processing device 20 can estimate whether the time of day is daytime, evening, or night, or whether the weather is sunny, cloudy, rainy, or snowing, and select a trained model according to these estimation results and the selected work area and type of crop.

[0166] The order of the processes from steps S321 to S325 may be reversed. Furthermore, at least one of the processes from steps S321 to S325 may be omitted. In this case, the image processing device 20 selects a trained model without considering the omitted information. The algorithm for estimating the type of work site, type of crop, time of day, weather, etc. from the acquired information such as images is not limited to a specific one and can be designed arbitrarily. An AI model may be used to determine the optimal model from the acquired information such as images.

[0167] As described above, according to this embodiment, an object detection process is performed using a trained model appropriately selected from multiple trained models according to the environment. This significantly improves object detection performance compared to when a single trained model is used. Furthermore, even when each model is trained with a relatively small amount of training data, the detection accuracy in each environment can be improved. For example, the accuracy of image recognition can be ensured regardless of day or night or weather.

[0168] (Fourth embodiment) Next, an agricultural machine according to a fourth exemplary embodiment of the present disclosure will be described.

[0169] In this embodiment, the image processing device mounted on the body of the agricultural machine has a function of performing relearning of the trained model (hereinafter also referred to as "optimized learning"). By relearning, the trained model can be improved to adapt to the actual usage environment of the agricultural machine, thereby improving the object detection performance.

[0170] Similar to the agricultural machines of the above-described embodiments, the agricultural machine of this embodiment includes an imaging device, an image processing device that detects a specific object (e.g., a person) from an image acquired by the imaging device, and a control device that controls the operation of the agricultural machine based on the object detection results. The image processing device stores one or more trained models for detecting objects from input images. The image processing device of this embodiment performs the following steps (S41) to (S43) to retrain the trained model: (S41) Generate an input image based on an image acquired by the imaging device; (S42) Detect an object by inputting the input image into the trained model; and (S43) Retrain the trained model based on one or more images in which an object is detected, among multiple images acquired by the imaging device during operation of the agricultural machine.

[0171] With the above configuration, the trained model can be improved to suit the actual usage environment of the agricultural machinery, thereby improving the object detection performance.

[0172] The configuration of the agricultural machine in this embodiment is similar to the configuration of the agricultural machine 100 shown in Fig. 1 or Fig. 5. The configuration of the image processing device is similar to the configuration of the image processing device 20 shown in Fig. 2 or Fig. 17. This embodiment differs from the above-described embodiments in that the image processing device 20 has a function of improving the trained model 27 by re-training.

[0173] The trained model 27 shown in FIG. 2 or each of the multiple trained models 27 shown in FIG. 17 is created by training using a large amount of training data that is adapted to that model before use of the agricultural machine 100 begins. As described above, each model may have a common model architecture and weight sets that differ from each other. Each trained model 27 can be created by determining model parameters such as the values ​​of the weight sets through training using training data appropriate for each model.

[0174] However, the actual use environments of the agricultural machine 100 vary, and it may be difficult to build a model that can maintain high detection performance in a variety of use environments using only prior learning. Therefore, the image processing device 20 in this embodiment is configured to re-learn the trained model using images in which objects are actually detected, among images acquired in the actual use environments of the agricultural machine 100. This makes it possible to continuously improve the model and enhance the object detection performance.

[0175] In this embodiment, the image processing device 20, which is an edge computing device mounted on the agricultural machine 100, is configured to re-learn the trained model. Therefore, it is possible to realize optimized training of the trained model adapted to the usage environment of the agricultural machine 100 without communicating with an external computer such as a cloud server.

[0176] The operation of the image processing device 20 in this embodiment will be described in more detail below.

[0177] The image processing device 20 in this embodiment determines whether the object detection result is correct for each of one or more images in which an object is detected, among multiple images acquired by the imaging device 10 while the agricultural machine 100 is in operation. Whether the object detection result is correct can be determined, for example, based on the time from when the movement of the agricultural machine 100 stops after the object detection process until the movement is resumed. This determination method is based on the idea that if an object is detected from an image even though it does not actually exist (i.e., in the case of a false detection), the user of the agricultural machine 100 will resume movement immediately after the agricultural machine 100 has stopped.

[0178] If the image processing device 20 determines that the object detection result is correct, the image is used for re-learning the trained model. More specifically, the image processing device 20 does not use, for re-learning, an image in which an object is erroneously detected even though the object does not actually exist, but uses, for re-learning, only images in which the object is correctly detected.

[0179] When the image processing device 20 detects an object from the image, it transmits a signal indicating that the object has been detected to the control device 30. Upon receiving the signal indicating that the object has been detected, the control device 30 performs specific control, such as stopping the agricultural machine 100. At this time, if the object actually exists on or near the path of the agricultural machine 100, the detection result is correct. Conversely, if the object does not actually exist, the detection result is incorrect. If the object actually exists, the user of the agricultural machine 100 moves the object to a position where it does not obstruct the movement of the agricultural machine 100, and then performs an operation to resume the movement of the agricultural machine 100. On the other hand, if the object does not actually exist, the user immediately resumes the movement of the agricultural machine 100 after confirming that the object does not exist. The user operates an input device, such as the operation switch group 132 or the terminal monitor 131 shown in FIG. 5 , to issue a resume command to the control device 30 (ECU 165 in the example of FIG. 5 ) to instruct the agricultural machine 100 to resume movement. In response to the input restart command, the control device 30 restarts the movement of the agricultural machine 100. The image processing device 20 can be configured to measure the time from when the agricultural machine 100 stops to when it restarts its movement, and to determine whether the object detection result is correct based on the time. For example, the image processing device 20 can be configured to determine that the object detection result is correct if the measured time is longer than a threshold, and that the object detection result is incorrect if the measured time is equal to or shorter than the threshold.

[0180] Fig. 22 is a flowchart showing a specific example of the operation of the image processing device 20 in this embodiment. In the example of Fig. 22 , the image processing device 20 executes the operations of steps S410 to S500, thereby performing object detection processing based on moving images acquired while the agricultural machine 100 is moving, and re-learning of the trained model. The operation shown in Fig. 22 starts, for example, when a command to start the operation of the agricultural machine 100 is given by the user via the input device.

[0181] In step S410, the image processing device 20 generates an input image to be input to the trained model based on the captured image generated by the imaging device 10. The image processing device 20 generates the input image, for example, by performing predetermined preprocessing on the captured image (e.g., image resizing, normalization, noise removal, etc.). Alternatively, the image processing device 20 may generate the input image by performing the processes of steps S110 to S130 shown in FIG. 3. By performing the processes of steps S110 to S130, it is possible to narrow down the area to be detected depending on the operating state of the agricultural machine 100, thereby improving the accuracy of detecting the object.

[0182] In step S420, the image processing device 20 executes a process of detecting an object by inputting an input image into a trained model stored in the memory 24. As in the third embodiment, one trained model according to the environment may be selected from a plurality of trained models and applied to the image, or as in the first and second embodiments, the trained model to be used may be predetermined.

[0183] In step S430, the image processing device 20 determines whether or not an object has been detected from the input image. If an object has been detected, the process proceeds to step S440. If an object has not been detected, the process proceeds to step S490.

[0184] 23 is a diagram showing an example of an object detection result. In this example, the image processing device 20 executes object detection software to detect a specific object from an input image. The detection results may include, for example, a label indicating the type of object (such as "Person" for a person), the coordinate values ​​of a representative point (such as the upper left vertex or center point) of a bounding box indicating the position of the object in the image, and a numerical value indicating the reliability of the detection result.

[0185] In step S430, the image processing device 20 may determine whether or not an object has been detected based on the distance between the agricultural machine 100 and the object. For example, the image processing device 20 may calculate the position of the object in a coordinate system fixed to the ground based on the position of the object in the input image and information about the position and orientation of the imaging device 10 in the agricultural machine 100, and determine that the object has been detected if the distance between that position and the agricultural machine 100 is less than a threshold. Here, the distance may be estimated using the method shown in Fig. 15 or Fig. 16. Note that the distance may be determined using a measurement value of a distance measuring sensor such as the laser sensor 125 shown in Fig. 5.

[0186] In step S440, the image processing device 20 transmits a signal indicating that an object has been detected to the control device 30. Upon receiving this signal, the control device 30 stops the movement of the agricultural machine 100 and transmits a signal indicating that the movement has been stopped to the image processing device 20.

[0187] In step S450, the image processing device 20 receives a signal indicating that the movement of the agricultural machine 100 has been stopped from the control device 30. Upon receiving this signal, the image processing device 20 waits until the movement of the agricultural machine 100 is resumed.

[0188] As described above, after the movement of the agricultural machine 100 is stopped, the user of the agricultural machine 100 checks whether an object actually exists in the direction of travel of the agricultural machine 100. If the object does not actually exist (i.e., in the case of a false detection), the user resumes the movement of the agricultural machine 100 in a relatively short time. On the other hand, if the object actually exists, the user resumes the movement of the agricultural machine 100 after confirming that the object is no longer on the planned route of the agricultural machine 100. When the movement of the agricultural machine 100 resumes, the control device 30 transmits a signal indicating that movement has resumed to the image processing device 20.

[0189] In step S460, the image processing device 20 receives from the control device 30 a signal indicating that the movement of the agricultural machine has resumed.

[0190] In step S470, the image processing device 20 compares the time from when the agricultural machine stops moving to when it resumes moving (hereinafter referred to as the "waiting time") with a predetermined threshold. If the waiting time is longer than the threshold, the process proceeds to step S480. If the waiting time is equal to or less than the threshold, it is determined that the object has been erroneously detected, and the process proceeds to step S490. If the waiting time is longer than the threshold, the object is treated as having been correctly detected. If the waiting time is equal to or less than the threshold, the object is treated as having been erroneously detected.

[0191] In step S480, the image processing device 20 records the input image as a re-learning image in the memory 24. At this time, information indicating the detection result may also be recorded in association with the input image. The information indicating the detection result may include a label indicating the type of object and coordinate values ​​indicating the position of the object in the input image (e.g., coordinate values ​​of a bounding box).

[0192] In step S490, the image processing device 20 determines whether or not a signal indicating that the agricultural work by the agricultural machine 100 has been completed (hereinafter referred to as the "work completion signal"). The work completion signal can be transmitted from the control device 30 to the image processing device 20 when the agricultural work is completed. "When the agricultural work is completed" can be, for example, when the agricultural machine 100 has completed its work travel along a target route in the field (e.g., the target route 74 shown in FIG. 6 ), or when the harvested product has been discharged after the work travel has been completed. If the image processing device 20 has not received the work completion signal, the process returns to step S410. The image processing device 20 repeats the processes of steps S410 to S490 until it receives the work completion signal. If the image processing device 20 has received the work completion signal, the process proceeds to step S500.

[0193] In step S500, the image processing device 20 re-learns the model based on the group of input images recorded in the memory 24 as re-learning images, creates a new trained model, and records it in the memory 24. For example, the image processing device 20 creates a new trained model by executing re-learning software using, as training data, data including each of the input images recorded as re-learning images, position information of objects in each input image, and label information indicating the type of object in each input image.

[0194] As described above, while agricultural work is being performed by the agricultural machine 100, the image processing device 20 in this embodiment repeatedly generates input images based on images acquired by the imaging device 10, detects objects by inputting the input images into a trained model, and records images when the objects are detected. After receiving a signal indicating that agricultural work has been completed, the image processing device 20 re-trains the trained model based on one or more images in which objects have been detected. Through this processing, the model can be re-trained and improved each time agricultural work in one field is completed, for example.

[0195] The timing at which the model is re-learned in step S500 does not have to be the timing at which agricultural work is completed by the agricultural machine 100. For example, the model may be re-learned in step S500 when the operation mode is switched from the automatic operation mode to the manual operation mode, or when the power of the agricultural machine 100 is turned off.

[0196] Furthermore, a method for determining whether the object detection result is correct may be a method other than a method based on the time from when the movement of the agricultural machine 100 stops after the object detection process until when the movement resumes. For example, the image processing device 20 may determine whether the object detected from the image actually exists based on the output from an object detection sensor (e.g., the laser sensor 125 and / or the millimeter-wave radar 127 in the example of FIG. 5 ) mounted on the agricultural machine 100. In a case where a person, who is an object, is present among the crops to be harvested, an object detection sensor such as a laser sensor or millimeter-wave radar may not be able to distinguish between the person and the crops, but it can detect the presence of some kind of object. Therefore, by using both image-based object detection and object detection by an object detection sensor, it is possible to determine whether the image-based object detection result is correct.

[0197] In the example of FIG. 22 , if the time from when the agricultural machine 100 stops moving until when it resumes moving is longer than a threshold (i.e., if it is determined that the detection result of the object based on the image is correct), the image processing device 20 stores the input image in the memory 24. Not limited to this process, the image processing device 20 may add information indicating whether the detection result of the object is correct to the input image and store the information in the memory 24. For example, if it is determined that the detection result of the object based on the input image is incorrect, information indicating that the detection result is incorrect may be associated with the input image and stored in the memory 24. Furthermore, if it is determined that the detection result of the object based on the input image is correct, the image processing device 20 may store information indicating that the detection result is correct in the memory 24, associated with the input image. Such information may be stored as metadata associated with the input image. The image processing device 20 can determine an image to use for re-training the trained model based on the information. For example, the image processing device 20 may not use an image associated with information indicating that the detection result is incorrect for re-training, but may use an image associated with information indicating that the detection result is correct for re-training.

[0198] In step S480, the image processing device 20 may record position information of the object in each of one or more images in which the object is detected, in association with the image. The position information may be, for example, the coordinate value of a representative point (e.g., the upper left vertex or the center point) of a bounding box indicating the position of the object in the image. By adding and recording such position information to the image, retraining of the trained model can be efficiently performed. For example, the image processing device 20 may store position information of the object in the image and a label indicating the type of the object in the memory 24. In this way, the position information of the object in the image and the label indicating the type of the object can be used as annotation data, thereby efficiently retraining the trained model.

[0199] After performing re-learning in step S500, the image processing device 20 may update the existing trained model with the re-trained trained model, or both may coexist. When both coexist, the image processing device 20 may allow a user to select whether to use the existing trained model or the re-trained trained model for the object detection process. Furthermore, the image processing device 20 may update the existing trained model with the re-trained trained model in accordance with a user instruction. The user instruction may be given by operating an input device such as the terminal monitor 131 shown in FIG. 5 . This configuration allows the user to update the model after confirming that the performance of the re-trained trained model is improved over the existing trained model, or to revert to the original trained model if the effect of re-learning is not confirmed.

[0200] In the above first to fourth embodiments, examples have been described in which the work machine is an agricultural machine such as a harvester, but the above-mentioned technologies may also be applied to work machines other than agricultural machines. For example, some or all of the functions of the first to fourth embodiments may be implemented in a construction work vehicle 200 such as the one shown in FIG. 24. Furthermore, the image acquisition device is not limited to the imaging device 10 such as the camera 126, but may also be a device capable of acquiring point cloud images, such as a LiDAR sensor.

[0201] As described above, the present disclosure includes the agricultural machine, image processing device, and image processing method described below.

[0202] [Item A1] A work machine that performs work while moving, comprising: an image capture device that captures images in the direction of movement of the work machine; an image processing device that detects a specific object from the images; and a control device that controls the operation of the work machine based on the detection results of the object, wherein the image processing device has a memory that stores a trained model for detecting the object from an input image, extracts a partial image that shows a region of interest determined based on the operating state of the work machine from the image captured by the image capture device, generates the input image based on the partial image, and detects the object by inputting the input image into the trained model.

[0203] [Item A2] The work machine according to Item A1, wherein the image processing device acquires information relating to at least one of a travel speed of the work machine, a turning state of the work machine, and a tilt state of the work machine, and changes the region of interest based on the information.

[0204] [Item A3] The work machine according to Item A2, wherein the image processing device changes the size of the region of interest in accordance with the movement speed of the work machine.

[0205] [Item A4] The work machine according to Item A3, wherein the image processing device reduces the region of interest as the movement speed of the work machine increases.

[0206] [Item A5] The work machine according to Item A2, wherein the image processing device shifts the region of interest to the right when the work machine turns right, and shifts the region of interest to the left when the work machine turns left.

[0207] [Item A6] The work machine according to Item A5, wherein the image processing device shifts the region of interest by a larger amount as the steering angle during turning of the work machine increases.

[0208] [Item A7] The work machine according to Item A2, further comprising an inclination sensor that measures an amount of inclination of the work machine, and the image processing device changes the position of the area of ​​interest according to the amount of inclination.

[0209] [Item A8] The work machine according to Item A7, wherein the tilt sensor measures a pitch angle of the work machine as the tilt amount, and the image processing device shifts the region of interest up or down depending on the pitch angle.

[0210] [Item A9] The work machine according to any of Items A1 to A8, further comprising: a storage device that stores map data of an area including a field where work is to be carried out by the work machine; and a positioning device that acquires positioning data of the work machine, wherein the image processing device determines the area of ​​interest based on the map data, the positioning data, and the operating state of the work machine.

[0211] [Item A10] The work machine according to Item A9, wherein the image processing device identifies an area corresponding to a work area in which the work is to be performed in the field from the image acquired by the image acquisition device based on the map data and the positioning data, and determines the area of ​​interest from within the area based on the operating state of the work machine.

[0212] [Item A11] The work machine according to any one of Items A1 to A10, wherein the image processing device generates the input image by processing that includes compressing the number of pixels of the partial image to a preset number of pixels.

[0213] [Item A12] The work machine according to any one of items A1 to A11, wherein the work machine is an automatically driven work vehicle or unmanned aerial vehicle, and the control device controls the automatically driven operation of the work machine.

[0214] [Item A13] The work machine according to any one of Items A1 to A12, wherein the control device executes at least one of stopping the work machine, slowing down the work machine, and outputting a warning when the object is detected.

[0215] [Item A14] An image processing device that detects a specific object from an image acquired by an image acquisition device mounted on a work machine, comprising: a memory that stores a trained model for detecting the object from an input image; and an arithmetic circuit, wherein the arithmetic circuit extracts, from the image acquired by the image acquisition device, a partial image that indicates a region of interest determined based on the operating state of the work machine; generates the input image based on the partial image; and detects the object by inputting the input image to the trained model.

[0216] [Item A15] A method executed by an image processing device that detects a specific object from an image acquired by an image acquisition device mounted on a work machine, the method including: acquiring the image from the image acquisition device; acquiring information indicating the operating state of the work machine; extracting from the image a partial image that indicates a region of interest determined based on the operating state of the work machine; generating an input image based on the partial image; and detecting the object by inputting the input image into a trained model that has been generated in advance.

[0217] [Item B1] A work machine comprising: an aircraft; an image acquisition device; and an image processing device that detects a specific object from an image acquired by the image acquisition device, wherein the image processing device: detects the boundary between the sky and a feature other than the sky, and the object from the image; estimates the inclination of the aircraft based on the position of the boundary in the image; and estimates the position of the object in a coordinate system fixed to the ground based on the position of the object in the image and the estimated inclination.

[0218] [Item B2] The work machine according to Item B1, wherein the image processing device estimates the distance from the work machine to the object based on the position of the object in the coordinate system fixed to the ground.

[0219] [Item B3] The work machine according to Item B1, further comprising an inclination sensor that measures the inclination of the machine body, wherein the image processing device calculates the difference between the inclination estimated based on the position of the boundary in the image and the inclination measured by the inclination sensor, and if the difference is less than a threshold, estimates the position of the object in the coordinate system fixed to the ground based on the position of the object in the image and the estimated inclination.

[0220] [Item B4] The work machine according to any one of Items B1 to B3, wherein the image processing device estimates the inclination of the machine body based on the amount of displacement between the position of the boundary in the image and the position of the boundary detected from an image acquired by the image acquisition device at a past point in time.

[0221] [Item B5] The work machine according to Item B4, wherein the past point in time is a point in time when the inclination measured by the inclination sensor was less than a reference value.

[0222] [Item B6] The work machine according to Item B4 or B5, wherein the image processing device estimates the pitch angle of the machine body as the inclination based on the amount of displacement.

[0223] [Item B7] The work machine according to any one of Items B1 to B6, wherein the image processing device converts position coordinates of the object in the image into position coordinates in the coordinate system fixed to the ground based on the estimated tilt, and estimates the distance to the object based on the converted position coordinates.

[0224] [Item B8] The work machine according to any one of Items B1 to B7, further comprising a control device that controls the operation of the work machine based on the position of the object in the coordinate system fixed to the ground.

[0225] [Item B9] The work machine according to Item B8, wherein the control device executes at least one of stopping the work machine, slowing down the work machine, and outputting a warning when the distance estimated based on the position of the object in the coordinate system is smaller than a predetermined value.

[0226] [Item B10] The work machine according to item B8 or B9, wherein the work machine is an automatically driven work vehicle or unmanned aerial vehicle, and the control device controls the automatically driven operation of the work machine.

[0227] [Item B11] An image processing device that detects a specific object from an image acquired by an image acquisition device mounted on a work machine, the image processing device detecting the boundary between the sky and a feature other than the sky, and the object from the image, estimating the inclination of the machine based on the position of the boundary in the image, and estimating the position of the object in a coordinate system fixed to the ground based on the position of the object in the image and the estimated inclination.

[0228] [Item B12] A method executed by an image processing device that detects a specific object from an image acquired by an image acquisition device mounted on a work machine, the method including: detecting the boundary between the sky and a feature other than the sky, and the object from the image; estimating the inclination of the machine based on the position of the boundary in the image; and estimating the position of the object in a coordinate system fixed to the ground based on the position of the object in the image and the estimated inclination.

[0229] [Item C1] A work machine that performs work while moving, comprising: an image capture device that captures images in the direction of movement of the work machine; an image processing device that detects specific objects from the images; and a control device that controls the operation of the work machine based on the detection results of the objects, wherein the image processing device has a memory that stores a plurality of trained models for detecting the objects from input images, generates the input image based on the image captured by the image capture device, and detects the object by inputting the input image into one trained model selected from the plurality of trained models in accordance with the environment around the work machine.

[0230] [Item C2] The work machine according to Item C1, wherein the plurality of trained models are associated with a plurality of different brightness levels in the environment surrounding the work machine, and the image processing device selects one trained model that corresponds to the current brightness level of the environment from among the plurality of trained models.

[0231] [Item C3] The work machine according to Item C2, wherein the image processing device determines the brightness of the environment based on at least one of the image, an output from an illuminance sensor, an input from a user, and a lighting state of a light.

[0232] [Item C4] The work machine according to Item C1, wherein the plurality of trained models are associated with a plurality of different time periods, and the image processing device selects one trained model that corresponds to a current time period from among the plurality of trained models.

[0233] [Item C5] The work machine according to Item C4, wherein the plurality of trained models include a day model and a night model, and the image processing device selects the day model during daytime hours and selects the night model during nighttime hours.

[0234] [Item C6] The work machine described in Item C1, wherein the plurality of trained models include a backlit model and a non-backlit model, and the image processing device determines whether or not there is backlight based on at least one of the image, input from a user, and output from an illuminance sensor, and selects the backlit model if there is backlight, and selects the non-backlit model if there is no backlight.

[0235] [Item C7] The work machine according to Item C1, wherein the plurality of trained models correspond to a plurality of different types of crops, and the image processing device selects one trained model that corresponds to the type of crop to be worked on from among the plurality of trained models.

[0236] [Item C8] The work machine according to Item C7, wherein the image processing device identifies the type of crop to be worked on based on the image or input from a user.

[0237] [Item C9] The work machine according to Item C1, wherein the plurality of trained models are associated with a plurality of different weather conditions, and the image processing device selects one trained model that corresponds to current weather conditions from the plurality of trained models.

[0238] [Item C10] The work machine according to Item C9, wherein the image processing device identifies the current weather based on the image or information about the weather obtained from an external device.

[0239] [Item C11] The work machine according to any one of Items C1 to C10, wherein each of the plurality of trained models has a common model architecture and a set of weights that are different from each other.

[0240] [Item C12] The work machine described in any one of Items C1 to C11, wherein the control device moves the work machine along a set route, and the image processing device selects the trained model each time the work machine changes direction on the route.

[0241] [Item C13] The work machine according to any one of Items C1 to C12, wherein the image processing device selects the trained model at predetermined time intervals while the work machine is moving.

[0242] [Item C14] The work machine according to any one of Items C1 to C13, wherein the control device executes at least one of stopping the work machine, slowing down the work machine, and outputting a warning when the object is detected.

[0243] [Item C15] An image processing device that detects a specific object from an image acquired by an image acquisition device mounted on a work machine, comprising: a memory that stores a plurality of trained models for detecting the object from an input image; and an arithmetic circuit, wherein the arithmetic circuit generates the input image based on the image acquired by the image acquisition device, and detects the object by inputting the input image into one trained model selected from the plurality of trained models in accordance with the environment around the work machine.

[0244] [Item C16] A method executed by an image processing device that detects a specific object from an image acquired by an image acquisition device mounted on a work machine, the method including: generating the input image based on the image acquired by the image acquisition device; and detecting the object by inputting the input image into one trained model selected from a plurality of trained models according to the environment around the work machine.

[0245] [Item D1] A work machine that performs work while moving, comprising: an image capture device that captures images in the direction of movement of the work machine; an image processing device that detects a specific object from the images; and a control device that controls the operation of the work machine based on the results of the object detection, wherein the image processing device has a memory that stores a trained model for detecting the object from an input image, generates the input image based on the image captured by the image capture device, detects the object by inputting the input image into the trained model, and re-trains the trained model based on one or more images in which the object is detected out of a plurality of images captured by the image capture device while the work machine is operating.

[0246] [Item D2] The work machine according to Item D1, wherein the image processing device determines whether or not the detection result of the object is correct for each of the one or more images in which the object is detected, based on the operation of the work machine after the object is detected, and if it determines that the detection result of the object is correct, uses the image for relearning the trained model.

[0247] [Item D3] The work machine described in Item D2, wherein the control device stops the work machine when the object is detected and resumes movement of the work machine in response to an input restart command, and the image processing device determines whether the detection result of the object is correct based on the time from when the work machine stops to when it resumes movement.

[0248] [Item D4] The work machine according to Item D1, further comprising an object detection sensor, wherein the image processing device determines whether or not the detection result of the object is correct for each of the one or more images in which the object is detected, based on output from the object detection sensor, and when it is determined that the detection result of the object is correct, uses the image for re-learning the trained model.

[0249] [Item D5] The work machine according to any one of Items D1 to D4, wherein, when it is determined that the detection result of the object is incorrect, the image processing device records information indicating that the detection result is incorrect in association with the image, and determines an image to be used for relearning the trained model based on the information.

[0250] [Item D6] The work machine according to any one of Items D1 to D4, wherein, when the image processing device determines that the detection result of the object is correct, it records information indicating that the detection result is correct in association with the image, and determines an image to be used for relearning the trained model based on the information.

[0251] [Item D7] The work machine according to any one of Items D1 to D6, wherein the image processing device records position information of the object in each of the one or more images in which the object is detected, in association with the image, and performs relearning of the trained model based on the position information.

[0252] [Item D8] The work machine according to any one of Items D1 to D7, wherein after performing the re-learning, the image processing device updates the trained model with the trained model that has been re-learned in accordance with an instruction from a user.

[0253] [Item D9] The work machine described in any of Items D1 to D8, wherein the image processing device repeatedly generates the input image based on the image acquired by the image acquisition device while work is being performed by the work machine, detects the object by inputting the input image into the trained model, and records the input image when the object is detected; and after receiving a signal indicating that the work has been completed, re-trains the trained model based on one or more input images in which the object has been detected.

[0254] [Item D10] An image processing device that detects a specific object from an image acquired by an image acquisition device mounted on a work machine, comprising: a memory that stores a trained model for detecting the object from an input image; and an arithmetic circuit, wherein the arithmetic circuit generates the input image based on the image acquired by the image acquisition device, detects the object by inputting the input image to the trained model, and re-trains the trained model based on one or more images in which the object is detected, out of a plurality of images acquired by the image acquisition device while the work machine is in operation.

[0255] [Item D11] A method for detecting a specific object from an image acquired by an image acquisition device mounted on a work machine, the method comprising: generating an input image based on the image acquired by the image acquisition device; detecting the object by inputting the input image into the trained model; and retraining the trained model based on one or more images in which the object is detected, out of multiple images acquired by the image acquisition device while the work machine is in operation.

[0256] The technology of the present disclosure can be applied to various work machines, for example, agricultural machines such as harvesters, tractors, transplanters, and agricultural drones, construction machines such as backhoes, wheel loaders, and carriers, and snowplows.

[0257] 10: Imaging device, 20: Image processing device, 22: Processor, 24: Memory, 25: Computer program, 27: Trained model, 30: Control device, 100: Agricultural machine, 101: Machine body, 102: Running device, 103: Harvesting device, 104: Conveying device, 105: Threshing device, 106: Tank, 107: Discharge device, 108: Straw waste treatment device, 109: Reel, 110: Cabin, 111: Prime mover, 112: Transmission, 117: Discharge outlet, 120: GNSS unit, 121: GNSS receiver, 122: RTK receiver, 123: Inertial measurement unit (IMU), 124: Processing circuit, 12 5: LiDAR sensor, 126: camera, 127: obstacle sensor, 131: terminal monitor, 132: operation switch group, 133: buzzer, 140: drive unit, 141: power transmission mechanism, 142: light, 150: sensor group, 151: vehicle speed sensor, 152: steering angle sensor, 153: illuminance sensor, 160: control system, 164: storage device, 165-167: ECU, 190: communication device, 200: construction vehicle

Claims

1. A working machine that performs operations while moving, comprising: an image acquisition device that acquires an image in the moving direction of the working machine; an image processing device that detects a specific object from the image; and a control device that controls the operation of the working machine based on the detection result of the object. The image processing device includes a memory that stores a plurality of learned models for detecting the object from an input image, generates the input image based on the image acquired by the image acquisition device, and inputs the input image into one learned model selected from the plurality of learned models according to the environment around the working machine to detect the object. Working machine.

2. The plurality of learned models are associated with a plurality of different brightness levels in the environment around the working machine, and the image processing device selects one learned model corresponding to the current brightness level of the environment from the plurality of learned models. The working machine according to claim 1.

3. The image processing device determines the brightness of the environment based on at least one of the image, the output from an illuminance sensor, the input from a user, and the lighting state of a light. The working machine according to claim 2.

4. The plurality of learned models are associated with a plurality of different time zones, and the image processing device selects one learned model corresponding to the current time zone from the plurality of learned models. The working machine according to claim 1.

5. The plurality of learned models include a daytime model and a nighttime model, and the image processing device selects the daytime model during the daytime time zone and the nighttime model during the nighttime time zone. The working machine according to claim 4.

6. The plurality of learned models include a backlight model and a non-backlight model, and the image processing device determines whether it is backlit based on at least one of the image, the input from a user, and the output from an illuminance sensor. If it is backlit, the backlight model is selected, and if it is not backlit, the non-backlight model is selected. The working machine according to claim 1.

7. The plurality of learned models are associated with a plurality of different crop types, and the image processing device selects one learned model corresponding to the crop type of the work target from among the plurality of learned models. The working machine according to claim 1.

8. The image processing device identifies the crop type of the work target based on the image or an input from a user. The working machine according to claim 7.

9. The plurality of learned models are associated with a plurality of different weather conditions, and the image processing device selects one learned model corresponding to the current weather from among the plurality of learned models. The working machine according to claim 1.

10. The image processing device identifies the current weather based on the image or information on the weather acquired from an external device. The working machine according to claim 9.

11. Each of the plurality of learned models has a common model architecture and different weight groups. The working machine according to any one of claims 1 to 10.

12. The control device moves the working machine along a set path, and the image processing device executes selection of the learned model each time the working machine changes direction on the path. The working machine according to any one of claims 1 to 10.

13. The image processing device executes selection of the learned model at predetermined time intervals during movement of the working machine. The working machine according to any one of claims 1 to 10.

14. When the object is detected, the control device executes at least one of stopping the working machine, decelerating the working machine, and outputting a warning. The working machine according to any one of claims 1 to 10.

15. An image processing device that detects a specific object from an image acquired by an image acquisition device mounted on a working machine, comprising: a memory that stores a plurality of learned models for detecting the object from an input image; and an arithmetic circuit, wherein the arithmetic circuit generates the input image based on the image acquired by the image acquisition device, and inputs the input image into one learned model selected according to the environment around the working machine from among the plurality of learned models to detect the object. Image processing device.

16. A method executed by an image processing apparatus that detects a specific object from an image acquired by an image acquisition device mounted on a work machine, the method comprising: generating the input image based on the image acquired by the image acquisition device; and detecting the object by inputting the input image into one learned model selected from a plurality of learned models according to the environment around the work machine.

Images