Work machine, image processing device, and method for detecting a specific object from an image

Abstract

A work machine performs work while moving. The work machine has an image acquisition device that acquires an image in a moving direction of the work machine, an image processing device that detects a specific object from the image, and a control device that controls an action of the work machine based on a detection result of the object. The image processing device has a memory that stores a learned model for detecting the object from an input image, generates the input image based on the image acquired by the image acquisition device, detects the object by inputting the input image to the learned model, and performs relearning of the learned model based on one or more images in which the object is detected among a plurality of images acquired by the image acquisition device in the action of the work machine.

Description

Technical Field

[0001] This disclosure relates to operating machinery, image processing apparatus, and methods for detecting specific objects from images. Background Technology

[0002] As the next generation of agriculture, research and development is underway towards smart agriculture that effectively utilizes ICT (Information and Communication Technology) and IoT (Internet of Things). Research and development is also progressing towards the automation and unmanned operation of agricultural machinery such as tractors, harvesters, transplanters, and agricultural drones used in fields. For example, agricultural machinery that utilizes positioning systems such as GNSS (Global Navigation Satellite System) capable of precise positioning to move and perform agricultural operations in fields autonomously is gradually becoming practical.

[0003] Patent Document 1 discloses an agricultural machine having a camera capable of capturing images of the machine's path forward in a field. This agricultural machine can detect the presence of obstacles in the field based on the captured images and identify the type of obstacle. The agricultural machine can also perform output control (e.g., deceleration, stopping, warnings for obstacles, etc.) based on a control mode selected from a plurality of control modes according to the type of obstacle.

[0004] Existing technical documents

[0005] Patent documents

[0006] Patent Document 1: Japanese Patent Application Publication No. 2020-178619 Summary of the Invention

[0007] The problem the invention aims to solve

[0008] In agricultural machinery and construction machinery, which detect specific objects based on images (e.g., visible light images, infrared images, or point cloud images) acquired by imaging devices (cameras) or image acquisition devices such as LiDAR, there is a need to improve the detection performance of objects.

[0009] This disclosure provides a technique for further improving the detection performance of objects.

[0010] means for solving problems

[0011] One embodiment of this disclosure is a method performed by an image processing device that detects a specific object from an image acquired by an image acquisition device mounted on a work machine, comprising: acquiring the image from the image acquisition device; acquiring information representing the operational state of the work machine; extracting a partial image from the image representing a region of interest determined based on the operational state of the work machine; generating an input image based on the partial image; and detecting the object by inputting the input image into a pre-generated, fully learned model.

[0012] Other embodiments of this disclosure are methods performed by an image processing device that detects a specific object from an image acquired by an image acquisition device mounted on a work machine, comprising: detecting a boundary between the sky and ground objects other than the sky and the object from the image; estimating the tilt of the machine body based on the position of the boundary in the image; and estimating the position of the object in a coordinate system fixed to the ground based on the position of the object in the image and the estimated tilt.

[0013] Another embodiment of the present disclosure is a method performed by an image processing device that detects a specific object from an image acquired by an image acquisition device mounted on a work machine, comprising: generating the input image based on the image acquired by the image acquisition device; and detecting the object by inputting the input image into a learned model selected from a plurality of learned models according to the environment surrounding the work machine.

[0014] Another embodiment of this disclosure describes a method for detecting a specific object from an image acquired by an image acquisition device mounted on a work machine, comprising: generating an input image based on the image acquired by the image acquisition device; detecting the object by inputting the input image into the learned model; and performing relearning of the learned model based on one or more images from a plurality of images acquired by the image acquisition device during the operation of the work machine in which the object is detected.

[0015] The general or specific embodiments of this disclosure can be implemented by means of apparatus, systems, methods, integrated circuits, computer programs, or computer-readable non-transitory storage media, or any combination thereof. Computer-readable storage media can include both volatile and non-volatile storage media. An apparatus can also consist of a plurality of apparatuses. When an apparatus consists of two or more apparatuses, the two or more apparatuses can be configured within one device or separately within two or more separate devices.

[0016] Invention Effects

[0017] According to embodiments of this disclosure, in a work machine that detects a specific object based on an image acquired by an imaging device, the object detection performance can be improved. Attached Figure Description

[0018] Figure 1 This is a block diagram illustrating the general structure of an agricultural machine as an example of a working machine in an exemplary embodiment of this disclosure.

[0019] Figure 2 This is a block diagram illustrating an example of the structure of an image processing device.

[0020] Figure 3 This is a flowchart illustrating an example of the operation of an image processing device.

[0021] Figure 4 This is a side view that schematically represents an example of agricultural machinery.

[0022] Figure 5 This is a block diagram illustrating the structure of agricultural machinery.

[0023] Figure 6 This is a diagram illustrating an example of the path taken by agricultural machinery as it harvests crops in a field.

[0024] Figure 7 This is a schematic diagram illustrating the use of a camera mounted on agricultural machinery to detect objects (people).

[0025] Figure 8 It is a diagram that schematically illustrates the process flow performed by the ECU (Engine Control Unit).

[0026] Figure 9A This is an example of a photograph taken by a camera.

[0027] Figure 9B This diagram illustrates an example of the area of ​​interest that can be selected when agricultural machinery turns right.

[0028] Figure 9C It means and Figure 9B The image shows a portion of the image corresponding to the region of interest.

[0029] Figure 9D This is a diagram illustrating an example of the area of ​​interest that can be selected when agricultural machinery turns left.

[0030] Figure 9E It means and Figure 9D The image shows a portion of the image corresponding to the region of interest.

[0031] Figure 10AThis is a diagram illustrating an example of the area of ​​interest that can be selected when agricultural machinery is moving at a relatively high speed.

[0032] Figure 10B It means and Figure 10A The image shows a portion of the image corresponding to the region of interest.

[0033] Figure 11 This is a diagram showing an example of agricultural machinery moving around the outermost perimeter of a work area in a field.

[0034] Figure 12 It is a three-dimensional diagram that schematically represents the configuration relationship of the camera coordinate system Σc, the vehicle coordinate system Σv, the world coordinate system Σw, and the reference plane Re.

[0035] Figure 13 This is a flowchart illustrating an example of image processing performed by the image processing apparatus in Embodiment 2.

[0036] Figure 14 This is a diagram showing an example of the boundary between the sky and objects on the ground detected from a captured image.

[0037] Figure 15 This is a flowchart illustrating a variation of implementation method 2.

[0038] Figure 16 This is a flowchart illustrating another variation of implementation method 2.

[0039] Figure 17 This is a block diagram illustrating a structural example of the image processing apparatus in Embodiment 3.

[0040] Figure 18 It is a table representing an example of the correspondence between a plurality of learned models stored in memory and the environment.

[0041] Figure 19 This is a flowchart illustrating an example of the operation of the image processing apparatus in Embodiment 3.

[0042] Figure 20 This is a flowchart illustrating an example of the actions of an image processing device when selecting a learned model each time agricultural machinery changes direction.

[0043] Figure 21 This is a flowchart illustrating an example of the model selection process in step S320.

[0044] Figure 22 This is a flowchart illustrating a specific example of the operation of the image processing apparatus in this embodiment.

[0045] Figure 23 This is a diagram representing an example of the detection results for an object.

[0046] Figure 24 This is a diagram illustrating an example of a construction vehicle. Detailed Implementation

[0047] (Definition of the term)

[0048] In this disclosure, "operating machinery" refers to machinery used for specific purposes such as agriculture or construction. In this disclosure, "agricultural machinery" refers to operating machinery used for agricultural purposes. "Construction machinery" refers to operating machinery used for construction purposes. "Operation" includes, for example, agricultural operations, construction operations, rubble removal operations, snow removal operations, etc. The operating machinery of this disclosure can be mobile machinery capable of performing operations while moving. Examples of agricultural machinery include tractors, harvesters, rice transplanters, passenger management machines, plant transplanters, lawnmowers, seeders, fertilizer applicators, agricultural mobile robots, and agricultural unmanned aerial vehicles (e.g., drones). Examples of construction machinery include, for example, backhoe excavators, wheel loaders, transport vehicles, construction mobile robots, and construction unmanned aerial vehicles. Agricultural operating vehicles such as tractors or combine harvesters, or construction operating vehicles, can function as "operating machinery" individually, and the operating machine (working machinery) mounted on or towed by the operating vehicle and the operating vehicle as a whole can also function as "operating machinery." Agricultural machinery performs agricultural operations such as tilling, sowing, pest control, fertilization, and planting or harvesting crops on fields. Construction machinery performs operations such as moving sand, rubble, and other materials at construction sites. Sometimes these operations are referred to as "ground operations" or simply "operations." Sometimes, the situation where vehicle-type construction machinery is operating while moving is referred to as "operational travel."

[0049] "Automatic driving" refers to the movement of agricultural machinery and other work machines controlled by control devices without manual operation by a driver. Agricultural machinery operating in automatic driving mode is sometimes called "automatic agricultural machinery" or "robotic agricultural machinery." In automatic driving, not only the movement of the machinery but also the actions of the work itself (such as the actions of the workpieces mounted on the machinery) can be automatically controlled. When the machinery is a vehicle-type machine, the situation where it moves automatically is called "automatic driving." The control device can control at least one of the following necessary for the movement of the machinery: steering, speed adjustment, initiation of movement, and stopping. When controlling machinery with workpieces mounted on it, the control device can also control the lifting and lowering of the workpieces, the initiation and stopping of their actions. Movement based on automatic driving includes not only the movement of the machinery along a predetermined path towards a destination but also movement following and tracking a target. Automatically driven machinery can also move partially based on user instructions. In addition to automatic driving mode, automatically driven machinery can also operate in manual driving mode, where it is moved by the driver. The situation where the machinery is steered without manual intervention but by the action of the control device is called "automatic steering." Part or all of the control device can be located outside the machine. Communication of control signals, commands, or data is possible between the external control device and the machine. Automatically driven machine can sense its surroundings and move autonomously without human intervention. Autonomously moving machine can travel unattended in or outside fields (e.g., on roads). During autonomous movement, obstacle detection and obstacle avoidance maneuvers are also possible.

[0050] In this disclosure, "image acquisition device" means a device capable of acquiring images or similar information. Examples of image acquisition devices include cameras capable of acquiring images such as visible light images, infrared images, and ultraviolet images; LiDAR sensors capable of acquiring point cloud image data; and radars capable of acquiring image-like information using short-wavelength electromagnetic waves such as millimeter waves.

[0051] One example of a "control device" in this disclosure is a computing device having at least one processor and at least one memory storing a computer program (code) that defines the control process executed by the processor. Other examples of a "control device" are computing devices having hardware accelerators such as FPGAs (Field-Programmable Gate Arrays), ASSPs (Application Specific Standard Products), or ASICs (Application-Specific Integrated Circuits) configured to perform the control process.

[0052] Similarly, one example of an "image processing apparatus" in this disclosure is a computing device having at least one processor and at least one memory storing a computer program (code) defining an image processing procedure executed by the processor. Other examples of an "image processing apparatus" are computing devices having a hardware accelerator such as an FPGA or ASIC configured to perform an image processing procedure.

[0053] In this disclosure, "processor" refers to hardware electronic circuits such as CPU (Central Processing Unit), GPU (Graphics Processing Unit), DSP (Digital Signal Processor), ISP (Image Signal Processor), or NPU (Neural Network Processing Unit). "Memory" refers to hardware electronic circuits such as ROM (Read Only Memory) or RAM (Random Access Memory). A portion of the memory may be a storage medium connected to the processor via wires or a network. These hardware electronic circuits can be mounted using more than one integrated circuit (IC) or large-scale integrated circuit (LSI). The functional units or blocks within the electronic circuits, as well as related components, can be individually manufactured as separate integrated circuit chips, or some or all of these functional units or blocks can be combined and manufactured as a single integrated circuit chip.

[0054] The program that defines the actions of the processor is designed to cause the processor to perform one or more functions, operations, steps or processes in the embodiments of the present invention.

[0055] The embodiments of this disclosure will now be described. However, sometimes unnecessary detailed descriptions are omitted. For example, detailed descriptions of well-known matters and repetitive descriptions related to substantially the same structures are sometimes omitted. This is to avoid making the following description unnecessarily lengthy and to facilitate understanding by those skilled in the art. Furthermore, the inventors have provided the drawings and the following description to enable those skilled in the art to fully understand this disclosure, but do not intend to limit the subject matter of the claims by these. In the following description, the same reference numerals are used to denote components having the same or similar functions.

[0056] The following embodiments are illustrative, and the technology disclosed herein is not limited to these embodiments. For example, the numerical values, shapes, materials, steps, order of steps, and layout of the display screen shown in the following embodiments are merely examples, and various changes can be made as long as there is no technical contradiction. In addition, one method can be combined with other methods.

[0057] The following describes several embodiments of applying the technology of this disclosure to agricultural machinery as an example of operational machinery. In the following description, the various technologies described with respect to agricultural machinery can also be applied to construction machinery such as construction vehicles used at construction sites, operational vehicles used at disaster sites, snowplows used in snowy areas, and unmanned aerial vehicles (UAVs) used for tasks such as transporting or monitoring goods.

[0058] (Implementation Method 1)

[0059] Figure 1 This is a block diagram illustrating the general structure of agricultural machinery 100 in an exemplary embodiment of the present disclosure. Figure 1 The agricultural machinery 100 shown includes a camera 10, an image processing device 20, and a control device 30. The agricultural machinery 100 is configured to perform agricultural operations while moving. Although in Figure 1 Although not shown, agricultural machinery 100 may have a power source such as an internal combustion engine or an electric motor for driving, as well as a driving device such as wheels with tires or a propeller or other means of movement (e.g., driving or flying).

[0060] The imaging device 10 is a camera that acquires images of the agricultural machinery 100 in its direction of movement. The imaging device 10 is mounted on the agricultural machinery 100 in a manner capable of acquiring images of the direction of movement (e.g., front, rear, right, or left). The orientation of the imaging device 10 (i.e., the direction of the optical axis of the optical system in the imaging device 10) does not necessarily need to be consistent with the direction of movement of the agricultural machinery 100; it can also be tilted relative to the direction of movement. For example, the imaging device 10 can be positioned diagonally downwards relative to the front, rear, right, or left of the agricultural machinery 100.

[0061] Agricultural machinery 100 may have a plurality of imaging devices 10 mounted in different directions from each other. The imaging devices 10 generate image data by performing imaging during the movement of agricultural machinery 100. In one embodiment, agricultural machinery 100 may be configured to generate moving image data at a predetermined frame rate, such as 30fps or 60fps.

[0062] exist Figure 1 In the example, the imaging device 10 is used as an example of an image acquisition device. Alternatively, or based on the imaging device 10, a device capable of acquiring point cloud data similar to an image, such as a LiDAR sensor, may be used. Alternatively, a radar capable of acquiring distribution information of surrounding objects similar to an image may be used. Such a device capable of acquiring images or similar data can be used as an "image acquisition device." In this specification, data generated by the image acquisition device to produce dynamic or static images, or data similar to images, is represented as "acquiring an image."

[0063] Image processing apparatus 20 is a computing device that processes images acquired by imaging apparatus 10. Image processing apparatus 20 may have one or more processors and one or more memories. Image processing apparatus 20 may be configured or programmed to perform processing to detect specific objects from images acquired by imaging apparatus 10 (hereinafter, sometimes referred to as "captured images"). Specific objects may be, for example, people, animals, other agricultural machinery, vehicles, crops, obstacles such as stones or rocks, or any combination of these obstacles. In one embodiment, the specific object is a person, and image processing apparatus 20 is configured or programmed to perform processing to detect people from captured images.

[0064] The control device 30 is a device for controlling the movement of the agricultural machinery 100. The control device 30 may be, for example, a computing device such as an electronic control unit (ECU). In this embodiment, the control device 30 controls the movement of the agricultural machinery 100 based on the detection results of the object by the image processing device 20. For example, the control device 30 may be configured to stop the movement of the agricultural machinery 100 or to sound an alarm via a sound output device such as a buzzer when a specific object is detected by the image processing device 20. Through such control, collisions between the agricultural machinery 100 and objects can be avoided, or attention can be drawn to objects (e.g., people). If the agricultural machinery 100 has an automatic driving function, the control device 30 may also be configured to perform automatic driving control.

[0065] Figure 2 This is a block diagram illustrating an example of the structure of the image processing apparatus 20. Figure 2The image processing apparatus 20 shown has one or more processors 22 and one or more memories 24. The processor 22 is, for example, an electronic circuit (arithmetic circuit) that performs computational processing, such as a CPU, GPU, or NPU. The image processing apparatus 20 may also have multiple processors. Image processing, as described later, can also be performed collaboratively by multiple processors. The memory 24 may be, for example, a ROM such as EPROM (Erasable Programmable Read-Only Memory) or EEPROM (Electrically Erasable Programmable Read-Only Memory), or a RAM such as DRAM (Dynamic Random Access Memory) or SRAM (Static Random Access Memory). The memory 24 stores a computer program 25 executed by the processor 22 and a learned model 27 for detecting objects from an input image. The program 25 and the learned model 27 may also be stored separately in multiple memories. The processor 22 executes a computer program 25 stored in the memory 24 to perform the process of detecting specific objects from the captured image. Alternatively, a hardware accelerator such as an FPGA or ASIC configured to perform the image processing described in this embodiment can be installed in the image processing apparatus 20 to replace the processor 22, such as a CPU or GPU. Such a hardware accelerator can replace the processing performed by the processor in the following description.

[0066] The learned model 27 can be, for example, a machine learning model trained using algorithms based on machine learning or artificial intelligence (AI) techniques, such as convolutional neural networks (CNNs) or visual transducers (ViTs). The learned model 27 can also be, for example, a model that detects objects from images based on object detection algorithms such as SSD (Single Shot Multibox Detector), YOLO (You Only Look Once), R-CNN (Regions with Convolutional Neural Network), Fast R-CNN, Faster R-CNN, and RetinaNet (Focus Loss Network).

[0067] The image processing device 20 generates an input image for input to the model 27 by performing necessary preprocessing on the captured image generated by the imaging device 10. The image processing device 20 then inputs the input image into the model 27 to detect specific objects (e.g., people) from the image.

[0068] In this embodiment, the image processing device 20 does not detect objects from the entire captured image obtained by the imaging device 10, but rather from a region of interest that is part of the captured image. Specifically, the image processing device 20 extracts a portion of the image representing the region of interest from the captured image and generates an input image for the learned model 27 by performing prescribed preprocessing on this portion of the image. At this time, the image processing device 20 dynamically changes at least one of the position and size of the region of interest according to the operating state of the agricultural machinery 100. For example, when the agricultural machinery 100 turns right, the image processing device 20 moves the region of interest to the right, and when the agricultural machinery 100 turns left, the image processing device 20 moves the region of interest to the left. Alternatively, when the agricultural machinery 100 is tilted upwards, the image processing device 20 may move the region of interest downwards, and when the agricultural machinery 100 is tilted downwards, the image processing device 20 may move the region of interest upwards. In addition, the image processing device 20 may change the size of the region of interest according to the moving speed of the agricultural machinery 100. For example, when the moving speed of the agricultural machinery 100 is high, the region of interest may be set to be smaller. This process enables more accurate detection of specific objects such as people.

[0069] Typically, in processing images to detect specific objects using machine learning models, the image is resized to fit the specified number of pixels (e.g., 300×300 pixels) of the model and then input into it. The resized input image generally has fewer pixels than the original image. Therefore, when the original image is large, the reduction in sharpness when converted to an input image becomes significant, and the performance of object detection may be significantly degraded.

[0070] Therefore, in this embodiment, the image processing apparatus 20 extracts a partial image from the captured image showing the region of interest determined based on the operating state of the agricultural machinery 100, and generates an input image based on this partial image. Depending on the operating state of the agricultural machinery 100, regions that may obstruct the movement of the agricultural machinery 100 when objects such as people are present are adaptively selected as regions of interest. By adjusting the size of the partial image to a specified number of pixels suitable for the model, the reduction in clarity of the regions of interest where objects may be present can be suppressed compared to adjusting the size of the original captured image. Thus, objects that may become obstacles to the movement of the agricultural machinery 100 can be detected more accurately.

[0071] Figure 3 This is a flowchart illustrating an example of the operation of the image processing device 20. Figure 3 In the example shown, during the operation of agricultural machinery 100, image processing device 20 detects specific objects from captured images by performing steps S110 to S150, and sends the detection results to control device 30. Steps S110 to S150 are repeatedly performed during the operation of agricultural machinery 100.

[0072] In step S110, the image processing device 20 acquires the image acquired by the imaging device 10 and information indicating the operational state of the agricultural machinery 100. The information indicating the operational state of the agricultural machinery 100 may be, for example, information regarding at least one of the following: the agricultural machinery 100's moving speed, turning state, and tilting state. The image processing device 20 may be configured to acquire the information indicating the operational state of the agricultural machinery 100 from, for example, a control device 30. The control device 30 may be configured to generate the information indicating the operational state of the agricultural machinery 100 based on signals from one or more sensors mounted on the agricultural machinery 100.

[0073] In step S120, the image processing device 20 determines a region of interest from the image acquired by the imaging device 10 based on the operating state of the agricultural machinery 100, and extracts a portion of the image representing that region of interest. The region of interest is the area in the image that is the object of the detection processing. The image processing device 20 may be configured to acquire information about at least one of the agricultural machinery 100's moving speed, turning state, and tilting state, and change the region of interest based on this information. For example, when the agricultural machinery 100's moving speed is high, the image processing device 20 may reduce the region of interest. Alternatively, when the agricultural machinery 100 turns right, the image processing device 20 may move the region of interest to the right, and when the agricultural machinery 100 turns left, the image processing device 20 may move the region of interest to the left. In this case, the larger the turning angle of the agricultural machinery 100 when turning, the more the image processing device 20 can move the region of interest. Additionally, the agricultural machinery 100 may have a tilt sensor that measures the amount of tilt of the agricultural machinery 100. In this case, the image processing device 20 can change the position of the region of interest based on the measured amount of tilt. In one embodiment, a tilt sensor measures the pitch angle of the agricultural machinery 100 as a tilt amount. In this case, the image processing device 20 can move the region of interest up or down based on the measured pitch angle. For example, when the agricultural machinery 100 tilts upward, such as when the agricultural machinery 100 is traveling uphill, the upper part of the captured image may correspond to the sky. In this case, since the object to be detected may be located at the lower part of the image, the image processing device 20 can move the region of interest downward. Conversely, when the agricultural machinery 100 tilts downward, the image processing device 20 can move the region of interest upward.

[0074] In step S130, the image processing device 20 generates an input image for input to the learned model 27 based on the extracted partial image. The image processing device 20 generates the input image through preprocessing, including compressing (sizing) the number of pixels in the partial image to a preset number of pixels. The preprocessing may include various processes such as normalization, color space transformation, and noise removal, in addition to image resizing. Therefore, an input image representing the region of interest appropriately selected according to the operating state of the agricultural machinery 100 is generated.

[0075] In step S140, the image processing device 20 detects a specific object (e.g., a person) by inputting the input image into the learned model 27 stored in the memory 24. For example, the image processing device 20 may be configured to output the coordinates of the rectangular box (called a "bounding box") representing the region where the object exists, along with the width and height of the bounding box, as a detection result when a specific object is present in the image. Alternatively, the image processing device 20 may output a signal indicating whether a specific object exists in the input image as a detection result. The image processing device 20 may also determine that an object exists in the input image if the distance from the capturing device 10 to the object, calculated based on the position of the bounding box in the input image, is less than a threshold.

[0076] In step S150, the image processing device 20 sends the detection result of the object to the control device 30. For example, the image processing device 20 may transmit a signal indicating whether a specific object exists in the input image to the control device 30. Alternatively, the image processing device 20 may only send a signal indicating the presence of an object to the control device 30 if a specific object is detected in the input image. Furthermore, the image processing device 20 may not only detect a specific object in the input image but also estimate the distance from the imaging device to the object based on the input image, and only send a signal indicating the presence of an object to the control device 30 if the distance is less than a threshold.

[0077] When the image processing device 20 receives a signal indicating a detection result, the control device 30 controls the operation of the agricultural machinery 100 based on that detection result. For example, the control device 30 may be configured to stop the movement of the agricultural machinery 100 or sound an alarm via a buzzer or speaker when it receives a signal indicating that an object has been detected. By doing so, collisions between the agricultural machinery 100 and objects can be avoided, or attention can be alerted to objects (e.g., people) or passengers of the agricultural machinery 100.

[0078] The following describes an embodiment in which the technology of this disclosure is applied to a harvester, exemplified as agricultural machinery 100. The technology of this disclosure is not limited to harvesters, but can also be applied to other types of agricultural machinery, such as tractors, transplanters, or agricultural drones. Furthermore, the technology of this disclosure can also be applied to operational machinery used for purposes other than agriculture (e.g., construction vehicles, snowplows, or mobile robots). In the following description, an image acquisition device (camera) is used as an example, but other types of image acquisition devices, such as LiDAR sensors capable of acquiring point cloud data similar to images, or radar capable of acquiring distance distribution information of surrounding objects similar to images, can also be used.

[0079] [1. Structure]

[0080] Figure 4 This is a side view schematically illustrating an example of agricultural machinery 100. The agricultural machinery 100 in this embodiment is, for example, a combine harvester. The agricultural machinery 100 performs tasks such as harvesting crops in a field, threshing the harvested crops, and discharging the threshed harvest. Crops include, for example, rice, wheat, corn, soybeans, and other plants capable of producing grains. Figure 4 The accompanying diagram shows F, B, U, and D representing front, back, top, and bottom, respectively.

[0081] Agricultural machinery 100 has a body 101 and a running gear 102. Figure 4 The illustrated running gear 102 has a plurality of wheels (tracked) equipped with tracks. The running gear 102 may also have tire-equipped wheels instead of tracks. A driver's cab 110 is located above the body 101.

[0082] A harvesting device 103 for harvesting crops is installed in front of the traveling device 102 with adjustable height. Above the harvesting device 103, a reel 109 for raising the stems of the crops is installed with adjustable height. Behind the cab 110, a threshing device 105 and a storage tank 106 for storing the harvest are arranged in a left-right direction. The threshing device 105 threshes the harvested crops. The storage tank 106 stores the harvested grains and other harvested materials obtained through threshing. Behind the threshing device 105, a straw discharge treatment device 108 is installed. The straw discharge treatment device 108 cuts the stems and other parts of the harvested materials, after the grains and other harvested materials have been removed, into smaller pieces and discharges them to the outside.

[0083] A conveying device 104 for transporting the harvested crop is provided between the harvesting device 103 and the threshing device 105. A discharge device 107 for discharging the harvested crop from the tank 106 is provided in the tank 106. The harvested crop is discharged to the outside from the discharge port 117 located at the top of the cylindrical discharge device 107. The discharge device 107 is capable of undulating and rotating movements, and the position of the discharge port 117 can be changed. The structure and operation of various devices that perform harvesting operations, such as the harvesting device 103, the conveying device 104, the threshing device 105, the discharge device 107, the straw discharge and treatment device 108, and the reel 109, are known, therefore, their detailed descriptions are omitted here.

[0084] The agricultural machinery 100 in this embodiment can operate in both manual and automatic driving modes. In automatic driving mode, the agricultural machinery 100 can harvest crops in the field while operating without human intervention.

[0085] like Figure 4 As shown, the agricultural machinery 100 has a prime mover (engine) 111 and a transmission (gearbox) 112. Inside the cab 110, there is a driver's seat, control levers, an operating terminal (terminal monitor), and a set of switches for operation.

[0086] Agricultural machinery 100 has a plurality of sensing devices for sensing the environment surrounding the agricultural machinery 100. Figure 4 In the example shown, the plurality of sensing devices include a laser sensor 125, a plurality of cameras 126, and a plurality of millimeter-wave radars 127.

[0087] The laser sensor 125 is a ranging device that measures the distance to a reflecting point by emitting a laser and detecting its reflected light; it is also known as a LiDAR sensor. By changing the direction of the laser emission, the laser sensor 125 can obtain information on the distance distribution of surrounding objects on the ground. Figure 4 The laser sensor 125 shown is disposed at the front of the agricultural machinery 100. The laser sensor 125 may also be disposed at the side or rear of the agricultural machinery 100. The laser sensor 125 may include a light source for generating laser light, a detector for detecting reflected light, and processing circuitry for processing the signal of the detected reflected light. The laser sensor 125 may also include a beam scanner for changing the direction of the emitted beam. The laser sensor 125 may be configured to generate sensor data, such as point cloud data representing the distance and direction of each measurement point to an object in the environment surrounding the agricultural machinery 100, or the three-dimensional or two-dimensional coordinate values ​​of each measurement point. The sensor data output from the laser sensor 125 is processed by the control device of the agricultural machinery 100. The control device can measure the height or lodging degree of crops present around the agricultural machinery 100 based on the sensor data, and adjust the height of the harvesting device 103 or the vehicle speed according to the crop height or lodging degree. The point cloud data output from the laser sensor 125 can also be used for object detection.

[0088] Camera 126 is an example of a photographing device that captures images of the environment surrounding agricultural machinery 100 and generates image data. Camera 126 can be positioned, for example, in front of, behind, to the left or right of agricultural machinery 100. The images acquired by camera 126 are sent to a control device mounted on agricultural machinery 100. These images are used to detect obstacles such as people around agricultural machinery 100 during autonomous driving via image processing.

[0089] The millimeter-wave radar 127 is a sensor used to detect metallic objects such as vehicles present around agricultural machinery 100. Figure 4 In the example shown, two millimeter-wave radars 127 are positioned at the front and rear of the agricultural machinery 100. The millimeter-wave radars 127 can also be configured at other locations, such as the sides of the agricultural machinery 100.

[0090] The agricultural machinery 100 also includes a GNSS unit 120. The GNSS unit 120 includes a GNSS receiver, which functions as a positioning device to acquire positioning data of the agricultural machinery 100. The GNSS receiver may have an antenna for receiving signals from GNSS satellites and a processor for calculating the position of the agricultural machinery 100 based on the signals received by the antenna. The GNSS unit 120 receives satellite signals transmitted from a plurality of GNSS satellites and performs positioning based on the satellite signals. GNSS is a general term for satellite positioning systems such as GPS (Global Positioning System), QZSS (Quasi-Zenith Satellite System, e.g., Guide), GLONASS, Galileo, and BeiDou. In this embodiment, the GNSS unit 120 is located on the upper part of the cab 110, but it may also be located in other positions.

[0091] GNSS unit 120 may include an inertial measurement unit (IMU). Signals from the IMU can be used to supplement position data. The IMU is capable of measuring the tilt and minute movements of the agricultural machinery 100. By supplementing satellite-based position data with data acquired by the IMU, positioning performance can be improved. The IMU may also be located at a different location than GNSS unit 120.

[0092] The prime mover 111 can be, for example, a diesel engine. An electric motor can also be used instead of a diesel engine. The transmission 112 can change the propulsion and speed of the agricultural machinery 100 by changing the speed. The transmission 112 can also switch between forward and reverse movement of the agricultural machinery 100.

[0093] In an embodiment where the agricultural machinery 100 has a tracked travel device 102, the travel direction of the agricultural machinery 100 can be changed by making the rotational speeds of the left and right wheels, which are equipped with infinite tracks, different, or by making the rotation directions of the left and right wheels different. In an embodiment where the agricultural machinery 100 has a travel device including wheels with tires, the control device of the agricultural machinery 100 can change the travel direction of the agricultural machinery 100 by controlling the power steering device to change the steering wheel angle (steering angle).

[0094] Figure 4 The agricultural machinery 100 shown can be driven by a man, but it can also be operated autonomously. In this case, components necessary only for manned operation, such as the cab 110, steering mechanism, and driver's seat, may not be included in the agricultural machinery 100. The autonomous agricultural machinery 100 can drive itself or be operated remotely by a user.

[0095] Figure 5 This is a block diagram representing a structural example of agricultural machinery 100. Figure 5 The agricultural machinery 100 shown includes a GNSS unit 120, a laser sensor 125, a camera 126, a millimeter-wave radar 127, a terminal monitor 131, an operating switch assembly 132, a buzzer 133, a drive unit 140, a power transmission mechanism 141, a light 142, a sensor assembly 150, a control system 160, and a communication device 190. These components are connected to each other via a bus in a manner that enables them to communicate with each other.

[0096] The GNSS unit 120 includes a GNSS receiver 121, an RTK receiver 122, an inertial measurement unit (IMU) 123, and processing circuitry 124. The sensor group 150 includes various sensors such as a vehicle speed sensor 151, a steering angle sensor 152, and an illuminance sensor 153. The control system 160 includes a storage device 164 and electronic control units (ECUs) 165, 166, and 167. Figure 5 The diagram shows the components that are relatively closely related to the automatic driving actions of the agricultural machinery 100, while illustrations of other components are omitted.

[0097] The GNSS receiver 121 of the GNSS unit 120 receives satellite signals transmitted from a plurality of GNSS satellites and generates GNSS data based on the satellite signals. The GNSS data is generated, for example, in a format specified by NMEA-0183. The GNSS data may include, for example, values ​​representing the identification number, elevation angle, azimuth angle, and received signal strength of the corresponding satellite from which the satellite signals are received.

[0098] Figure 5The GNSS unit 120 shown is capable of locating the agricultural machinery 100 using RTK (Real-Time Kinematic) GNSS. In RTK-GNSS-based positioning, in addition to satellite signals transmitted from multiple GNSS satellites, a correction signal transmitted from a reference station is also utilized. The reference station can be located near the field where the agricultural machinery 100 operates (e.g., within 10 km of the agricultural machinery 100). Based on the satellite signals received from the multiple GNSS satellites, the reference station generates a correction signal, for example, in RTCM format, and transmits it to the GNSS unit 120. The RTK receiver 122, including an antenna and a modem, receives the correction signal transmitted from the reference station. The processing circuitry 124 of the GNSS unit 120 corrects the positioning result of the GNSS receiver 121 based on the correction signal. By using RTK-GNSS, positioning can be achieved with an accuracy of, for example, a few centimeters. High-precision RTK-GNSS-based positioning provides location data including latitude, longitude, and altitude information. The GNSS unit 120 calculates the position of the agricultural machinery 100 at a frequency of, for example, approximately 1 to 10 times per second.

[0099] Furthermore, the positioning method is not limited to RTK-GNSS; any positioning method (interferometric positioning or relative positioning, etc.) that can obtain the required accuracy of position data can be used. For example, positioning using VRS (Virtual Reference Station) or DGPS (Differential Global Positioning System) can also be performed. If the required accuracy of position data can be obtained even without using correction signals transmitted from the reference station, position data can be generated without using correction signals. In this case, the GNSS unit 120 may not need to have an RTK receiver 122.

[0100] IMU123 can be equipped with a three-axis vehicle speed sensor and a three-axis gyroscope. IMU123 can also be equipped with an orientation sensor such as a three-axis geomagnetic sensor. IMU123 functions as a motion sensor, outputting signals representing various quantities such as acceleration, velocity, displacement, and attitude of the agricultural machinery 100. In addition to satellite signals and correction signals, processing circuitry 124 can estimate the position and orientation of the agricultural machinery 100 with higher accuracy based on the signals output from IMU123. The signals output from IMU123 can be used to correct or supplement the position calculated based on satellite signals and correction signals. IMU123 outputs signals at a higher frequency than GNSS receiver 121. Using this high-frequency signal, processing circuitry 124 can measure the position and orientation of the agricultural machinery 100 at even higher frequencies (e.g., above 10 Hz). Alternatively, a three-axis vehicle speed sensor and a three-axis gyroscope can be used instead of IMU123. Furthermore, IMU123 can be configured as a different device from GNSS unit 120. The IMU123 functions as a tilt sensor to measure the amount of tilt (e.g., pitch, roll, and yaw) relative to the reference attitude of the agricultural machinery 100.

[0101] Camera 126 is an example of a photographing device for capturing images of the environment surrounding agricultural machinery 100. Camera 126 may have an image sensor such as a CCD (Charge Coupled Device) or CMOS (Complementary Metal Oxide Semiconductor). In addition, camera 126 may have an optical system including one or more lenses and signal processing circuitry. During the operation of agricultural machinery 100, camera 126 captures images of the environment surrounding agricultural machinery 100, generating image data (e.g., moving images). Camera 126 may be capable of capturing moving images at a frame rate of 3 frames per second (fps) or higher. Images generated by camera 126 may be used, for example, for obstacle detection such as people. Images generated by camera 126 may also be used for positioning or remote monitoring. Multiple cameras 126 may be installed at different locations on agricultural machinery 100, or a single camera may be installed. A visible camera that generates visible light images and an infrared camera that generates infrared images may also be installed separately. Both a visible camera and an infrared camera may also be installed. Infrared cameras can be used for obstacle detection at night.

[0102] Millimeter-wave radar 127 is installed to detect obstacles, including metallic ones, such as vehicles, around agricultural machinery 100. When an object is located closer to the millimeter-wave radar 127 than a predetermined distance, the millimeter-wave radar 127 outputs a signal indicating the presence of an obstacle. For example... Figure 4As shown, a plurality of millimeter-wave radars 127 can be positioned at different locations on the agricultural machinery 100. By having a plurality of millimeter-wave radars 127, blind spots can be reduced when monitoring obstacles around the agricultural machinery 100.

[0103] Buzzer 133 is a sound output device that emits an alarm sound to notify of abnormalities. For example, during autonomous driving, buzzer 133 sounds an alarm when an obstacle is detected. Buzzer 133 is controlled by control system 160.

[0104] The drive unit 140 includes a prime mover 111, a transmission unit 112, and other devices necessary for driving the agricultural machinery 100. The prime mover 111 may be an internal combustion engine, such as a diesel engine. The drive unit 140 may replace the internal combustion engine or have an electric motor for traction along with the internal combustion engine.

[0105] The power transmission mechanism 141 transmits the power generated by the prime mover 111 to various devices that perform harvesting actions. These devices include a harvesting device 103, a conveying device 104, a threshing device 105, a discharge device 107, a straw discharge and treatment device 108, and a reel 109. The agricultural machinery 100 may also have a power source (such as an electric motor) that supplies power to at least one of the devices performing these harvesting actions, separate from the prime mover 111.

[0106] The lamp 142 is, for example, a device for illuminating the surroundings of the agricultural machinery 100, such as a headlight or work light. A plurality of lamps 142 may be mounted on the agricultural machinery 100. Each lamp 142 includes more than one light source. Each light source may be, for example, a light-emitting diode (LED), a halogen lamp, or a xenon lamp.

[0107] Vehicle speed sensor 151 is a sensor that measures the travel speed of agricultural machinery 100. For example, vehicle speed sensor 151 measures the rotational speed of the wheels or axles, and calculates the vehicle speed based on this measurement. Steering angle sensor 152 is a sensor that measures the steering angle of the steering wheel. Illumination sensor 153 is a sensor that measures the illuminance of the surrounding environment and can be configured outside or inside the cab of agricultural machinery 100.

[0108] Storage device 164 includes, for example, one or more storage media such as flash memory or hard disk. Storage device 164 stores various data generated by GNSS unit 120, laser sensor 125, camera 126, millimeter-wave radar 127, sensor group 150, and ECUs 165, 166, and 167. The data stored in storage device 164 may include map data of the area containing the field on which agricultural machinery 100 performs agricultural operations and data for target paths used in autonomous driving.

[0109] ECU165 controls the overall movement of agricultural machinery 100. ECU165 controls the movement of agricultural machinery 100 by controlling the prime mover 111, transmission 112, travel device 102, power transmission mechanism 141, etc., which are included in the drive unit 140.

[0110] ECU 166 performs calculations and control for autonomous driving based on data output from GNSS unit 120, laser sensor 125, camera 126, millimeter-wave radar 127, and sensor group 150. For example, ECU 166 determines the position and orientation of agricultural machinery 100 based on data output from GNSS unit 120. During autonomous driving, ECU 166 performs calculations required for agricultural machinery 100 to travel along a predetermined target path based on the position and orientation of agricultural machinery 100. ECU 166 can also perform processing to generate a target path from the starting point of the agricultural machinery 100's movement to its destination.

[0111] ECU 167 performs processing based on images acquired by camera 126 to detect specific objects (e.g., people) located around agricultural machinery 100. Although ECU 167 in this embodiment is an edge computing device mounted on agricultural machinery 100, at least a portion of the functions of ECU 167 can be performed by a computer outside agricultural machinery 100 (e.g., a server computer in the cloud).

[0112] Through the actions of these ECUs, the control system 160 achieves autonomous driving and crop harvesting. During autonomous driving, the control system 160 controls the drive unit 140 based on the measured position and orientation of the agricultural machinery 100 and the target path. Thus, the control system 160 enables the agricultural machinery 100 to travel along the target path.

[0113] The control system 160 includes multiple ECUs that can communicate with each other using vehicle bus standards such as CAN (Controller Area Network). Alternatively, higher-speed communication methods such as in-vehicle Ethernet (registered trademark) can be used instead of CAN. Figure 3 In this diagram, ECUs 165, 166, and 167 are shown as independent modules, but their respective functions can also be implemented by multiple ECUs. An on-board computer that combines at least a portion of the functions of ECUs 165, 166, and 167 can also be provided. The control system 160 may also have ECUs other than ECUs 165, 166, and 167, and any number of ECUs can be configured according to their functions. Each ECU has processing circuitry including one or more processors.

[0114] exist Figure 5 In the example shown, camera 126 is used as Figure 1The imaging device 10 shown functions. ECU 167 acts as... Figure 1 The image processing device 20 shown functions. The combination of ECU165 and ECU166 acts as... Figure 1 The control device 30 shown functions.

[0115] Communication device 190 is a device that includes circuitry for communicating with external devices. Communication device 190 includes circuitry for wireless communication. Communication device 190 may include an antenna and communication circuitry for transmitting and receiving signals via a network with, for example, an external terminal device or an external server computer. The network may include, for example, cellular mobile communication networks such as 3G, 4G, or 5G, and the Internet. Communication device 190 may have the capability to communicate with a portable terminal used by monitoring personnel near agricultural machinery 100. Communication with such a portable terminal can be performed using any wireless communication standard such as Wi-Fi (registered trademark), 3G, 4G, or 5G, or Bluetooth (registered trademark).

[0116] The terminal monitor 131 is a terminal used by the user to perform operations related to the driving and operation of the agricultural machinery 100, also known as a virtual terminal (VT). The terminal monitor 131 may have a display device such as a touchscreen and / or one or more buttons. The display device may be a display such as a liquid crystal display (LCD) or an organic light-emitting diode (OLED). By operating the terminal monitor 131, the user can perform various operations such as switching the automatic driving mode on / off, recording or editing field map data, setting target paths, setting crop types, and setting operation types. At least some of these operations can also be achieved by operating the operation switch group 132. The terminal monitor 131 may also be configured to be detachable from the agricultural machinery 100. A user located away from the agricultural machinery 100 can operate the detached terminal monitor 131 to control the operation of the agricultural machinery 100. Alternatively, instead of the terminal monitor 131, the user can operate a computer with the necessary application software installed to control the operation of the agricultural machinery 100.

[0117] The operating switch assembly 132 includes a plurality of switches for operating the agricultural machinery 100. In this specification, "switch" broadly means a device used by the driver for operation, such as a lever, pedal, or button. The operating switch assembly 132 may include, for example, a switch for switching between automatic and manual driving modes, a switch for switching between forward and reverse, an accelerator pedal, a brake pedal, a gear shift lever, and a switch for switching lights on / off.

[0118] [2. Action]

[0119] Next, the operation of agricultural machinery 100 will be explained.

[0120] Figure 6 This diagram illustrates an example of the path taken by agricultural machinery 100 while harvesting crops in a field 70. In this embodiment, the agricultural machinery 100 harvests crops while driving automatically in the field 70. Within the field 70, the agricultural machinery 100 travels along a predetermined target path 74 and performs the harvesting action. An ECU 166 for automatic driving control performs steering control of the agricultural machinery 100 to eliminate deviations between the position and orientation of the agricultural machinery 100 determined based on data output from the GNSS unit 120 and the position and orientation of the target path 74. This enables the agricultural machinery 100 to travel along the target path 74.

[0121] exist Figure 6 In the example shown, field 70 includes a work area 71 for agricultural machinery 100 to harvest crops and a field edge 72 located near the outer perimeter of field 70. Map data (also called a "field map") indicating which area within field 70 corresponds to work area 71 and which area corresponds to field edge 72, and the target path 74, can be determined by ECU 166 for autonomous driving control. For example, when agricultural machinery 100 manually drives along path 73 at the outermost perimeter of work area 71 to harvest, ECU 166 generates a field map based on the trajectory of path 73 and generates a target path 74 for autonomous driving inside path 73. Thus, starting from the second week, agricultural machinery 100 can operate automatically (e.g., in an unmanned manner) via the autonomous driving control of ECU 166. Agricultural machinery 100 follows the path 74... Figure 6 The target path 74 shown is driven automatically. Furthermore, Figure 6 The target path 74 shown is merely an example, and the method for determining the target path 74 is arbitrary. Furthermore, the method for generating the field map and the target path 74 is not limited to the methods described above. For example, the field map and the target path 74 can also be set via the user operation terminal monitor 131.

[0122] In this embodiment, the agricultural machinery 100 performs obstacle detection using a camera 126 and a millimeter-wave radar 127 during operation. The camera 126 is primarily used to detect specific objects such as people. The millimeter-wave radar 127 is primarily used to detect metallic objects such as other vehicles. In this embodiment, image processing based on the images acquired by the camera 126 is performed by the ECU 167, enabling high-precision detection of people within the field 70 where crops are present. The control system 160 in this embodiment is configured to detect people and vehicles, but does not react to obstacles such as birds that have a low impact on operational movement.

[0123] Figure 7This is a schematic diagram illustrating the state of an object 76 (in this case, a person) being detected using a camera 126 mounted on an agricultural machine 100. Figure 7 The accompanying diagram shows F, B, R, and L representing front, back, right, and left, respectively. Figure 7 The dashed lines in the image represent examples of the area captured by camera 126. For example... Figure 7 As shown, when object 76 is present in the travel direction of agricultural machinery 100, ECU 167 detects object 76 from the image captured by camera 126 and sends a signal indicating the presence of object 76 to ECU 165. For example, ECU 167 can calculate the position of object 76 in a coordinate system fixed to the ground based on the position of object 76 in the image and the position and orientation of camera 126 in agricultural machinery 100. When the distance between this position and agricultural machinery 100 or camera 126 is less than a threshold, a signal indicating the presence of object 76 can be sent to ECU 165. Furthermore, distance determination can also utilize measurements from other ranging sensors such as laser sensor 125. However, in agricultural machinery 100, which can be used for harvesting crops (e.g., rice) at relatively high heights as in this embodiment, methods using other ranging sensors such as laser sensor 125 sometimes cannot accurately detect objects such as people located in the crops. Even in this case, the distance to object 76 can be more accurately estimated by using captured images.

[0124] When a signal indicating the presence of an object is received, the ECU 165 controls the drive unit 140 to stop the agricultural machinery 100 and activates the buzzer 133 to sound an alarm. Furthermore, if an obstacle such as a vehicle is detected within a predetermined distance of the agricultural machinery 100 based on a signal output from the millimeter-wave radar 127, the ECU 165 stops the agricultural machinery 100 and activates the buzzer 133 to sound an alarm. Through this control, collisions between the agricultural machinery 100 and obstacles can be avoided.

[0125] In this embodiment, ECU167 is equivalent to Figure 1 The image processing device 20 shown. That is, ECU 167 has Figure 2 The structure shown executes... Figure 3 The image processing device 20 can detect objects such as people 76 with high precision by appropriately extracting a portion of the image to be input into the learned model 27 from the image of the wide-field camera 126 according to the operating state of the agricultural machinery 100. The processing will be described in more detail below.

[0126] Figure 8 This is a schematic diagram illustrating the processing flow performed by ECU167 (i.e., the image processing unit). Figure 8 In the example shown, ECU 167 first acquires an image captured by camera 126 (step S801). Next, ECU 167 determines the region of interest within the captured image, which is then used for object detection processing (step S802). ECU 167 determines the region of interest by referring to various information representing the operational state of agricultural machinery 100. Figure 8 In the example shown, ECU 167 also refers to map data (field map) of the area including the field on which agricultural machinery 100 performs agricultural operations to determine the area of ​​interest. Information indicating the operational state of agricultural machinery 100 may include, for example, information indicating the steering angle or steering operation value measured by steering angle sensor 152, information indicating the speed measured by vehicle speed sensor 151, and information indicating the amount of tilt of agricultural machinery 100 measured by IMU 123 (i.e., tilt sensor). ECU 167 determines the location and size of the area of ​​interest in the captured image based on at least a portion of this information.

[0127] ECU167 cuts out a portion of the image corresponding to the determined region of interest from the captured image (step S803). ECU167 generates an input image by performing prescribed preprocessing on the portion of the image (e.g., image resizing, normalization, noise removal, etc.). ECU167 inputs the input image into the AI ​​model (i.e., the learned model), performs the processing to detect specific objects, and outputs a signal representing the detection result to ECU165.

[0128] The following is about Figure 8 The following are some specific examples of the processing shown.

[0129] Figure 9A This illustrates an example of images captured by camera 126. When the agricultural machinery 100 is operating, camera 126 repeatedly acquires images by taking pictures at a predetermined frame rate. Figure 9A The image shown is a photograph. Figure 9A The captured image shown contains an object 76 (in this example, a person). ECU 167 extracts a portion of the image representing the area of ​​interest determined based on the operational state of the agricultural machinery 100 from such captured images.

[0130] Figure 9B An example of a focus area 77 that can be selected when agricultural machinery turns right at 100 degrees is shown. Figure 9C It shows the relationship with Figure 9B The image corresponding to the area of ​​interest 77 shown. When the agricultural machinery turns right at 100, as... Figure 9B As shown, ECU167 identifies the right-hand region of the captured image as region of interest 77, as... Figure 9CAs shown, a portion of the image representing the region of interest 77 is extracted. ECU167 extracts this portion of the image by... Figure 9C The partial image shown undergoes preprocessing such as pixel reduction (size adjustment) to generate an input image that should be input into the trained model 27. ECU167 detects object 76 by inputting the generated input image into the trained model 27. In this object detection process, as described above, any object detection algorithm such as SSD, YOLO, or R-CNN can be used.

[0131] Figure 9D An example of a focus area 77 that can be selected when agricultural machinery turns left at 100 degrees is shown. Figure 9E It shows the relationship with Figure 9D The partial image corresponding to the region of interest 77 shown. Conversely, when the agricultural machinery 100 turns left, the ECU 167 extracts a partial image by using the left-hand region of the captured image as the region of interest 77.

[0132] Thus, in Figures 9B to 9E In the example shown, when the agricultural machinery 100 turns right, the ECU 167 moves the area of ​​interest 77 to the right; when the agricultural machinery 100 turns left, the ECU 167 moves the area of ​​interest 77 to the left. Through this processing, objects 76 that might affect the movement of the agricultural machinery 100 when turning right or left can be detected more accurately.

[0133] In existing technologies, by... Figure 9A The larger captured image (e.g., 1280×960 pixels) is preprocessed to generate a smaller input image (e.g., 300×300 pixels), which is then fed into the AI ​​model. In this case, the image sharpness is significantly reduced, which may lead to a decrease in the accuracy of object detection, especially at the far end of the image.

[0134] In contrast, in this embodiment, a portion of the image corresponding to the region of interest 77, which may contain objects that could affect the operation of the agricultural machinery 100, is first cropped from the captured image. This portion of the image is then preprocessed to generate an input image, which is then input into the learned model. The number of pixels in the input image is the same as in previous examples (e.g., 300×300 pixels), but because the area captured in the input image is narrowed, the decrease in image sharpness is suppressed. Therefore, the detection accuracy of the objects can be improved.

[0135] The number of pixels in the area that is cropped into a partial image depends on the operating state of the agricultural machinery 100, but can be, for example, less than 2 / 3, less than 1 / 2, less than 1 / 3, or less than 1 / 4 of the number of pixels in the original captured image.

[0136] When performing the above processing, the larger the steering angle of the agricultural machinery 100 when turning, the more the ECU 167 can move the area of ​​interest 77. In other words, the larger the steering angle when turning right, the more the area of ​​interest 77 moves to the right; the larger the steering angle when turning right, the more the area of ​​interest 77 moves to the left. The ECU 167 can obtain the steering angle information at that moment from the measurement value of the steering angle sensor 152. Data such as tables showing the relationship between the size of the steering angle when turning right or left and the amount of movement of the area of ​​interest 77 in the image can be pre-stored in the memory 24 or storage device 164. The ECU 167 can determine the amount of movement based on this data and the measured steering angle. Figures 9B to 9E In the example shown, the vertical dimension of the cropped portion of the image is the same as the vertical dimension of the original captured image, while the horizontal dimension of the cropped portion is smaller than the horizontal dimension of the captured image. This cropping method is not limited to this; a region smaller than the captured image can also be cropped vertically to create a cropped portion of the image.

[0137] The area of ​​interest 77 is not limited to the steering state of the agricultural machinery 100, but can also be determined based on other states. For example, the area of ​​interest 77 can be determined based on the travel speed of the agricultural machinery 100.

[0138] Figure 10A An example of a selectable area of ​​interest 77 is shown when the agricultural machinery 100 is moving at a high speed. Figure 10B It shows the relationship with Figure 10A The image shows a portion corresponding to the region of interest 77. In this example, the ECU 167 changes the size of the region of interest 77 according to the travel speed of the agricultural machinery 100. Specifically, the higher the travel speed of the agricultural machinery 100, the smaller the region of interest 77 is made by the ECU 167; the lower the travel speed, the larger the region of interest 77 is made by the ECU 167. When the agricultural machinery 100 is traveling at high speed, it is necessary to accurately detect objects 76 at a greater distance. Therefore, the higher the travel speed, the smaller the region of interest 77 is made by the ECU 167 to improve the detection accuracy of objects 76 at a greater distance. Not only does the size of the region of interest 77 change according to the travel speed of the agricultural machinery 100, but the position of the region of interest 77 can also change according to the travel speed. For example, the higher the travel speed, the higher the position of the region of interest 77 can be moved upwards. Through this processing, it becomes easier to detect objects at a greater distance when the agricultural machinery 100 is traveling at high speed. The ECU 167 obtains the travel speed information at that moment from the measurement value of the vehicle speed sensor 151. Data such as tables showing the relationship between driving speed and the size and / or position of the region of interest 77 within the image can be pre-stored in memory 24 or storage device 164. ECU 167 can determine the size and / or position of the region of interest 77 based on this data and the measured driving speed.

[0139] The area of ​​interest 77 can be determined based on the tilt amount of the agricultural machinery 100. The tilt amount of the agricultural machinery 100 refers to the magnitude of the tilt angle relative to a reference attitude, such as the pitch angle (i.e., the rotation angle centered on the axis in the left-right direction) or the roll angle (i.e., the rotation angle centered on the axis in the front-back direction). Here, the reference attitude refers to the attitude when the agricultural machinery 100 is on a level surface. The ECU 167 can move the area of ​​interest up or down based on the tilt amount, such as the pitch angle, measured by the IMU 123. When the agricultural machinery 100 is tilted upward relative to the horizontal plane, for example when the agricultural machinery 100 is traveling uphill, the pitch angle is positive; when the agricultural machinery 100 is tilted downward relative to the horizontal plane, for example when the agricultural machinery 100 is traveling downhill, the pitch angle is negative. When the agricultural machinery 100 is tilted upward (i.e., when the pitch angle is positive), the area of ​​interest in the captured image can be moved downward. Conversely, when the agricultural machinery 100 is tilted downward (i.e., when the pitch angle is negative), the area of ​​interest can be moved upward. Therefore, by avoiding including most of the sky or ground within the area of ​​interest, the detection performance of objects can be improved. The larger the absolute value of the pitch angle, the more the ECU 167 can increase the amount of movement of the area of ​​interest. In this case, data representing the relationship between the absolute value of the pitch angle and the amount of movement is stored in memory 24 or storage device 164. The ECU 167 can determine the amount of movement of the area of ​​interest based on this data and the measurement value of IMU 123.

[0140] ECU167 can further determine the area of ​​interest based on field map data. For example, ECU167 can determine the area of ​​interest based on map data, positioning data output from GNSS unit 120, and the operating status of agricultural machinery 100.

[0141] Figure 11 This diagram illustrates an example of agricultural machinery 100 traveling along the outermost perimeter of a work area 71 in a field. In this example, a person exists as an object 76 outside the work area 71. Since the agricultural machinery 100 does not travel outside the work area 71 during operation, such an object 76 does not need to be detected as an obstacle. Therefore, based on map data and positioning data, the ECU 167 can determine the area corresponding to the work area 71 where agricultural operations are performed in the field from the images acquired by the camera 126, and determine the area of ​​interest from that area based on the operating state of the agricultural machinery 100.

[0142] Here, an example of a method for determining the area corresponding to the work area 71 from the captured image will be explained.

[0143] Figure 12This is a three-dimensional diagram schematically representing the configuration relationships between the camera coordinate system Σc fixed to camera 126, the vehicle coordinate system Σv fixed to agricultural machinery 100, the world coordinate system Σw fixed to the ground, and the reference plane Re extending parallel to the horizontal plane. The camera coordinate system Σc has mutually orthogonal Xc, Yc, and Zc axes. The vehicle coordinate system Σv has mutually orthogonal Xv, Yv, and Zv axes. The world coordinate system Σw has mutually orthogonal Xw, Yw, and Zw axes. Figure 12 In this example, the Xw and Yw axes of the world coordinate system Σw lie on the reference plane Re. Camera 126 is fixed to the agricultural machinery 100. Therefore, the position and orientation of the camera coordinate system Σc relative to the vehicle coordinate system Σv are fixed in a known state. The camera coordinate system Σc is tilted such that its Zc axis intersects the reference plane Re at an angle. When the agricultural machinery 100 does not rotate in the pitch and roll directions, the plane including the Xv and Yv axes of the vehicle coordinate system Σv is parallel to the reference plane Re.

[0144] An imaginary image plane Im exists at a position, a distance from the origin O of the camera coordinate system Σc, along the Zc axis away from the focal length of the camera 126. Image plane Im is orthogonal to the Zc axis and the optical axis λ of the camera 126. Pixel positions on image plane Im are defined by an image coordinate system with mutually orthogonal u-axis and v-axis. For example, the coordinates of points P1 and P2 located on the reference plane Re are set as (X1, Y1, Z1) and (X2, Y2, Z2) respectively in the world coordinate system Σw. Figure 12 In this example, the Xw and Yw axes of the world coordinate system Σw lie on the reference plane Re. Therefore, Z1=Z2=0.

[0145] Points P1 and P2 on the reference plane Re are transformed into points P1 and P2 on the image plane Im of camera 126 through perspective projection of the pinhole camera model. On the image plane Im, points P1 and P2 are located at pixel positions represented by coordinates (u1, v1) and (u2, v2), respectively.

[0146] When the configuration of the camera coordinate system ∑c relative to the reference plane Re in the world coordinate system ∑w is given, the point (X, Y, 0) on the reference plane Re can be obtained from any point (u, v) on the image plane Im through homography transformation. This homography transformation, when representing the coordinates of the point in a homogeneous coordinate system, is defined by a 3x3 transformation matrix H. When the coordinates of a point on the image plane Im of camera 126 are (u, v, 0), the coordinates (X, Y, 0) of the corresponding point on the reference plane Re are shown in equation (1) below, and are established with the point (u, v, 0) through the homography transformation matrix H.

[0147] [Mathematical Expression 1]

[0148] The transformation matrix H depends on the configuration of the camera coordinate system Σc relative to the reference plane Re in the world coordinate system Σw. When the position of the reference plane Re changes, the content of the transformation matrix H also changes. The reference plane Re can be set to be flush with the ground or at a specified distance from the ground. If the object to be detected is a person, the reference plane Re can, for example, be set at a height of more than 1 meter but less than 2 meters from the ground.

[0149] By utilizing such homography transformation, it is possible to establish a correspondence between the coordinates of any point on the image plane Im of camera 126 and the coordinates of a point on the reference plane Re.

[0150] During the operation of the agricultural machinery 100, the ECU 167 can determine the position and attitude of the agricultural machinery 100 based on the positioning data output from the GNSS unit 120. Furthermore, the configuration relationship between the vehicle coordinate system Σv and the camera coordinate system Σc is known. Therefore, the ECU 167 can determine the configuration relationship between the world coordinate system Σw and the camera coordinate system Σc based on the positioning data, and can determine the transformation matrix H based on this configuration relationship.

[0151] Based on the result of the calculation in Equation (1) and the location information of the work area contained in the map data of the farmland, ECU167 can determine which pixel region in the captured image corresponds to the work area 71. ECU167 can be configured to exclude the pixel region in the captured image corresponding to the outer side of the work area 71 from the objects detected, and to determine the region of interest from the pixel region corresponding to the work area 71. This avoids the situation where the agricultural machinery 100 stops due to the detection of an object 76 that does not affect the operation of the agricultural machinery 100. Furthermore, instead of determining the region of interest from the pixel region corresponding to the work area 71 in the captured image, ECU167 can also determine the region of interest from the pixel region corresponding to the unoperated area in the work area 71. This is because even if an object 76 exists in an area where the work has been completed, since it does not hinder the operation of the agricultural machinery 100, the area where the work has been completed can be excluded from the detected objects.

[0152] (Implementation Method 2)

[0153] Next, an agricultural machine according to an exemplary second embodiment of this disclosure will be described.

[0154] Similar to the agricultural machinery of Embodiment 1, the agricultural machinery of this embodiment has a body, a shooting device, and an image processing device for detecting specific objects based on images acquired by the shooting device (image acquisition device). The image processing device performs the following steps (S21) to (S23).

[0155] (S21) Detect the boundary between the sky and ground objects outside the sky, as well as the objects in the image.

[0156] (S22) The tilt of the body is inferred based on the position of the boundary between the sky and the ground objects outside the sky in the image.

[0157] (S23) Based on the position of the object in the image and the inferred tilt of the machine, the position of the object in the coordinate system fixed to the ground is inferred.

[0158] Through this operation, the tilt of the machine can be estimated from the image without using a sensor that measures the tilt of the machine, and the position of the object can be estimated. The following is a detailed description of the agricultural machinery in this embodiment.

[0159] The agricultural machinery in this embodiment has the same characteristics as... Figure 1 or Figure 5 The structure is the same as that of the agricultural machinery 100 shown in the figure. In this embodiment, the image processing device 20 (e.g., Figure 5 The ECU167 shown also has Figure 2 The structure shown. When the image processing apparatus 20 in this embodiment detects a specific object based on the image acquired by the imaging apparatus 10, it estimates the position of the object in a coordinate system fixed to the ground (the aforementioned world coordinate system ∑w) and the distance from the agricultural machinery 100 to the object based on the position of the object in the image. (See reference...) Figure 12 As explained, this estimation process is performed by transforming the image coordinate system to the world coordinate system Σw. This coordinate transformation requires information about the configuration of the camera coordinate system Σc relative to the reference plane Re in the world coordinate system Σw. Therefore, it is necessary to utilize sensors capable of measuring the tilt of the agricultural machinery 100 (e.g., during the movement of the agricultural machinery 100) Figure 5 Using the IMU123 shown, the attitude of the shooting device 10 in the agricultural machinery 100 is estimated in turn, and the transformation matrix H in the above formula (1) is determined using the attitude information.

[0160] However, when using measurements from sensors such as the IMU123, there is a challenge in obtaining time synchronization between the sensor measurements and the images (i.e., individual frames in the video) generated by the imaging device 10. Sometimes, due to time lag, it is impossible to accurately determine the attitude of the agricultural machinery 100 corresponding to each frame.

[0161] To address this issue, the image processing apparatus 20 of this embodiment estimates the tilt (tilt angle) of the agricultural machinery 100 based on the captured images obtained by the imaging device 10. Specifically, it detects the boundary between the sky and ground objects outside the sky from the captured images, and estimates the tilt angle (e.g., pitch angle) of the agricultural machinery 100 based on the temporal variation of the position of this boundary. This allows for a more accurate determination of the attitude of the agricultural machinery 100 corresponding to each frame of the captured images, and improves the accuracy of estimating the position or distance of objects.

[0162] Furthermore, in this embodiment, the agricultural machinery 100 may also have a tilt sensor for measuring the tilt of the machine body (e.g., Figure 5 (IMU 123 shown). The image processing device 20 calculates the difference between the tilt of the agricultural machinery 100 estimated based on the position of the boundary between the sky and the ground objects outside the sky in the image and the tilt measured by the tilt sensor. When this difference is less than a threshold, the position of the object in the image and the estimated tilt can be estimated in the coordinate system fixed to the ground. In this case, when the difference between the tilt of the agricultural machinery 100 estimated based on the position of the boundary between the sky and the ground objects outside the sky in the image and the tilt measured by the tilt sensor is greater than or equal to the threshold, the image processing device 20 can be configured to reject the tilt estimation result of the former and not perform tilt-based processing. In this way, the reliability of the tilt estimated based on the image can also be evaluated using the measurement results of the tilt sensor.

[0163] The following is for reference Figure 13 A more specific example of the operation of the image processing apparatus 20 in this embodiment will be described.

[0164] Figure 13 This is a flowchart illustrating an example of image processing performed by the image processing apparatus 20 of this embodiment. During the operation of the agricultural machinery 100, the image processing apparatus 20 repeatedly performs the processing shown in steps S210 to S270.

[0165] In step S210, the image processing device 20 acquires the image generated by the imaging device 10.

[0166] In step S220, the image processing device 20 performs processing to detect the boundary between the sky and ground objects outside the sky, as well as specific objects (e.g., people) from the acquired image. The boundary between the sky and ground objects outside the sky can be, for example, the boundary between the sky and the ground (horizon) or the boundary between the sky and mountains. The boundary between the sky and ground objects can be detected, for example, through various methods such as image hue analysis, edge detection, and / or texture analysis. The processing to detect specific objects can, for example, be performed by... Figure 3The processing in steps S120 to S140 shown is performed in the same way. Alternatively, a learned model for detecting the boundary between the sky and ground objects and specific objects from the image can be used to detect both at once. Furthermore, in this embodiment, the partial image extraction processing in step S120 can be omitted. That is, the image processing apparatus 20 can also detect specific objects and / or the boundary between the sky and ground objects by inputting an input image generated from a pre-processed image of the captured image into the learned model.

[0167] Figure 14 This is a diagram illustrating an example of the boundary between the sky and ground objects detected from a captured image. In this example, the boundary 78 between the sky and the mountains is illustrated by a thick curve. The image processing device 20 can be configured to perform the process of detecting such a boundary 78 on each frame of the image (video) generated by the capturing device 10.

[0168] In the subsequent step S230, the image processing device 20 determines whether an object is detected in the image. If an object is detected, the process proceeds to step S240. If no object is detected, the process returns to step S210.

[0169] In step S240, the image processing device 20 estimates the tilt of the agricultural machinery 100 based on the position of the boundary 78 between the sky and the ground in the image. For example, the image processing device 20 estimates the tilt based on the amount of displacement between the position of the boundary 78 in the image and the position of the boundary 78 detected in an image acquired by the imaging device 10 at a past time point. The past time point can be a time point when the tilt measured by the tilt sensor is less than a reference value. The image processing device 20 estimates the tilt of the machinery based on the amount of displacement, such as the pitch angle of the machinery. More specifically, the image processing device 20 can estimate the tilt angle of the agricultural machinery 100 relative to a reference attitude based on the temporal variation of the position of the boundary 78 detected for each frame of the image generated by the imaging device 10. The reference attitude can be the attitude of the agricultural machinery 100 when it is on flat ground. The image processing device 20 can estimate the pitch angle of the agricultural machinery 100 based on the position of the boundary 78 in an image frame acquired when the agricultural machinery 100 is in a reference posture, and by the amount by which the boundary 78 is shifted up and down in the image from that position. For example, the image processing device 20 can estimate the pitch angle through the following process.

[0170] (1) Determine the corresponding points of the plurality of pixels (or a portion of the pixels in the central part, etc.) arranged along the boundary 78 in the reference frame obtained when the agricultural machinery 100 is in the reference posture.

[0171] (2) Calculate the displacement from the position of the reference frame for each corresponding point.

[0172] (3) Calculate the average value of the displacement of a complex number of corresponding points as the displacement of boundary 78.

[0173] (4) Calculate the pitch angle based on the displacement of boundary 78.

[0174] The correspondence between the vertical displacement of the boundary 78 within the image and the pitch angle can be pre-stored, for example, in... Figure 2 The memory 24 is shown. The image processing device 20 can estimate the pitch angle based on this correspondence. In addition, the image processing device 20 can also calculate the rotation angle of the boundary 78 between frames based on the time change of the position of the boundary 78 in the image, thereby estimating the tilt angle. By estimating the tilt angle in addition to the pitch angle, the attitude of the imaging device 10 can be estimated more accurately.

[0175] In step S250, the image processing device 20 estimates the position of the object in a coordinate system (world coordinate system) fixed to the ground based on the position of the object in the image and the estimated tilt of the machine body. Specifically, the image processing device 20 estimates the position of the object in the coordinate system (world coordinate system) based on the estimated tilt of the machine body, the known configuration of the imaging device 10 relative to the machine body, and the position of the object in the coordinate system (e.g., from the positioning device). Figure 5 The transformation matrix H in equation (1) is determined by the position and orientation of the aircraft obtained by the GNSS unit 120 in the image coordinate system. By performing the operation of equation (1) on the position of the object in the image coordinate system (e.g., the position of the bounding box), the position of the object in the world coordinate system can be calculated.

[0176] In step S260, the image processing device 20 estimates the distance from the agricultural machinery 100 to the object based on the object's position in the world coordinate system. The position of the agricultural machinery 100 in the world coordinate system is measured by a positioning device such as the GNSS unit 120. The image processing device 20 calculates the distance between the position of the agricultural machinery 100 and the position of the object in the world coordinate system. Here, the "position" of the agricultural machinery 100 is, for example, the predetermined position of a specific part of the agricultural machinery 100, such as the position of the GNSS unit 120.

[0177] In step S270, the image processing device 20 sends information representing the distance estimated in step S260 to the control device 30. Upon receiving this information, the control device 30 determines whether the distance is less than a predetermined value. If the distance is less than the predetermined value, it performs controls such as stopping the movement of the agricultural machinery 100 and emitting an alarm sound.

[0178] In this way, the image processing device 20 detects the boundary between the sky and ground objects outside the sky in the image acquired by the imaging device 10, and infers the tilt of the machine body based on the temporal change of the boundary position. Based on the inferred tilt, the position coordinates of the objects in the image are transformed into position coordinates in a coordinate system fixed to the ground, and the distance to the objects is inferred based on the transformed position coordinates. The control device 30 controls the operation of the agricultural machinery 100 based on the inferred distance.

[0179] in addition, Figure 13 The processing in step S260 shown can also be performed by the control device 30 instead of the image processing device 20. In this case, the image processing device 20 can be configured to send information representing the position coordinates of the object in the world coordinate system to the control device 30 after step S250. In this case, the control device 30 controls the operation of the agricultural machinery based on the position of the object in the world coordinate system. The control device 30 can be configured to perform at least one of stopping the agricultural machinery 100, slowing down the agricultural machinery 100, and issuing a warning when the distance estimated based on the position of the object in the world coordinate system is less than a predetermined value.

[0180] Figure 15 This is a flowchart illustrating a variation of this embodiment. Figure 15 In the example, it is not the control device 30 but the image processing device 20 that performs the determination of whether the distance from the agricultural machinery 100 to the object is less than a specified value. Figure 15 Examples and Figure 14 The difference in this example is that step S265 is added after step S260, and step S270 is replaced by step S275. In step S265, the image processing device 20 determines whether the distance from the agricultural machinery 100 to the object is less than a predetermined value. If the determination result is "yes", it proceeds to step S275. If the determination result is "no", it returns to step S210. In step S275, the image processing device 20 sends a signal indicating the presence of an object to the control device 30. Upon receiving this signal, the control device 30 performs controls such as stopping the movement of the agricultural machinery 100 and emitting an alarm sound.

[0181] Figure 16 This is a flowchart illustrating other variations of this embodiment. Figure 16 In the example shown, a tilt sensor (e.g., Figure 5 The tilt (e.g., pitch angle) measured by the IMU123 shown. Figure 16 Examples and Figure 14 The difference in the example is that step S245 is added after step S240, and if the result is negative in step S245, the process proceeds to step S255.

[0182] In step S245, the image processing device 20 determines whether the difference between the tilt of the machine estimated in step S240 and the tilt measured by the tilt sensor is less than a threshold. The tilt being compared can be, for example, a pitch angle or a roll angle. Alternatively, the value obtained by averaging the tilt measured by the tilt sensor over a predetermined time (e.g., approximately 0.1 seconds to several seconds) can be compared with the tilt of the machine estimated in step S240. If the difference in tilt is less than the threshold, the process proceeds to step S250, where further comparison is performed. Figure 14 The same processing applies to the example. If the difference in tilt between the two is above a threshold, proceed to step S255. A difference in tilt above a threshold means that the reliability of the tilt of the body estimated based on the image may be low. Therefore, in such a case, the image processing device 20 estimates the position of the object in the coordinate system fixed to the ground based on the tilt measured by the tilt sensor, rather than the tilt estimated from the image. After step S255, proceed to step S260.

[0183] In addition, it is also possible to Figure 16 In the example, step S270 is replaced with Figure 15 Steps S265 and S275 in the process. In this case, it is not the control device 30 but the image processing device 20 that performs the determination of whether the distance from the agricultural machinery 100 to the object is less than a predetermined value.

[0184] (Implementation Method 3)

[0185] Next, an agricultural machine according to an exemplary third embodiment of this disclosure will be described.

[0186] Similar to the agricultural machinery of Embodiment 1, the agricultural machinery of this embodiment includes an imaging device, an image processing device for detecting specific objects from images acquired by the imaging device, and a control device for controlling the movement of the agricultural machinery based on the detection results of the objects. The difference between this embodiment and Embodiment 1 is that the image processing device has a memory that stores a plurality of learned models corresponding to the environment surrounding the agricultural machinery, and switches between these models according to the environment. The image processing device performs the following steps (S31) to (S33).

[0187] (S31) Generate an input image based on the image acquired by the imaging device.

[0188] (S32) Select one learned model from a plurality of learned models based on the surrounding environment of the agricultural machinery.

[0189] (S33) Detect objects by inputting the input image into the selected learned model.

[0190] Through this process, an appropriate learning model corresponding to the surrounding environment of agricultural machinery can be used to detect objects from captured images with higher accuracy.

[0191] The environment surrounding agricultural machinery varies greatly. Changes in environmental conditions, such as time of day, weather, type of crop being harvested, and type of field, will alter the characteristics of the images captured by the imaging device (e.g., brightness, chroma, and hue of each pixel). Therefore, even with a single trained model, object detection performance may decrease depending on the environment. Maintaining high detection performance across various conditions requires training the model with a large amount of data; however, even with this, a single model may sometimes struggle to handle diverse environments.

[0192] Therefore, in this embodiment, the image processing apparatus is configured to prepare in advance a plurality of learned models corresponding to a plurality of environments, and to use these models differently depending on the situation. More specifically, the image processing apparatus generates an input image based on an image acquired by the imaging device, and inputs this input image into a learned model selected from the plurality of learned models according to the environment surrounding the agricultural machinery, thereby detecting objects. Through this operation, even if the environment surrounding the agricultural machinery changes, an appropriate learned model corresponding to the environment can be used to detect objects from the image with high accuracy.

[0193] The agricultural machinery in this embodiment has the same characteristics as... Figure 1 or Figure 5 The agricultural machinery 100 shown has the same structure. In this embodiment, the image processing device 20 (e.g., Figure 5 The ECU167 shown also has the same Figure 2 The example shown uses the same hardware structure. However, in this embodiment, the memory 24 stores a plurality of learned models, which is different from the example shown. Figure 2 The examples are different.

[0194] Figure 17This is a block diagram illustrating a structural example of the image processing apparatus 20 in this embodiment. In this example, the memory 24 of the image processing apparatus 20 stores a computer program 25 executed by the processor 22 and a plurality of learned models 27 for detecting objects from an input image. The plurality of learned models 27 are stored in association with various environmental conditions surrounding the agricultural machinery 100. For example, the plurality of learned models 27 associated with different brightness levels and / or different time periods surrounding the agricultural machinery 100 can be stored in the memory 24. The image processing apparatus 20 selects and utilizes one learned model from the plurality of learned models 27 that corresponds to the current ambient brightness or the current time period. The image processing apparatus 20 can use the image acquired by the imaging device 10 (e.g., a histogram of pixel values ​​or average brightness in the image), and the illuminance sensor provided on the agricultural machinery 100 (e.g., ... Figure 5 The output of the illuminance sensor 153 shown, and the input from the user (e.g., via...) Figure 5 The current ambient light level is determined by at least one of the crop type and / or work location type input into the terminal monitor 131 shown, and the illumination status of the lights on the agricultural machinery 100. The illuminance sensor can be mounted on the body of the agricultural machinery 100 (e.g., near the imaging device 10). Additionally, the image processing device 20 can obtain information about the current time period, for example, from a clock function of the processor 22 or a timing circuit such as a real-time clock that can be set separately from the processor 22.

[0195] exist Figure 17 In the example shown, the plurality of learned models 27 include daytime models 27A and 27B and a nighttime model 27C. In this case, the image processing device 20 selects either daytime model 27A or 27B during the daytime period and selects nighttime model 27C during the nighttime period. Figure 17 In the example, for daytime use, two models are prepared: a non-backlight model 27A and a backlight model 27B. In this case, the image processing device 20 can be configured to determine whether there is backlighting during daytime periods, for example, based on at least one of the image acquired by the shooting device 10, input from the user, and output from the illuminance sensor; if there is backlighting, select the backlight model 27B; if there is no backlighting, select the non-backlight model 27A.

[0196] Examples of models are not limited to Figure 17The example shown. For example, multiple learned models that establish correspondences with a plurality of different weather conditions, such as a sunny day model, a cloudy day model, and a rainy day model, can also be stored in memory 24. In this case, image processing device 20 selects and uses one learned model corresponding to the current weather from these multiple learned models. Image processing device 20 can use images acquired by imaging device 10 or from external devices (e.g., Figure 5 The current weather is determined by obtaining weather-related information from the communication device 190 (shown) and the server computer connected via a network. Alternatively, multiple learned models corresponding to different crop types can be prepared. In this case, the image processing device 20 selects a learned model corresponding to the crop type of the work object from the multiple learned models 27. The multiple models corresponding to the crop type may include, for example, models for rice, wheat, and soybean, which are various models corresponding to the crop type being harvested or other operations. In addition, multiple learned models corresponding to the type of field or work site, such as models for paddy fields, vegetable fields, and pasture, can also be prepared. Depending on the crop type and the type of work site, the tone and texture of the image acquired by the imaging device 10 may vary significantly. Therefore, by preparing multiple models according to the type of crop or work site, environmental adaptability can be improved. The image processing device 20 can determine the type of crop or work site based on the image acquired by the imaging device 10 or input from the user. Furthermore, different learned models can be pre-created for each of two or more combinations selected from factors such as ambient light, time of day, weather, crop type, and work site type, and these models can be switched according to environmental conditions. By switching between multiple models based on the environment, higher-precision object detection can be achieved.

[0197] Figure 18 This is a table representing an example of the correspondence between a plurality of trained models stored in memory 24 and their corresponding environments. In this example, a plurality of trained models A to L are stored in memory 24, differing based on a combination of three items: whether the time period is daytime or nighttime, whether the weather is sunny or cloudy, whether there is backlighting in sunny conditions, and the type of crop. Each model is pre-trained using a large amount of learning data corresponding to its respective environment and is stored in memory 24. The correspondence between models and environments is not limited to... Figure 18 The example shown can be modified in various ways. For instance, a model can be created that corresponds not only to day and night but also to evening. Alternatively, a model can be created to correspond to other weather conditions such as rain or snow. Figure 18In the example, the crop types are crop A, B, and C, but it can also be one type, multiple types, or more than four types. Furthermore, models can be created for each type of work site, such as paddy fields, vegetable fields, and pasture.

[0198] Each of the plurality of trained models 27 can be a machine learning model trained using a deep learning-based algorithm, such as a CNN or a visual converter. Each model shares a common model architecture but has distinct weightings. By determining the combination of weightings for each model using appropriate learning data and software, it is possible to create a plurality of trained models 27 adapted to various environments. The plurality of trained models 27 can be generated by the image processing device 20 itself or by an external device such as a server computer. The plurality of trained models 27 are stored in memory 24 before the operation of the agricultural machinery 100 begins or before the model is used.

[0199] Figure 19 This is a flowchart illustrating an example of the operation of the image processing apparatus 20 in this embodiment. Figure 19 In the example, the image processing device 20 repeatedly performs the actions of steps S310 to S340 during the operation of the agricultural machinery 100.

[0200] In step S310, the image processing device 20 generates an input image to be input into the learned model based on the captured image generated by the imaging device 10. The image processing device 20 generates the input image, for example, by performing prescribed preprocessing on the captured image (e.g., image resizing, normalization, noise removal, etc.). Alternatively, the image processing device 20 may also perform... Figure 3 The input image is generated through the processing steps S110 to S130 shown. By performing the processing steps S110 to S130, the area of ​​the object to be detected can be narrowed down according to the operating state of the agricultural machinery 100, thereby improving the detection accuracy of the object.

[0201] In step S320, the image processing device 20 selects one learned model corresponding to the surrounding environment of the agricultural machinery 100 from the plurality of learned models 27 stored in the memory 24. A specific example of the processing in step S320 will be described later.

[0202] In step S330, the image processing device 20 detects a specific object (e.g., a person) by inputting the input image into the selected, fully learned model. This step is related to... Figure 3The processing in step S140 is the same. The image processing device 20 detects objects such as people by inputting the input image into the learned model 27 stored in the memory 24. For example, the image processing device 20 may be configured to output the coordinate information of the position of the rectangular box (boundary box) representing the region where the object exists, as well as the width and height of the bounding box, as the detection result when a specific object exists in the image. Alternatively, the image processing device 20 may output a signal indicating whether a specific object exists in the input image as the detection result. The image processing device 20 may also determine that an object exists in the input image if the distance from the capturing device 10 to the object calculated based on the position of the bounding box in the input image is less than a threshold.

[0203] In step S340, the image processing device 20 sends the detection result to the control device 30. This step is related to... Figure 3 The processing in step S150 is the same. The image processing device 20 sends the detection result of the object to the control device 30. For example, the image processing device 20 may send a signal indicating whether a specific object exists in the input image to the control device 30. Alternatively, the image processing device 20 may send a signal indicating the presence of an object only if a specific object is detected from the input image. Furthermore, the image processing device 20 may send a signal indicating the presence of an object only if the distance to the detected object is less than a predetermined value. The distance to the object may be determined by, for example, using... Figure 15 or Figure 16 The method shown is used to estimate.

[0204] When the image processing device 20 receives a signal indicating a detection result, the control device 30 controls the operation of the agricultural machinery 100 based on the detection result. For example, the control device 30 may be configured to, upon receiving a signal indicating that an object has been detected, execute at least one of the following: stopping the agricultural machinery 100, slowing down the agricultural machinery 100, or outputting a warning to a device such as a buzzer or display. Through such actions, collisions between the agricultural machinery 100 and the object can be avoided, or the attention of the object (e.g., a person) or the rider of the agricultural machinery 100 can be drawn.

[0205] The processing in steps S310 to S340 can be performed by the imaging device 10 (e.g., Figure 5The frame rate of the camera 126 shown is periodically executed. Alternatively, the processing steps S310 to S340 can be performed every predetermined number of frames (e.g., 10 frames, 30 frames, 60 frames, etc.) or every predetermined time interval (e.g., 0.5 seconds, 1 second, 2 seconds, 3 seconds, etc.). The processing step S320 can also be performed before step S310. In addition, the processing step S320 can be performed at a lower frequency than the processing steps S310, S330, and S340. For example, the processing step S320 can be performed every predetermined time interval (e.g., every few seconds, every tens of seconds, every few minutes, etc.) or whenever the agricultural machinery 100 changes direction. The environment around the agricultural machinery 100 does not change in a short period of time, but it does change depending on whether the agricultural machinery 100 is facing the light or not. Therefore, it is not necessary to select the model at the same frequency as object detection; the model can also be selected at the moment of turning or other direction changes.

[0206] like Figure 6 As shown, the control device 30 of the agricultural machinery 100 (e.g., Figure 5 The ECU 166 shown can be configured to move agricultural machinery 100 along a set target path 74. The target path 74 is not limited to... Figure 6 The example shown could also be a path that includes a round trip section. The image processing device 20 can also select the learned model each time the agricultural machinery 100 changes direction, or each time the travel direction is reversed within the round trip section. The image processing device 20 can acquire a signal from the control device 30 indicating that the agricultural machinery 100 has changed direction. By performing the processing in step S320 in response to this signal, the image processing device 20 can select an appropriate model. In this case, the same model is used for object detection until the next turn.

[0207] Figure 20 This is a flowchart illustrating an example of the actions of the image processing device 20 when the agricultural machinery 100 selects a learned model each time it changes direction. Figure 20In the example, the image processing device 20 first selects a learned model corresponding to the surrounding environment of the agricultural machinery from a plurality of learned models in step S320. Then, it executes the processing steps S310, S330, and S340. After step S340, it proceeds to step S350, where the image processing device 20 determines whether the agricultural machinery 100 has changed direction. If it receives a signal from the control device 30 indicating that the agricultural machinery 100 has changed direction, the image processing device 20 determines that the agricultural machinery 100 has changed direction. If a direction change has occurred, it returns to step S320 to reselect a learned model. If no direction change has occurred, it returns to step S310 and uses the previously selected learned model to perform object detection processing based on the captured image again.

[0208] Next, refer to Figure 21 The model selection process in step S320 will be explained in more detail.

[0209] Figure 21 This is a flowchart illustrating an example of the model selection process in step S320. Step S320 in this example includes steps S321 to S326.

[0210] In step S321, the image processing device 20 acquires information related to the type of work site and the type of crop set by the user. This information can be obtained, for example, through user operation. Figure 5 The terminal monitor 131 and other input devices are used for setting.

[0211] In step S322, the image processing device 20 acquires information related to the date and time. For example, the image processing device 20 can acquire the information related to the date and time from the clock function of the processor 22 of the image processing device 20 or from a real-time clock that can be set separately from the processor 22.

[0212] In step S323, the image processing device 20 acquires information related to whether the lights of the agricultural machinery 100 are on. Information related to whether the lights are on can be acquired from the control device 30. For example, the user can obtain this information by... Figure 5 The operating switch group 132 shown includes a switch for lamp 142, which is operated to switch the lamp 142 on and off. A signal indicating whether lamp 142 is on can be received from control device 30 (e.g., ...). Figure 5 The ECU165 shown is sent to the image processing device 20.

[0213] In step S324, the image processing device 20 acquires information from an illumination sensor (e.g., ...). Figure 5The illuminance sensor 153 is shown as a measurement. The illuminance sensor's measurement indicates the brightness of the environment surrounding the agricultural machinery 100.

[0214] In step S325, the image processing apparatus 20 generates a histogram of pixel values ​​in the image. For example, when the pixel values ​​of each pixel are represented by 256 gray levels (8 bits) from 0 to 255 for each of the red, green, and blue colors, a histogram representing the frequency of each pixel value from 0 to 255 can be generated for each color. The image processing apparatus 20 can evaluate the overall brightness, hue, chroma, etc., of the image based on this histogram. In this step, instead of generating a histogram from the entire image, the image processing apparatus 20 can use, for example, the method described in Embodiment 2 to detect the region corresponding to the sky in the image and generate a histogram of the region corresponding to the sky.

[0215] In step S326, the image processing device 20 selects the learned model to be used based on the type of work site, the type of crop, the date and time, whether the lights are on or off, the measurement value of the illuminance sensor, and the histogram of the image. Based on this information or signal, the image processing device 20 can estimate whether the time period is daytime, evening, or nighttime, or whether the weather is sunny, cloudy, rainy, or snowy, and select a learned model corresponding to these estimation results and the selected work site and crop type.

[0216] Furthermore, the order of processing steps S321 to S325 can be interchanged. Alternatively, at least one of steps S321 to S325 can be omitted. In this case, the image processing apparatus 20 selects the learned model without considering the omitted information. The algorithm for estimating the type of work site, crop type, time period, weather, etc., based on acquired images and other information is not limited to a specific algorithm and can be arbitrarily designed. An AI model for determining the optimal model based on acquired images and other information can also be used.

[0217] As described above, according to this embodiment, object detection is performed using a learned model appropriately selected from a plurality of learned models based on the environment. Therefore, compared to using a single learned model, object detection performance can be significantly improved. Furthermore, even when each model is trained with relatively little training data, detection accuracy can be improved in various environments. For example, image recognition accuracy can be ensured regardless of day or night or weather conditions.

[0218] (Implementation Method 4)

[0219] Next, an agricultural machine according to an exemplary fourth embodiment of this disclosure will be described.

[0220] In this embodiment, the image processing device mounted on the agricultural machinery has the function of performing relearning (hereinafter also referred to as "optimization learning") of the learned model. Through relearning, the learned model can be improved in a way that is suitable for the actual use environment of the agricultural machinery, thereby improving the detection performance of objects.

[0221] The agricultural machinery of this embodiment is similar to that of the agricultural machinery in the aforementioned embodiments, including an imaging device, an image processing device for detecting specific objects (e.g., people) from images acquired by the imaging device, and a control device for controlling the actions of the agricultural machinery based on the detection results of the objects. The image processing device stores one or more learned models for detecting objects from input images. In this embodiment, the image processing device performs relearning of the learned models by executing the following steps (S41) to (S43).

[0222] (S41) Generate an input image based on the image acquired by the imaging device.

[0223] (S42) Detect objects by inputting the input image into the learned model.

[0224] (S43) Based on the image of the object detected in one or more of the multiple images acquired by the imaging device during the operation of the agricultural machinery, the learned model is relearned.

[0225] The above structure can improve the learned model to match the actual use environment of agricultural machinery and enhance the detection performance of objects.

[0226] The structure of the agricultural machinery in this embodiment and Figure 1 or Figure 5 The structure of the agricultural machinery 100 shown is the same. The structure of the image processing device is the same as that of the agricultural machinery 100 shown. Figure 2 or Figure 17 The image processing apparatus 20 shown has the same structure. In this embodiment, the image processing apparatus 20 differs from the aforementioned embodiments in that it has the function of improving the learned model 27 through relearning.

[0227] Figure 2 The learning completed model 27 or shown Figure 17 Each of the plurality of trained models 27 shown was created before the agricultural machinery 100 was put into use, by training with a large amount of training data suitable for that model. As mentioned earlier, the models may have a common model architecture and different weightings. By using appropriate training data for each model, model parameters such as the values ​​of the weightings are determined through learning, thereby enabling the creation of each trained model 27.

[0228] However, the actual operating environments of agricultural machinery 100 are diverse, and there are situations where it is difficult to build a model that can maintain high detection performance in various operating environments simply through pre-learning. Therefore, the image processing apparatus 20 in this embodiment is configured to perform relearning of the learned model by using images of actually detected objects from images acquired in the actual operating environment of the agricultural machinery 100. As a result, the model can be continuously improved, and the object detection performance can be enhanced.

[0229] In this embodiment, the image processing device 20, which is an edge computing device mounted on the agricultural machinery 100, is configured to perform relearning of the learned model. Therefore, optimal learning of the learned model suitable for the operating environment of the agricultural machinery 100 can be achieved without communicating with an external computer such as a cloud server.

[0230] The operation of the image processing apparatus 20 in this embodiment will now be described in more detail.

[0231] In this embodiment, the image processing device 20 determines whether the detection result of an object is correct for each of the plurality of images acquired by the imaging device 10 during the operation of the agricultural machinery 100, in which one or more images of an object are detected. Whether the object detection result is correct can be determined, for example, based on the time elapsed from when the agricultural machinery 100 stops moving until when it resumes movement after the object detection processing. This determination method is based on the consideration that, in the case where an object is detected from an image even though it does not actually exist (i.e., a false detection), the user of the agricultural machinery 100 should immediately resume movement after the agricultural machinery 100 has temporarily stopped.

[0232] If the image processing device 20 determines that the detection result of the object is correct, it uses the image for relearning the learned model. More specifically, the image processing device 20 does not use images that falsely detect objects even though the objects do not actually exist, but only uses images that correctly detect objects during relearning.

[0233] When the image processing device 20 detects an object in an image, it sends a signal indicating that an object has been detected to the control device 30. Upon receiving the signal indicating that an object has been detected, the control device 30 performs specific control, such as stopping the agricultural machinery 100. At this time, the detection result is correct when the object actually exists on or near the path of the agricultural machinery 100, and incorrect when the object does not actually exist. If the object actually exists, the user of the agricultural machinery 100 restarts the movement of the agricultural machinery 100 after the object has been moved to a position that does not obstruct its movement. On the other hand, if the object does not actually exist, the user immediately restarts the movement of the agricultural machinery 100 after confirming that the object does not exist. For example, the user... Figure 5 The operation switch group 132 or terminal monitor 131, etc., are operated to provide a restart command indicating that the movement should be restarted to the control device 30. Figure 5 In the example, this is ECU165. In response to the input restart command, the control device 30 restarts the movement of the agricultural machinery 100. The image processing device 20 can be configured to measure the time from when the agricultural machinery 100 stops to when it restarts movement, and determine whether the detection result of the object is correct based on this time. For example, it can be configured such that if the measured time is longer than a threshold, the detection result of the object is determined to be correct; if the measured time is less than the threshold, the detection result of the object is determined to be incorrect.

[0234] Figure 22 This is a flowchart illustrating a specific example of the operation of the image processing apparatus 20 in this embodiment. Figure 22 In the example, the image processing device 20 performs object detection processing based on dynamic images acquired during the movement of the agricultural machinery 100 and relearns the learned model by executing actions from steps S410 to S500. Figure 22 The action shown begins, for example, when the user provides an instruction via an input device to start the operation of the agricultural machinery 100.

[0235] In step S410, the image processing device 20 generates an input image to be input into the learned model based on the captured image generated by the imaging device 10. The image processing device 20 generates the input image, for example, by performing prescribed preprocessing on the captured image (e.g., image resizing, normalization, noise removal, etc.). Alternatively, the image processing device 20 may also perform... Figure 3 The input image is generated through the processing steps S110 to S130 shown. By performing the processing from steps S110 to S130, the area of ​​the object to be detected can be reduced according to the operating state of the agricultural machinery 100, thereby improving the detection accuracy of the object.

[0236] In step S420, the image processing device 20 performs a process of detecting objects by inputting an input image into a learned model stored in the memory 24. As in Embodiment 3, one learned model corresponding to the environment can be selected from a plurality of learned models and applied to the image, or as in Embodiments 1 and 2, the learned model to be used can be predetermined.

[0237] In step S430, the image processing device 20 determines whether an object is detected in the input image. If an object is detected, the process proceeds to step S440. If no object is detected, the process proceeds to step S490.

[0238] Figure 23 This is an example diagram illustrating the detection results of an object. In this example, the image processing device 20 detects a specific object from an input image by executing object detection software. As the detection results, it can output, for example, a label indicating the type of object (e.g., "Person" if it is a person), the coordinates of the representative point of the bounding box indicating the object's position within the image (e.g., the top-left vertex or center point), and a numerical value indicating the reliability of the detection results.

[0239] In step S430, the image processing device 20 can determine whether an object has been detected based on the distance between the agricultural machinery 100 and the object. For example, it can also calculate the position of the object in a coordinate system fixed to the ground based on the position of the object in the input image and the position and orientation of the imaging device 10 in the agricultural machinery 100, and determine that an object has been detected if the distance between that position and the agricultural machinery 100 is less than a threshold. Here, it can be done by... Figure 15 or Figure 16 The method shown is for estimating distance. Furthermore, it can also be used in distance determination. Figure 5 The measured values ​​of the laser sensor 125 and other distance measuring sensors are shown.

[0240] In step S440, the image processing device 20 sends a signal indicating that an object has been detected to the control device 30. When the control device 30 receives the signal, it stops the movement of the agricultural machinery 100 and sends a signal indicating that the movement has stopped to the image processing device 20.

[0241] In step S450, the image processing device 20 receives a signal from the control device 30 indicating that the movement of the agricultural machinery 100 has stopped. Upon receiving this signal, the image processing device 20 remains in standby mode until the movement of the agricultural machinery 100 resumes.

[0242] As described above, after the agricultural machinery 100 stops moving, the user of the agricultural machinery 100 confirms whether the object actually exists in the direction of travel of the agricultural machinery 100. If the object does not actually exist (i.e., in the case of false detection), the user restarts the movement of the agricultural machinery 100 within a relatively short time. On the other hand, if the object actually exists, the user restarts the movement of the agricultural machinery 100 after confirming that the object no longer exists on the predetermined path of the agricultural machinery 100. When the movement of the agricultural machinery 100 restarts, the control device 30 sends a signal indicating that the movement has restarted to the image processing device 20.

[0243] In step S460, the image processing device 20 receives a signal from the control device 30 indicating the restart of the movement of the agricultural machinery.

[0244] In step S470, the image processing device 20 compares the time from when the agricultural machinery stops moving until it resumes moving (hereinafter referred to as "standby time") with a predetermined threshold. If the standby time is longer than the threshold, the process proceeds to step S480. If the standby time is less than the threshold, it is determined that an object has been detected incorrectly, and the process proceeds to step S490. If the standby time is longer than the threshold, it is processed as if an object has been detected normally. If the standby time is less than the threshold, it is processed as if an object has been detected incorrectly.

[0245] In step S480, the image processing device 20 records the input image as a relearning image in the memory 24. At this time, information indicating the detection result may also be recorded in association with the input image. The information indicating the detection result may include a label indicating the type of object and coordinate values ​​(e.g., coordinate values ​​of a bounding box) indicating the location of the object within the input image.

[0246] In step S490, the image processing device 20 determines whether it has received a signal indicating that the agricultural operation performed by the agricultural machinery 100 has been completed (hereinafter referred to as the "operation completion signal"). The operation completion signal can be sent from the control device 30 to the image processing device 20 when the agricultural operation is completed. "When the agricultural operation is completed" can be, for example, based on the target path of the agricultural machinery 100 along the field (e.g., Figure 6 When the operation of the target path 74 shown is completed, or when the harvesting operation is completed after the operation is completed, the image processing device 20 returns to step S410 if it does not receive an operation completion signal. The image processing device 20 repeats steps S410 to S490 until an operation completion signal is received. If the image processing device 20 receives an operation completion signal, it proceeds to step S500.

[0247] In step S500, the image processing device 20 performs model relearning based on the input image set recorded in the memory 24 as relearning images, creates a new learned model, and records it in the memory 24. For example, the image processing device 20 uses data including each input image recorded as relearning images, the position information of objects in each input image, and information of labels indicating the types of objects in each input image as learning data, executes relearning software, and thereby creates a new learned model.

[0248] As described above, during agricultural operations performed by the agricultural machinery 100, the image processing device 20 in this embodiment repeatedly performs the following actions: generating an input image based on an image acquired by the imaging device 10; detecting objects by inputting the input image into a learned model; and recording an image when an object is detected. After receiving a signal indicating the end of agricultural operations, the image processing device 20 performs relearning of the learned model based on one or more images of detected objects. Through this processing, for example, the model can be relearned whenever agricultural operations in a field are completed, thereby improving the model.

[0249] Furthermore, the relearning of the model in step S500 may not occur at the moment when the agricultural machinery 100 completes the agricultural operation. For example, the model relearning in step S500 may also occur when switching from automatic driving mode to manual driving mode, or when the power to the agricultural machinery 100 is disconnected.

[0250] Furthermore, as a method for determining whether the object detection result is correct, a method different from the method based on the time from when the agricultural machinery 100 stops moving until it resumes moving after the object detection processing can be used. For example, the image processing device 20 can base its judgment on the time from the object detection sensor mounted on the agricultural machinery 100 (e.g., Figure 5 The output of the laser sensor 125 and / or millimeter-wave radar 127 in the example is used to determine whether the object detected from the image actually exists. In the case where a person is present among crops to be harvested, the object detection sensor, such as the laser sensor or millimeter-wave radar, may fail to distinguish between the person and the crops, but may be able to detect the presence of certain objects. Therefore, by combining image-based object detection with object detection performed by the object detection sensor, it is possible to determine whether the image-based object detection result is correct.

[0251] exist Figure 22In the example, if the time from the stopping of the agricultural machinery 100 to its restart is longer than a threshold (i.e., if the detection result of the object based on the image is determined to be correct), the image processing device 20 stores the input image in the memory 24. Not limited to this processing, the image processing device 20 may also append information indicating whether the detection result of the object is correct to the input image and store it in the memory 24. For example, if the detection result of the object based on the input image is determined to be incorrect, the information indicating the incorrect detection result may be stored in the memory 24 in association with the input image. Alternatively, if the detection result of the object based on the input image is determined to be correct, the image processing device 20 may also store the information indicating the correct detection result in association with the input image in the memory 24. Such information can be stored as metadata associated with the input image. The image processing device 20 can determine the image to be used in the relearning of the learned model based on this information. For example, the image processing device 20 may also use the image associated with the information indicating the correct detection result instead of the image associated with the information indicating the incorrect detection result for relearning.

[0252] In step S480, the image processing device 20 may also record the position information of the object in each of more than one image in which the object is detected, corresponding to that image. The position information may be, for example, the coordinates of a representative point (e.g., the upper left vertex or center point) of the bounding box representing the position of the object in the image. By attaching such position information to the image and recording it, relearning of the learned model can be performed efficiently. For example, the image processing device 20 may also store the position information of the object in the image and a label indicating the type of the object in the memory 24. In this way, the position information of the object in the image and the label indicating the type of the object can be used as annotation data, making the relearning of the learned model more efficient.

[0253] After relearning in step S500, the image processing apparatus 20 can update the existing learned model using the relearned model, or it can use both simultaneously. When both are used simultaneously, the image processing apparatus 20 can also select which model—the existing learned model or the relearned model—to use for object detection processing based on user input. Alternatively, the image processing apparatus 20 can update the existing learned model using the relearned model according to user instructions. User instructions can be, for example, by using... Figure 5The operation of input devices such as the terminal monitor 131 shown is provided. With this structure, the model can be updated after the user confirms that the performance of the relearned model is better than the existing relearned model, or it can revert to the original relearned model if the effect of relearning cannot be confirmed.

[0254] In the above embodiments 1 to 4, examples were mainly described where the operating machinery is agricultural machinery such as harvesters. However, the aforementioned technologies can also be applied to operating machinery other than agricultural machinery. For example, it is also possible to... Figure 24 The construction work vehicle 200 illustrated above may be equipped with some or all of the functions described in embodiments 1 to 4. Furthermore, the image acquisition device is not limited to the imaging device 10 such as the camera 126, but may also be a device capable of acquiring point cloud images, such as a LiDAR sensor.

[0255] As described above, this disclosure includes the following agricultural machinery, image processing apparatus, and image processing method.

[0256] [Project A1]

[0257] A type of work machinery that performs work while moving, wherein it has: An image acquisition device acquires images along the direction of movement of the operating machinery; An image processing apparatus detects a specific object from the image; and The control device controls the operation of the working machinery based on the detection results of the object. The image processing apparatus has a memory for storing a learned model for detecting the object from the input image. The image processing device extracts a portion of the image representing the region of interest determined based on the operating state of the working machinery from the image acquired by the image acquisition device, generates the input image based on the portion of the image, and detects the object by inputting the input image into the learned model.

[0258] [Project A2]

[0259] According to the operating machinery described in Project A1, among which... The image processing device acquires information related to at least one of the moving speed of the working machinery, the turning state of the working machinery, and the tilting state of the working machinery, and changes the region of interest based on the information.

[0260] [Project A3]

[0261] According to the operating machinery described in Project A2, among which... The image processing device adjusts the size of the region of interest according to the moving speed of the machinery.

[0262] [Project A4]

[0263] According to the operating machinery described in Project A3, among which... The higher the moving speed of the operating machinery, the smaller the area of ​​interest is made by the image processing device.

[0264] [Project A5]

[0265] According to the operating machinery described in Project A2, among which... The image processing device moves the area of ​​interest to the right when the machine turns right, and moves the area of ​​interest to the left when the machine turns left.

[0266] [Project A6]

[0267] According to the operating machinery described in Project A5, among which... The greater the turning angle of the working machinery when it turns, the greater the area of ​​interest that the image processing device can move.

[0268] [Project A7]

[0269] According to the operating machinery described in Project A2, among which... It also includes a tilt sensor for measuring the tilt amount of the operating machinery. The image processing device changes the position of the region of interest according to the tilt amount.

[0270] [Project A8]

[0271] According to the operating machinery described in Project A7, among which... The tilt sensor measures the pitch angle of the operating machinery as the tilt amount. The image processing device moves the region of interest up or down according to the pitch angle.

[0272] [Project A9]

[0273] The operating machinery according to any one of items A1 to A8, wherein it further comprises: Storage device, storing map data of an area including the field where the machinery is operating; and The positioning device acquires the positioning data of the operating machinery. The image processing device determines the area of ​​interest based on the map data, the positioning data, and the operating status of the machinery.

[0274] [Project A10]

[0275] According to the operating machinery described in Project A9, among which... The image processing device, based on the map data and the positioning data, determines an area from the image acquired by the image acquisition device that corresponds to the work area in the field where the operation is performed, and determines the area of ​​interest from the area based on the operating state of the work machinery.

[0276] [Project A11]

[0277] The operating machinery according to any one of items A1 to A10, wherein, The image processing apparatus generates the input image by a process including compressing the number of pixels in the partial image to a preset number of pixels.

[0278] [Project A12]

[0279] The operating machinery according to any one of items A1 to A11, wherein, The operating machinery refers to autonomous vehicles or unmanned aerial vehicles. The control device controls the automatic driving operation of the machine.

[0280] [Project A13]

[0281] The operating machinery according to any one of items A1 to A12, wherein, When the object is detected, the control device performs at least one of stopping the working machinery, slowing down the working machinery, and outputting a warning.

[0282] [Project A14]

[0283] An image processing apparatus for detecting a specific object from an image acquired by an image acquisition device mounted on a work machine, wherein the apparatus comprises: A memory, storing a learned model for detecting the objects from an input image; and Operational circuits The computing circuit extracts a portion of the image representing the region of interest determined based on the operating state of the machinery from the image acquired by the image acquisition device, generates the input image based on the portion of the image, and detects the object by inputting the input image into the learned model.

[0284] [Project A15]

[0285] A method performed by an image processing device for detecting a specific object from an image acquired by an image acquisition device mounted on a work machine, comprising: The image is acquired from the image acquisition device; Obtain information indicating the operational status of the operating machinery; Extract a portion of the image representing the region of interest determined based on the operational state of the machinery. Generate an input image based on the aforementioned partial image; and The object is detected by feeding the input image into a pre-generated, fully learned model.

[0286] [Project B1]

[0287] A type of work machinery, wherein: Organism; Image acquisition device; and An image processing device detects a specific object from an image acquired by the image acquisition device. The image processing device detects the boundary between the sky and ground objects outside the sky, as well as the object, in the image. Based on the position of the boundary in the image, it infers the tilt of the machine body. Based on the position of the object in the image and the inferred tilt, it infers the position of the object in a coordinate system fixed to the ground.

[0288] [Project B2]

[0289] According to the operating machinery described in Project B1, among which... The image processing device estimates the distance from the working machinery to the object based on the position of the object in the coordinate system fixed to the ground.

[0290] [Project B3]

[0291] According to the operating machinery described in Project B1, among which... It also includes a tilt sensor for measuring the tilt of the machine body. The image processing device calculates the difference between the tilt estimated based on the position of the boundary in the image and the tilt measured by the tilt sensor. If the difference is less than a threshold, the image processing device estimates the position of the object in the coordinate system fixed to the ground based on the position of the object in the image and the estimated tilt.

[0292] [Project B4]

[0293] The operating machinery according to any one of items B1 to B3, wherein... The image processing device estimates the tilt of the machine body based on the displacement between the position of the boundary in the image and the position of the boundary detected from an image acquired by the image acquisition device at a past time point.

[0294] [Project B5]

[0295] According to the operating machinery described in Project B4, among which... The past time point is the time point when the tilt is less than the reference value as measured by the tilt sensor.

[0296] [Project B6]

[0297] According to the operating machinery described in item B4 or B5, among which... The image processing device estimates the tilt angle of the machine body based on the displacement.

[0298] [Project B7]

[0299] The operating machinery according to any one of items B1 to B6, wherein, The image processing device converts the position coordinates of the object in the image into position coordinates in the coordinate system fixed to the ground based on the estimated tilt, and estimates the distance from the object based on the converted position coordinates.

[0300] [Project B8]

[0301] The operating machinery according to any one of items B1 to B7, wherein... It also has a control device that controls the operation of the working machinery based on the position of the object in the coordinate system fixed to the ground.

[0302] [Project B9]

[0303] According to the operating machinery described in Project B8, among which... If the distance estimated based on the position of the object in the coordinate system is smaller than a predetermined value, the control device performs at least one of stopping the working machinery, slowing down the working machinery, and issuing a warning.

[0304] [Project B10]

[0305] According to the operating machinery described in item B8 or B9, among which... The operating machinery refers to autonomous vehicles or unmanned aerial vehicles. The control device controls the automatic driving operation of the machine.

[0306] [Project B11]

[0307] An image processing apparatus detects a specific object from an image acquired by an image acquisition device mounted on a work machine, wherein... Detect the boundary between the sky and ground objects outside the sky, as well as the object itself, from the image. The tilt of the machine is inferred based on the position of the boundary in the image; Based on the position of the object in the image and the estimated tilt, the position of the object in a coordinate system fixed to the ground is estimated.

[0308] [Project B12]

[0309] A method performed by an image processing device for detecting a specific object from an image acquired by an image acquisition device mounted on a work machine, comprising: Detect the boundary between the sky and ground objects outside the sky, as well as the object itself, from the image; The tilt of the machine is inferred based on the position of the boundary in the image; and Based on the position of the object in the image and the inferred tilt, the position of the object in a coordinate system fixed to the ground is inferred.

[0310] [Project C1]

[0311] A type of work machinery that performs work while moving, wherein it has: An image acquisition device acquires images along the direction of movement of the operating machinery; An image processing apparatus detects a specific object from the image; and The control device controls the operation of the working machinery based on the detection results of the object. The image processing apparatus has a memory that stores a plurality of learned models for detecting the object from the input image. The image processing device generates the input image based on the image acquired by the image acquisition device, and detects the object by inputting the input image into a learned model selected from the plurality of learned models according to the environment surrounding the working machinery.

[0312] [Project C2]

[0313] According to the operating machinery described in Project C1, among which... The plurality of learned models are associated with a plurality of different brightness levels in the surrounding environment of the operating machinery. The image processing device selects one of the learned models from the plurality of learned models that corresponds to the brightness of the current environment.

[0314] [Project C3]

[0315] According to the operating machinery described in Project C2, among which... The image processing device determines the brightness of the environment based on at least one of the image, the output from the illuminance sensor, the input from the user, and the lighting status of the lights.

[0316] [Project C4]

[0317] According to the operating machinery described in Project C1, among which... The plurality of learned models are associated with a plurality of different time periods. The image processing device selects one of the learned models corresponding to the current time period from the plurality of learned models.

[0318] [Project C5]

[0319] According to the operating machinery described in Project C4, among which... The plurality of learned models includes daytime models and nighttime models. The image processing device selects the daytime model during the daytime period and the nighttime model during the nighttime period.

[0320] [Project C6]

[0321] According to the operating machinery described in Project C1, among which... The plurality of learned models include models for backlighting and models for non-backlighting. The image processing device determines whether it is backlighting based on at least one of the image, input from the user, and output from the illuminance sensor. If it is backlighting, it selects the backlighting model; if it is not backlighting, it selects the non-backlighting model.

[0322] [Project C7]

[0323] According to the operating machinery described in Project C1, among which... The plurality of learned models are associated with a plurality of different crop types. The image processing device selects one learned model from the plurality of learned models that corresponds to the type of crop of the target crop.

[0324] [Project C8]

[0325] According to the operating machinery described in Project C7, among which... The image processing device determines the type of crop for the task based on the image or input from the user.

[0326] [Project C9]

[0327] According to the operating machinery described in Project C1, among which... The plurality of learned models are associated with a plurality of different weather conditions. The image processing device selects one of the plurality of learned models that corresponds to the current weather.

[0328] [Project C10]

[0329] According to the operating machinery described in Project C9, among which... The image processing device determines the current weather based on the image or weather-related information obtained from an external device.

[0330] [Project C11]

[0331] The operating machinery according to any one of items C1 to C10, wherein... The plurality of learned models each have a common model architecture and different weightings.

[0332] [Project C12]

[0333] The operating machinery according to any one of items C1 to 11, wherein, The control device causes the operating machinery to move along a predetermined path. Whenever the working machine changes direction on the path, the image processing device performs the selection of the learned model.

[0334] [Project C13]

[0335] The operating machinery according to any one of items C1 to 12, wherein, The image processing device performs the selection of the learned model at regular intervals during the movement of the working machinery.

[0336] [Project C14]

[0337] The operating machinery according to any one of items C1 to 13, wherein... When the object is detected, the control device performs at least one of stopping the working machinery, slowing down the working machinery, and outputting a warning.

[0338] [Project C15]

[0339] An image processing apparatus for detecting a specific object from an image acquired by an image acquisition device mounted on a work machine, wherein the apparatus comprises: Memory, storing a plurality of learned models for detecting the objects from the input image; and Operational circuits The computing circuit generates the input image based on the image acquired by the image acquisition device, and detects the object by inputting the input image into a learned model selected from the plurality of learned models according to the environment around the working machinery.

[0340] [Project C16]

[0341] A method performed by an image processing device for detecting a specific object from an image acquired by an image acquisition device mounted on a work machine, comprising: The input image is generated based on the image acquired by the image acquisition device; and The object is detected by inputting the input image into a learned model selected from a plurality of learned models based on the environment surrounding the working machinery.

[0342] [Project D1]

[0343] A type of work machinery that performs work while moving, wherein it has: An image acquisition device acquires images along the direction of movement of the operating machinery; An image processing apparatus detects a specific object from the image; and The control device controls the operation of the working machinery based on the detection results of the object. The image processing apparatus has a memory for storing a learned model for detecting the object from the input image. The image processing device generates the input image based on the image acquired by the image acquisition device, detects the object by inputting the input image into the learned model, and performs relearning of the learned model based on one or more images of the object detected from a plurality of images acquired by the image acquisition device during the operation of the working machinery.

[0344] [Project D2]

[0345] According to the operating machinery described in Project D1, among which... The image processing device, for each of the more than one images in which the object is detected, determines whether the detection result of the object is correct based on the action of the operating machinery after the object detection. If the detection result of the object is determined to be correct, the image processing device uses the image for the relearning of the learned model.

[0346] [Project D3]

[0347] According to the operating machinery described in Project D2, among which, When the control device detects the object, it stops the working machinery; in response to an input restart command, it restarts the working machinery. The image processing device determines whether the detection result of the object is correct based on the time from when the working machinery stops until it resumes movement.

[0348] [Project D4]

[0349] According to the operating machinery described in Project D1, among which... It also has an object detection sensor. The image processing device, for each of the more than one images in which the object is detected, determines whether the detection result of the object is correct based on the output from the object detection sensor. If the detection result of the object is determined to be correct, the image processing device uses the image for the relearning of the learned model.

[0350] [Project D5]

[0351] The operating machinery according to any one of items D1 to D4, wherein... If the image processing device determines that the detection result of the object is incorrect, it records the information indicating that the detection result is incorrect in association with the image. Based on the information, images are determined for relearning using the learned model.

[0352] [Project D6]

[0353] The operating machinery according to any one of items D1 to D4, wherein... The image processing apparatus includes, when determining that the detection result for the object is correct, recording information indicating that the detection result is correct in association with the image. Based on the information, images are determined for relearning using the learned model.

[0354] [Project D7]

[0355] The operating machinery according to any one of items D1 to D6, wherein... The image processing device records the position information of the object in each of the more than one images that detect the object, establishing a correspondence between the images and the images. The relearning of the learned model is performed based on the location information.

[0356] [Project D8]

[0357] The operating machinery according to any one of items D1 to D7, wherein... After performing the relearning, the image processing device updates the learned model using the relearned model according to instructions from the user.

[0358] [Project D9]

[0359] The operating machinery according to any one of items D1 to D8, wherein... During the operation performed by the machinery, the image processing device repeatedly performs the following actions: generating the input image based on the image acquired by the image acquisition device; detecting the object by inputting the input image into the learned model; and recording the input image when the object is detected. After receiving a signal indicating the end of the task, the learned model is relearned based on the detection of one or more input images of the object.

[0360] [Project D10]

[0361] An image processing apparatus for detecting a specific object from an image acquired by an image acquisition device mounted on a work machine, wherein the apparatus comprises: A memory, storing a learned model for detecting the objects from an input image; and Operational circuits The computing circuit generates the input image based on the image acquired by the image acquisition device, detects the object by inputting the input image into the learned model, and performs relearning of the learned model based on one or more images of the object detected from a plurality of images acquired by the image acquisition device during the operation of the working machinery.

[0362] [Project D11]

[0363] A method for detecting a specific object from an image acquired by an image acquisition device mounted on a work machine, comprising: The input image is generated based on the image acquired by the image acquisition device; The object is detected by inputting the input image into the learned model; and Based on one or more images of the object detected from a plurality of images acquired by the image acquisition device during the operation of the machinery, the learned model is relearned.

[0364] [Industry availability]

[0365] The technology disclosed herein can be applied to various types of operational machinery, such as agricultural machinery like harvesters, tractors, transplanters, and agricultural drones; construction machinery like backhoe excavators, wheel loaders, and transporters; and snowplows.

[0366] Explanation of reference numerals in the attached figures

[0367] 10: Imaging device; 20: Image processing device; 22: Processor; 24: Memory; 25: Computer program; 27: Completed learning model; 30: Control device; 100: Agricultural machinery; 101: Machine body; 102: Traveling device; 103: Harvesting device; 104: Transporting device; 105: Threshing device; 106: Tank; 107: Discharge device; 108: Straw discharge processing device; 109: Reel; 110: Cab; 111: Prime mover; 112: Transmission device; 117: Discharge outlet; 120: GNSS unit; 121: GNSS receiver; 122: RTK receiver, 123: Inertial Measurement Unit (IMU), 124: Processing circuit, 125: LiDAR sensor, 126: Camera, 127: Obstacle sensor, 131: Terminal monitor, 132: Operating switch group, 133: Buzzer, 140: Drive unit, 141: Power transmission mechanism, 142: Light, 150: Sensor group, 151: Vehicle speed sensor, 152: Steering angle sensor, 153: Illuminance sensor, 160: Control system, 164: Storage device, 165-167: ECU, 190: Communication device, 200: Construction vehicle.

Claims

1. A type of work machinery that performs work while moving, wherein, have: An image acquisition device acquires images along the direction of movement of the operating machinery; An image processing device that detects specific objects from the image; as well as The control device controls the operation of the working machinery based on the detection results of the object. The image processing apparatus has a memory for storing a learned model for detecting the object from the input image. The image processing device generates the input image based on the image acquired by the image acquisition device, detects the object by inputting the input image into the learned model, and performs relearning of the learned model based on one or more images of the object detected from a plurality of images acquired by the image acquisition device during the operation of the working machinery.

2. The operating machinery according to claim 1, wherein, The image processing device, for each of the more than one images in which the object is detected, determines whether the detection result of the object is correct based on the action of the operating machinery after the object detection. If the detection result of the object is determined to be correct, the image processing device uses the image for the relearning of the learned model.

3. The operating machinery according to claim 2, wherein, When the control device detects the object, it stops the working machinery; in response to an input restart command, it restarts the working machinery. The image processing device determines whether the detection result of the object is correct based on the time from when the working machinery stops until it resumes movement.

4. The operating machinery according to claim 1, wherein, It also has an object detection sensor. The image processing device, for each of the more than one images in which the object is detected, determines whether the detection result of the object is correct based on the output from the object detection sensor. If the detection result of the object is determined to be correct, the image processing device uses the image for the relearning of the learned model.

5. The operating machinery according to any one of claims 1 to 4, wherein, If the image processing device determines that the detection result of the object is incorrect, it records the information indicating that the detection result is incorrect in association with the image. The image processing device determines images for relearning the learned model based on the information.

6. The operating machinery according to any one of claims 1 to 4, wherein, The image processing apparatus includes, when determining that the detection result for the object is correct, recording information indicating that the detection result is correct in association with the image. The image processing device determines images for relearning the learned model based on the information.

7. The operating machinery according to any one of claims 1 to 4, wherein, The image processing device records the position information of the object in each of the more than one images that detect the object, establishing a correspondence between the images and the images. The image processing device performs relearning of the learned model based on the location information.

8. The machine tool according to any one of claims 1 to 4, wherein, After performing the relearning, the image processing device updates the learned model using the relearned model according to instructions from the user.

9. The operating machinery according to any one of claims 1 to 4, wherein, During the operation performed by the machinery, the image processing device repeatedly performs the following actions: generating the input image based on the image acquired by the image acquisition device; detecting the object by inputting the input image into the learned model; and recording the input image when the object is detected. After receiving a signal indicating the end of the task, the image processing device performs relearning of the learned model based on detecting one or more input images of the object.

10. An image processing apparatus for detecting a specific object from an image acquired by an image acquisition device mounted on a work machine, wherein, have: A memory for storing a learned model for detecting the objects from an input image; as well as Operational circuits The computing circuit generates the input image based on the image acquired by the image acquisition device, detects the object by inputting the input image into the learned model, and performs relearning of the learned model based on one or more images of the object detected from a plurality of images acquired by the image acquisition device during the operation of the working machinery.

11. A method for detecting a specific object from an image acquired by an image acquisition device mounted on a work machine, wherein, include: The input image is generated based on the image acquired by the image acquisition device; The object is detected by inputting the input image into the learned model; as well as Based on one or more images of the object detected from a plurality of images acquired by the image acquisition device during the operation of the machinery, the learned model is relearned.

Images