Combine harvester

Abstract

A combine harvester is provided, comprising: a threshing device (4) for threshing rice stalks; a feed chain (4B) for supplying rice stalks to the threshing device (4); cameras (103, 104) and a distance measuring device (105) for monitoring the operator's manual threshing operation; and a work status estimation device (120) for evaluating the degree of danger of the operator's work based on images captured by the cameras (103, 104) and distance information obtained by the distance measuring device (105), thereby enabling appropriate evaluation of the danger of manual threshing operations. Furthermore, the cameras (103, 104) are positioned above the rice stalk conveying height of the feed chain (4B), and the distance measuring device (105) is positioned below the rice stalk conveying height of the feed chain (4B).

Description

Technical Field

[0001] This invention relates to combine harvesters. Background Technology

[0002] Previously, there were known combine harvesters equipped with cameras for filming manual threshing operations in order to assess the dangers of manual threshing.

[0003] Existing technical documents

[0004] Patent documents

[0005] Patent Document 1: Japanese Patent Application Publication No. 2013-9653 Summary of the Invention

[0006] The problem that the invention aims to solve

[0007] However, Patent Document 1 describes a situation where image analysis is used to determine if an operator has moved backward from the manual threshing position for a specified time. However, since only the moving time is evaluated, it cannot be considered sufficient as an assessment of the hazard, and there is considerable room for improvement.

[0008] Therefore, the main objective of this invention is to provide a combine harvester capable of appropriately assessing the dangers of manual threshing operations.

[0009] Methods for solving problems

[0010] The present invention, which solves the above-mentioned problems, is described below.

[0011] That is, the invention described in Scheme 1 is a combine harvester comprising: at least one camera (103, 104) for monitoring the manual threshing operation of an operator; and a work status estimation device (120) for evaluating the danger of the manual threshing operation based on images captured by said cameras (103, 104). The combine harvester is characterized in that the work status estimation device (120) comprises a posture estimation unit (122) for outputting coordinate information of the operator in said images. The evaluation of the danger of the manual threshing operation is based on the coordinate information of the operator output by said posture estimation unit (122), said coordinate information including coordinates corresponding to any one of the shoulder, elbow, head, neck, and waist.

[0012] The invention described in Scheme 2 is a combine harvester according to Scheme 1, characterized in that it further comprises a distance measuring device (105), and the operation state estimation device (120) is configured to evaluate the danger of manual threshing operation based on the distance information of the object obtained by the distance measuring device (105).

[0013] The invention described in Scheme 3 is a combine harvester according to Scheme 1 or 2, characterized in that the working state estimation device (120) has an approach determination unit (124) which evaluates the distance between a predefined danger zone and the position of the operator's torso in the coordinate system of the shooting area based on the coordinate information of the operator output by the posture estimation unit (122).

[0014] The invention described in Scheme 4 is a combine harvester according to Scheme 2, characterized in that the camera (103, 104) is positioned above the height of the feed chain (4B) that supplies the threshing device (4) with the stalks, and the ranging device (105) is positioned below the height of the feed chain (4B) with the stalks.

[0015] The invention described in Scheme 5 is a combine harvester according to Scheme 2, characterized in that the working state estimation device (120) is capable of evaluating the risk of manual threshing operations using the estimation results of the posture estimation unit (122) and evaluating the risk of manual threshing operations without using the estimation results of the posture estimation unit (122).

[0016] Invention Effects

[0017] According to the present invention, the hazards of manual threshing operations can be appropriately assessed. Attached Figure Description

[0018] Figure 1 This is a left-side view of a combine harvester.

[0019] Figure 2 This is a right-side view of a combine harvester.

[0020] Figure 3 This is a top view of the left side of the control unit.

[0021] Figure 4 This is a right-side view of the side panel of the control unit.

[0022] Figure 5 This is an explanatory diagram showing the control device of a combine harvester. Detailed Implementation

[0023] like Figure 1 , Figure 2 As shown, the combine harvester has a traveling device 2 consisting of a pair of left and right tracks that travel on the soil surface on the lower side of the machine frame 1, a harvesting device 3 for harvesting rice stalks in the field is set on the front side of the machine frame 1, a threshing device 4 for threshing and screening the harvested rice stalks is set on the left rear of the harvesting device 3, and an operating unit 5 for the operator to ride on is set on the right rear of the harvesting device 3.

[0024] An engine compartment 6 housing the engine E is located below the control unit 5. A grain bin 7 for storing threshed and screened grains is located behind the control unit 5. A discharge spiral 8, consisting of a winnowing section extending vertically and a transverse discharge section extending horizontally, is located behind the grain bin 7 to discharge the grains to the outside. The control unit 5 is covered by a cab 9 equipped with lighting at the top.

[0025] like Figure 3 As shown, a front panel 45 is provided in front of the operator's seat of the control unit 5, and a side panel 50 is provided on the left side.

[0026] A monitor 46 is provided on the front panel 45 to display the travel speed of the traveling device 2 and the output rotation of the engine E. In addition, an operating lever (not shown) for rotating the traveling device 2 and raising and lowering the harvesting device 3 is provided on the right side of the monitor 46.

[0027] The left side panel 50 is positioned close to the left front and rear frame 16D supporting the left wall of the cab 9. A gear shift lever 51 for operating the continuously variable transmission is located at the front of the side panel 50. An emergency stop switch (also called a "switch") 52 for stopping the engine E is located behind the gear shift lever 51. A harvesting and threshing lever 53 for operating the harvesting clutch that rotates the output of the engine E to the harvesting device 3 and the threshing clutch that rotates the output of the threshing device 4 is located to the right of the emergency stop switch 52.

[0028] An emergency stop switch 52 is provided in a recess 55 of the side panel 50, and the upper part of the emergency stop switch 52 is positioned approximately at the same level as the upper surface of the side panel 50. This prevents the emergency stop switch 52 from being accidentally pressed.

[0029] When viewed from the side, the front wall of the recess 55 is formed to slope forward and upward, and the rear wall is formed to slope backward and upward. Furthermore, in the longitudinal direction, the emergency stop switch 52 is positioned rearward of the handle of the harvesting and threshing lever 53, which is in a forward-tilted position after moving forward, and also rearward of the handle of the harvesting and threshing lever 53, which is in a rearward-tilted position after moving backward. Therefore, in the event of engine E overheating or other issues, the emergency stop switch 52 can be pressed quickly.

[0030] The rear of the side panel 50 has a power port 71 for drawing power from the vehicle's battery into the cab 9. This power port 71 is a USB (Universal Serial Bus) type port that supplies power to portable information terminals or similar devices held by the operator.

[0031] Additionally, a power steering lever is located at the front right of the cab 9. This lever is used to raise and lower the harvesting device 3 by operating it forward and backward, and to turn the machine by operating it left and right. Furthermore, a camera is mounted in the upper front of the cab 9 to photograph the front of the machine. This camera is used for visual and auditory monitoring by managers to check the operational status from a remote location, or for image analysis in evaluation systems such as machine learning. However, when the machine is turning, this camera cannot capture the operator's intended destination as a means of confirmation until the machine's direction of travel has actually changed, leaving room for improvement in convenience. Therefore, when operating the power steering lever, it is preferable to align the camera's optical axis with the direction of operation of the power steering lever, based on the operating angle and duration. Conversely, it is also possible to align the camera with the direction opposite to the operation of the power steering lever.

[0032] Next, the manual threshing section will be described. The manual threshing section refers to the part used to supply manually harvested rice stalks to the threshing device 4. More specifically, in the area where the front part of the feed chain 4B supplies rice stalks to the threshing device 4 is located forward of the front wall 4A of the threshing device 4, the conveying mechanism protrudes upwards. Therefore, this area on the front side of the threshing device 4, from the front wall 4A of the threshing device 4 to the front end of the feed chain 4B, constitutes the manual threshing section.

[0033] The manual threshing operation, in which straw is fed into the manual threshing unit, is dangerous when clothing, the human body, and the high-speed moving feed chain 4B interfere with each other. Therefore, when a dangerous situation occurs during operation, the vehicle control device 110 ( Figure 5 Correspondingly, controls are implemented to ensure the safety of feed chain 4B, etc.

[0034] An emergency stop switch 101 and a harvester clutch sensor 102 are connected to the input interface of the operating vehicle control device 110.

[0035] The output interface of the working vehicle control device 110 is connected to an engine control device 131, a harvesting clutch 132, a threshing clutch 133, a warning output device 134, a threshing drum cover release mechanism 135, etc.

[0036] Additionally, a work status estimation device 120 is connected to the work vehicle control device 110. A first camera (also called a "camera") 103 and a second camera (also called a "camera") 104 are connected to the work status estimation device 120, capable of inputting captured images. Furthermore, a LIDAR device (also called a "range measuring device") 105 is connected to the work status estimation device 120. The LIDAR device 105 is a known optical detection range measuring device; as long as it can search the distance to the surrounding environment in three dimensions, it can be replaced by other range measuring devices.

[0037] Although detailed illustrations are omitted, the emergency stop switch 101 is located on the front and left outer side of the threshing unit 4 and is operated when the operator feels danger. The harvesting clutch sensor 102 detects the engagement status of the harvesting clutch, which is located on the transmission path from the engine E to the harvesting unit 3.

[0038] The engine control unit 131 is described as being connected to an output interface for convenience, but in reality, it is connected to the work vehicle control unit 110 via a so-called CAN interface to communicate command signals, obtain status data of the engine E, or control its rotation speed. The harvesting clutch 132 and threshing clutch 133 connect and disconnect power from the engine E to the harvesting device 3 and threshing device 4. The warning output device 134 notifies the manual threshing operator of dangerous situations; its location is arbitrary, but it is preferably located around the front periphery of the threshing device 4. The threshing drum cover release mechanism 135 is located on the upper part of the threshing device 4 and is an actuator for forcibly opening the threshing drum cover covering the upper half of the threshing drum. Since a clamping rod opposing the upper side of the feed chain 4B is installed on the threshing drum cover, opening the threshing drum cover exposes the upper half of the threshing drum, and the stalk clamping in the feed chain 4B is released.

[0039] The first camera 103 and the second camera 104, connected to the work status estimation device 120 via wired or wireless means, are both cameras used to photograph the periphery of the manual threshing unit. The first camera 103 is fixed to the left frame on the lower left side of the cab 9. The first camera 103 is positioned with its optical axis oriented left-right, photographing the periphery of the manual threshing unit from the cab 9 side. The second camera 104 is located at the rear of the harvesting device 3, positioned in the space below the conveyor hood covering the upper side of the straw conveying mechanism leading to the threshing device 4, and fixed to the support frame of the conveyor hood. The optical axis of the second camera 104 is oriented front-back, photographing the periphery of the manual threshing unit from the front of the manual threshing unit, i.e., from the harvesting device 3 side.

[0040] The operation status estimation device 120 primarily evaluates the status of manual threshing operations based on image data captured by the first camera 103 and the second camera 104. Markers are provided on the front wall 4A of the threshing device 4 to determine the angle and position of the images captured by the cameras relative to the cameras. These marks are black-and-white images with a number of pixels in length and width, and their arrangement is recognized by the operation status estimation device 120. Multiple marks are assigned, and the relationship between the actual coordinate system and the camera coordinate system is evaluated based on their positional relationship within the image. For example, on the front surface of the front wall 4A, marks are assigned to the four vertices of a rectangle in a single plane. Based on the shape of the quadrilateral reflected in the camera image, the actual coordinate system can be identified, and the position and size of objects reflected in the image can be evaluated.

[0041] The first camera 103 and the second camera 104 perform coordinate system identification processing respectively. By considering their relative posture and position, the position of the object captured by the two cameras can be evaluated more accurately. In this way, the personnel detection unit 121, posture estimation unit 122, clothing determination unit 123, proximity determination unit 124, and motion determination unit 125, which are the various functions of the work state estimation device 120, use the coordinate information obtained by the above coordinate system identification processing together with the images of the first camera 103 and the second camera 104 to perform various processing.

[0042] It should be noted that the various estimation and judgment units of the operation state estimation device 120 are based on machine learning learned models. After basic learning for judgment is performed by providing supervision data, the vehicle body and camera are fine-tuned for optimization.

[0043] The personnel detection unit 121 of the work status estimation device 120 detects the area where a person is captured based on the images from the first camera 103 and the second camera 104. The personnel detection unit 121 outputs the determination information arbitrarily, and it outputs the area coordinates, including the size and position of the rectangle surrounding the area where the person is captured, along with a probability value for the person.

[0044] The posture estimation unit 122 estimates the posture of a person based on information about the person region extracted by the person detection unit 121. In posture estimation, the posture is estimated by applying the person in the image to a pre-determined human body shape model. That is, the positions corresponding to several key points such as the shoulder, elbow, and head are determined from the image, and a model of straight skeletal segments connecting these points is defined in the evaluation space of the actual coordinate system to estimate the posture and position of the worker. Furthermore, since the area captured by the first camera 103 and the second camera 104 is roughly limited to the upper body, only the posture of the upper body is estimated to reduce the computational load of image resolution.

[0045] The posture estimation unit 122 outputs meta-evaluation data related to the posture of the personnel by summarizing the coordinate information of each key point.

[0046] The clothing determination unit 123 estimates and determines the clothing status of the personnel in the images of the first camera 103 and the second camera 104, respectively. In particular, it estimates factors that have a significant impact on the hazard level of the operation, such as whether gloves are worn and whether there is any clothing disorder that could lead to potential involvement.

[0047] The clothing determination unit 123 determines whether gloves are worn. Specifically, based on the head or neck position information obtained by the posture estimation unit 122, and referring to the image pixel information of the area corresponding to that part, the color of the worker's skin is calculated. Similarly, the color of the skin in the area between the hands is calculated. Using the differences in these colors, and the evaluation results of a function that evaluates skin tone similarity based on a separately prepared color code (RGB value, etc.), it is determined whether gloves are worn. Furthermore, the clothing determination unit 123 also determines whether a towel is draped around the neck. Preferably, this determination is also based on the color of the neck, similar to the determination of glove wearing.

[0048] The above-described determination is performed in the clothing determination unit 123. However, since it is insufficient to evaluate the risk solely based on the presence or absence of problematic clothing, the risk level of the clothing is quantified and output.

[0049] The proximity determination unit 124 determines the extent to which the operator's body is close to the movable part of the manual threshing unit. During manual threshing, the operator positions their arms above the feed chain 4B on the manual threshing unit, and their forearms are obscured by straw bundles, making it difficult to assess the dangerous approach to the movable part in the image. Therefore, the proximity determination unit 124 uses the output of the posture estimation unit 122 to assess the dangerous approach to the movable part based on the extent to which the operator's torso is close to the danger zone. That is, the positions and shapes of the threshing device 4 and the feed chain 4B are known, so it is possible to pre-determine which area exists within the coordinate system of the camera area. Around them, the danger zone is defined as the area within a certain distance from the component, and the degree of danger is assessed based on the distance of the closest part of the torso (roughly the bone segment connecting the key points of the neck and waist). When defining a hazardous area, it is not necessary to maintain a uniform distance from the movable parts. It can be appropriately specified based on the actual risk. For example, it can be defined as being wider towards the downstream side of the feed chain 4B, i.e., behind the machine body, or as being wider at the exposed portion of the feed chain 4B or the opening of the threshing device 4. Furthermore, the proximity determination unit 124 quantifies and outputs the hazard level based on proximity, and preferably further distinguishes between the hazard level based on the threshing section (threshing cylinder) and the hazard level based on the feed chain 4B before outputting these values.

[0050] The motion determination unit 125 determines whether the operator's body has performed a dangerous movement. Dangerous movements are generally divided into two types: the first is a movement that rapidly approaches the movable part, and the second is a movement that rapidly deviates downstream (towards the rear of the machine) in the conveying direction of the feed chain 4B. Both may indicate signs that the operator is entangled in the movable part. In addition, even if the movement that rapidly approaches the movable part does not result in entanglement, it is appropriate to evaluate it as a high-risk operation.

[0051] Therefore, the motion determination unit 125 uses the output of the posture estimation unit 122 to monitor the movement of the torso, and in particular evaluates whether there is a large acceleration in the movement of the torso. In this way, it evaluates the speed of approach to the movable part, the speed of movement to the rear of the body, and the degree of danger of acceleration, and outputs them numerically.

[0052] With the structure described above, the work status estimation device 120 comprehensively evaluates the hazard of the work based on the output results from the clothing determination unit 123, the proximity determination unit 124, and the action determination unit 125, and outputs a response action command for the specified hazard elimination action to the work vehicle control device 110.

[0053] The response action instructions, based on the degree of hazard assessed by the work status estimation device 120, are in descending order of hazard level and include stopping the engine E, releasing the threshing drum cover based on the threshing drum cover release mechanism 135, stopping the feed chain 4B, slowing down the feed chain 4B, and issuing a warning based on the warning output device 134. Multiple actions listed above can also be combined. That is, for example, when performing an action with a higher hazard level, actions prepared for a lower hazard level can also be combined and executed. Furthermore, the threshold for distinguishing response actions can be adjusted by the operator and the machine.

[0054] It should be noted that in the above example, the job state estimation device 120 comprehensively considers the output results from the clothing determination unit 123, the proximity determination unit 124, and the action determination unit 125. However, the evaluation can also be performed by taking the weighted average of either the maximum value of their output risk level or a simple average. Alternatively, only one determination unit can be used. In this case, the computational load of the job state estimation device 120 can be reduced, and processing can be achieved with a device that requires low computational resources.

[0055] In addition, the operation status estimation device 120 evaluates the images of the first camera 103 and the second camera 104, but it can also use the images of a single camera or three or more cameras.

[0056] Additionally, the posture estimation unit 122 uses images from cameras (first camera 103 and second camera 104) to estimate the posture of the personnel, but it can also use cameras (first camera 103 and second camera 104) and a LIDAR device 105 to estimate the posture. In this case, it is preferable to position the LIDAR device 105 lower than the straw conveying height of the feed chain 4B (the height of the straw supply port formed on the side of the threshing device 4) so ​​that distance measurement can be performed in the left direction of the machine. That is, if the LIDAR device 105 is positioned near the straw conveying height, the conveying straw will obstruct the distance measurement, and it may be impossible to obtain the distance measurement data required for estimating the posture of the personnel. Alternatively, the LIDAR device 105 can also be mounted on the harvesting device 3, but it is necessary to consider the change in the relative position with the threshing device 4 due to the lifting and lowering operation.

[0057] By setting up the LIDAR device 105 as described above, the first camera 103 and the second camera 104 primarily acquire personnel information above the straw conveyor height, while the LIDAR device 105 primarily acquires personnel information below the straw conveyor height. As mentioned above, the posture of the torso is important for the posture estimation unit 122, but the neck position is inferred by image analysis from the cameras (first camera 103 and second camera 104), and the waist is inferred based on the ranging results of the LIDAR device 105, thereby estimating the posture of the torso.

[0058] According to this structure, at a position above the height of the grain conveying, the camera can be used to estimate the position of key points such as conveyed grain and grain scraps that are difficult to estimate due to noise in electromagnetic wave ranging methods, and can also obtain other hazard assessment information such as clothing. At a position below the height of the grain conveying, the camera can accurately obtain personnel posture information in the blind spot of the camera, and can accurately measure the relative distance to the threshing device 4.

[0059] Furthermore, while the LIDAR device 105 is used for posture determination by the posture estimation unit 122, unlike the processing performed by the posture estimation unit 122, the work state estimation device 120 can also output response action commands without going through the proximity determination unit 124 and the action determination unit 125 when an object is detected approaching at close range or moving at high speed. In this case, response action commands can be issued quickly by reducing the delay caused by information processing load, thereby improving operator safety.

[0060] Symbol Explanation

[0061] 4 Threshing device; 4B Feed chain; 103 First camera; 104 Second camera; 105 LIDAR device (range measuring device); 120 Operation status estimation device; 122 Attitude estimation unit.

Claims

1. A combine harvester comprising: at least one camera (103, 104) for monitoring a worker’s manual threshing operation; and a work status estimation device (120) for evaluating the degree of danger of the manual threshing operation based on images captured by said cameras (103, 104). The combine harvester is characterized in that... The work status estimation device (120) includes a posture estimation unit (122) that outputs the coordinate information of the worker within the image, and evaluates the risk level of manual threshing operations based on the coordinate information of the worker output by the posture estimation unit (122). The coordinate information includes coordinates corresponding to any one of the shoulder, elbow, head, neck, and waist.

2. The combine harvester according to claim 1, characterized in that, It also has a range measuring device (105). The operation status estimation device (120) is configured to evaluate the degree of danger of manual threshing operation based on the distance information of the object obtained by the ranging device (105).

3. The combine harvester according to claim 1 or 2, characterized in that, The work status estimation device (120) has a proximity determination unit (124) that evaluates the distance between a predefined danger zone and the position of the worker's torso in the coordinate system of the shooting area based on the coordinate information of the worker output by the posture estimation unit (122).

4. The combine harvester according to claim 2, characterized in that, The cameras (103, 104) are positioned at the upper part of the straw conveying height of the feed chain (4B) that supplies straw to the threshing device (4). The ranging device (105) is positioned below the height of the grain conveying of the feed chain (4B).

5. The combine harvester according to claim 2, characterized in that, The operation state estimation device (120) is capable of evaluating the risk of manual threshing operations using the estimation results of the posture estimation unit (122) and evaluating the risk of manual threshing operations without using the estimation results of the posture estimation unit (122).

Images