Work vehicle

Abstract

According to the present invention, an imaging device is given the function of measuring a distance from an object, a motion, and a shape, and given the function of determining a color tone, and image analysis is performed by artificial intelligence to determine a human and allows a work vehicle to perform safe operation for prevention of an accident. Thus, determination of a human by an unmanned robot work vehicle in an agricultural field is performed with high accuracy, and thereby an accident can be prevented.

Description

Work vehicles

[0001] The present invention relates to a work vehicle that determines whether a person is present or not by analyzing images captured by an imaging device, thereby driving the vehicle and operating a work machine.

[0002] Robotic work machines and remote-controlled driving technologies are desirable for agricultural and construction machinery, but there are also cases where people approach work vehicles in farm fields, making safety measures necessary. Human recognition is particularly important (Patent Document 1).

[0003] JP 2022-123742 A

[0004] Conventional technology includes a technique that recognizes specific work clothes in a specific location using an image and issues an audible alarm to alert the wearer.

[0005] However, it is difficult to identify a moving object as a human unless the location and clothing are limited. The present invention aims to provide a work vehicle that can use a neural network to identify humans in a field based on the position of their face, the movement of their limbs, and their color, and then safely control the vehicle.

[0006] The first aspect of the present invention is achieved by the following technical means.

[0007] The imaging device attached to the vehicle body has the functions of recognizing the shape of a moving object, measuring the position of the object, and recognizing the color of the object. The color recognition function is capable of recognizing the color of human skin among the colors of the object, and has the function of comparing the image of the object with pre-registered human shape data when the height position of the human skin color is at a predetermined height or higher from the ground.

[0008] The second invention is solved by the following technical means.

[0009] The imaging device attached to the traveling vehicle body has the functions of recognizing the shape of a moving object, measuring the position of the object, and recognizing the color of the object, and the color recognition function is capable of recognizing the color of human skin among the colors of the object, and as a means for identifying that the imaged object has a limb equivalent part and a head equivalent part which are part of a human shape, it is equipped with a size detection unit which detects the height and width of the object, the position and amount of movement of the limb equivalent part, and the position and amount of movement of the head equivalent part, and a color detection unit which detects the height position of the human skin color and the color of the limb equivalent part, and an image whose state has been changed is generated by matching the detection data of the size detection unit with the detection data of the color detection unit.

[0010] The third aspect of the invention is solved by the following technical means.

[0011] In the human-like shape, a stably fixed portion and a portion that partially oscillates are detected, and the limb-corresponding portion is the portion that partially oscillates.

[0012] The fourth aspect of the present invention is achieved by the following technical means.

[0013] The imaging device is provided with a lighting device near a color identification device with a color recognition function, and the color tone emitted from the lighting device can be modulated. If the image data detected by the size detection unit and the color tone detection unit do not match, the lighting device has the function of modulating the color tone emitted from the lighting device and changing the masking color determination.

[0014] The fifth aspect of the invention is achieved by the following technical means.

[0015] The color recognition device and the lighting device are disposed in front of and below the operator's seat, and are disposed parallel to the ground or facing diagonally downward. The sixth invention is solved by the following technical means: When the imaged object is determined to be a human, and the distance between the object and the vehicle is equal to or less than a first predetermined distance (α), the work equipment is stopped or the vehicle is stopped, and when the distance between the object and the vehicle is equal to or less than a second predetermined distance (β) that is shorter than the first predetermined distance (α), the operating function of the vehicle operating unit equipped with a steering wheel is cut off and operations are accepted only via external communication.

[0016] According to the first aspect of the present invention, a configuration can be adopted in which a human-like shape can be determined in a farm field.

[0017] According to the second and third aspects of the present invention, it is possible to determine whether a human-like shape in a field is a human or not by the action of a neural network.

[0018] According to the fourth aspect of the present invention, by taking measures when the detected data of the size detector and the color detector do not match, it is possible to prevent erroneous detection.

[0019] The fifth aspect of the present invention makes it possible to detect the placement position of the imaging device with higher accuracy.

[0020] The sixth aspect of the present invention enables safety control when the distance between a person and a work vehicle is within a dangerous range, and also enables response to accidents such as theft.

[0021] FIG. 1 is a perspective view of a work vehicle according to an embodiment of the present invention; FIG. 2 is a left side view of a work vehicle according to an embodiment of the present invention; FIG. 3 is a perspective view of the periphery of an imaging device of a work vehicle according to an embodiment of the present invention; FIG. 4 is a top view of the periphery of an imaging device of a work vehicle according to an embodiment of the present invention; FIG. 5 is a skin color matching diagram of a humanoid shape in the conversion process of a captured image of the present invention; FIG. 6 is a limb analysis diagram of a humanoid shape in the conversion process of a captured image of the present invention; FIG. 7 is a linear conversion diagram of a humanoid shape in the conversion process of a captured image of the present invention; FIG. 8 is an arm amplitude diagram of a humanoid shape in the conversion process of a captured image of the present invention; FIG. 9 is a foot amplitude diagram of a humanoid shape in the conversion process of a captured image of the present invention; FIG. 10 is a matching diagram of a humanoid shape face in the conversion process of a captured image of the present invention; FIG. 11 is a relationship diagram of the face color of a humanoid shape and ground height of the present invention; FIG. 12 is a relationship diagram of the amplitude of a humanoid shape and ground height of the present invention; FIG. 13 is a relationship diagram of the movement amount of the main body and limbs of a humanoid shape according to the present invention; A diagram showing the relationship between reflectivity and the height of a human-shaped shape. A block control diagram for determining a human being and changing the work process using shape change analysis using a neural network in an embodiment of the present invention. A diagram showing an imaging device arranged on the work machine when the work machine is located below the machine body in another embodiment of the present invention. A diagram showing an imaging device arranged on the front frame when the work machine is located below the machine body in another embodiment of the present invention. A diagram showing an imaging device arranged in the center of the cabin in another embodiment of the present invention. A diagram showing an imaging device arranged at the four corners of the cabin in another embodiment of the present invention. A diagram showing an imaging device arranged on the safety frame in another embodiment of the present invention. A diagram showing an imaging device arranged on the top link in another embodiment of the present invention. A diagram showing a millimeter-wave radar and an imaging device in another embodiment of the present invention. A diagram showing an imaging device arranged below the spare seedling tray in another embodiment of the present invention. A diagram showing an imaging device arranged to check the working status of the work machine in another embodiment of the present invention.

[0022] The present invention will be described below based on the embodiments shown in the drawings.

[0023] The work vehicle and related part diagrams shown in FIGS. 1 to 15 show an example of this embodiment.

[0024] The background of the present invention will be explained.

[0025] To address the labor shortage in agriculture, unmanned agricultural machinery, known as robotic work vehicles, are being used. These robotic work vehicles are operated under the supervision of a human in a remote location. The main reason for the human supervision is to prevent accidents such as a person approaching the work vehicle and being caught in the vehicle during operation.

[0026] To address this issue, a system is needed in which the robotic work vehicle itself can identify humans and perform safe control. This invention utilizes an imaging device to accurately analyze people and enables the robotic work vehicle itself to perform safe control, thereby realizing an unmanned work vehicle.

[0027] The construction of the work vehicle of the present invention will be described.

[0028] The present invention relates to the configuration and control of an imaging device generally provided on vehicles that travel and work in autonomous agricultural machinery, such as tractors that are fitted with work implements for tilling fields, rice transplanters that plant rice and vegetable seedlings, vegetable transplanters, combine harvesters that harvest rice and vegetables, and dedicated vegetable harvesters.

[0029] The configuration of the work vehicle in this description will be explained using a tractor 100 in Figure 1. As an agricultural work vehicle, the tractor 100 is a robotic vehicle that tows a work implement and is equipped with a satellite positioning device 111 and an inertial positioning device 112. Below this device that confirms the vehicle's position, an imaging device called a LIDAR (Light Detection and Ranging) 120 is installed. This is a type of remote sensing technology that uses light and is capable of measuring scattered light in response to pulsed laser irradiation to analyze the shape and properties of an object. It also makes it possible to measure the distance to an object, from long distances to short distances, and the moving speed of an object.

[0030] A CCD camera 130 serving as an imaging device is provided at the center of the front portion of the vehicle body. A left light 140 and a right light 150 are provided at a position rearward of the vehicle body with respect to the position of the CCD camera 130. As shown in Figures 2, 3, and 4, the left light 140 and the right light 150 are positioned above the CCD camera 130 and rearward in the direction of travel of the vehicle. They are located a distance rearward, corresponding to a dimension 142. Therefore, even if the CCD camera 130 has a wide-angle imaging range 131, the illumination range of the left light 140 is 141 and the illumination range of the right light 150 is 151, making it possible to sufficiently illuminate the imaging range 131.

[0031] In Figure 4, by configuring the illumination range 151 symmetrically so that the angle of spread is smaller than the line connecting the position of the end 132 of the CCD camera 130 and the right end 152 of the right lighting 150, the illumination ranges 141 and 151 do not directly illuminate the lens position of the CCD camera 130, so no halation or the like occurs and the color of the imaged object can be accurately acquired.

[0032] Furthermore, the CCD camera 130, which is an imaging device, is disposed below the left lighting 140 and the right lighting 150. This positional relationship also makes it possible to prevent halation.

[0033] It is well known that the positions of the CCD camera and lighting can be moved in the forward / backward, left / right, and up / down directions in this way, but it is important to consider the design of the mobile device when arranging it, and by locating the lighting unit in a recessed position, functionality and design are both achieved.

[0034] The satellite positioning device 111 is disposed above the CCD camera 130 and the left and right lighting devices 140 and 150, all of which are disposed forward of the operator's seat 101, or in the form of an operation box, in front of the operation box. The satellite positioning device 111 may be disposed further forward of the operator's seat 101 and operation box, but is disposed above or below the operator's seat 101 in a position that is not within the field of view of the operator's seat 101, and is disposed parallel to the ground or obliquely downward. In the embodiment shown in FIG. 2, the satellite positioning device 111 is disposed in front of the cabin, which is the operation box.

[0035] A method for analyzing an imaged object using a combination of two types of imaging devices, a CCD camera 130 and a LIDAR 120, will be described.

[0036] 5 shows an image of a person (humanoid figure), and the captured image 200 is an image taken by the CCD camera 130, showing the color and shape. Since the figure is not shown in color, it is shown using diagonal lines. Note that the relationship between the shape of the diagonal lines and the color is not specifically shown, but the difference in the diagonal lines means that the color has changed.

[0037] In this way, the work vehicle has basic data for recognizing humans registered in the data storage unit S15-20 in Figure 15. The basic data includes a large number of registered shape images of human figures, including image data of people walking in a field, standing upright, and standing with their legs spread naturally, and also includes data on the external shape, limb position, head, amplitude range and color of each part. In particular, skin color recognition of the head position is the basis for the human figure search of the present invention. The human figure may be controlled by roughly matching human figures that are similar to the skin color recognition of the head position.

[0038] In human recognition, it is possible to make a highly accurate judgment if the image is set to specific imaging conditions and is easy to analyze, but the present invention is to identify humans in unexpected situations, so first the skin color at the head position of the human-like shape and the human-like shape are matched, and then the search target is focused on, and an image of that target is taken and a detailed analysis is performed; this detailed analysis will now be explained.

[0039] The CCD camera 130 may be a monocular camera, but as explained above, its relationship with the lighting device is important for accurate color recognition. In this image, captured image 201 detects a color tone corresponding to skin color, but the reflectivity is medium to slightly high, and the object surface has small irregularities, preventing a significant increase in reflectivity. In contrast, captured images 202, 203, and 206 are of colors other than skin color, and compared to captured image 201, they show areas with higher and lower reflectivity. Captured images 204 and 205 may be used to recognize skin color or may detect other colors.

[0040] The captured image 210 is an image of the same object captured by the LIDAR 120. The image is point cloud data. In this figure, the points are displayed as large circles (circles) for easy understanding, but in the actual image, they are fine dots, and the point cloud shape can also be configured as a smooth shape.

[0041] Shape recognition on a two-dimensional screen is almost the same type of planar recognition as that of the CCD camera 130, but because the distance between the object and the LIDAR 120 can be calculated, it is possible to estimate the three-dimensional shape of the captured image 210. In addition, by detecting the movement of the object through point cloud movement detection, it is possible to determine the detailed three-dimensional shape of the captured image 210.

[0042] Therefore, captured image 220 can be synthesized by combining captured image 200 and captured image 210. A feature of the synthesized image is that it allows for color to be added to a three-dimensional image. By analyzing local movements, it is possible to identify parts of an object that move instantly and parts that remain stably fixed. Furthermore, by analyzing the manner in which an object moves in a linked manner, it is possible to identify the parts that serve as the fulcrum of the link. By analyzing the differences in the speed of this movement and the link configuration, it is possible to distinguish between the main body and the branches and leaves of the object. For example, in the case of a human, it is possible to distinguish between the torso, limbs, and head. In captured image 220, only skin color is extracted and color-recognized in the point cloud data. Captured image 221 is the skin-color recognition region. In this way, captured image 201 from the CCD camera is synthesized with captured image 211 using LIDAR to produce captured image 221. This image analysis trend is the same for each region, from captured image 206 ⇒ 216 ⇒ 226 and captured image 205 ⇒ 215 ⇒ 225. This image analysis method is a technique for analyzing image data, corresponds to the first state change, and is the control section of S15-21 in FIG.

[0043] Regarding the recognition of human-like shapes, priority is given to detecting the skin-colored position from the field in the images captured by the LIDAR 120 and CCD camera 130, and if the skin-colored position is at a specified height from the ground in the field and the corresponding object is a moving object, it is analyzed as a human-like shape, and if multiple human-like shapes are recognized, a means is taken to analyze each individual individually.

[0044] Next, referring to FIG. 6, the flow of analyzing object movement from image movement will be described. The left diagram in FIG. 6 is the same as the right diagram in FIG. 5, and is a captured image 220. The captured image 220 moves, resulting in captured image 220A. The object's total height 230 changes to total height 230A, but there is no difference in size. Similarly, there is no difference in size between the height of the skin-colored portion 231 and 231A. However, there is a difference in size between dimension 232 and 232A and dimension 233 and 233A. This difference can be analyzed as a change, even if it becomes longer or shorter. However, since the object's total height does not change, dimensions 232 and 233 are the branches and leaves of the object, which would correspond to the limbs of a person, and this can be analyzed from the change in the first state change.

[0045] By narrowing down the LIDAR image, it is also possible to convert the point cloud data into linear data. Captured image 220B in Figure 7 is a diagram obtained by linearly converting captured image 220. Similarly, captured image 220C is a diagram obtained by linearly converting captured image 220A. By converting both to linear data, the main body part and the peripheral part can be more clearly distinguished.

[0046] Dimension 232B changes to dimension 232C when the mounting angle changes from dimension 234B to dimension 234C. Dimension 233B changes to dimension 233C when the mounting angle changes from dimension 235B to dimension 235C. However, since dimensions 230B and 230C, which are the overall height, do not change, it is inferred that dimensions 232B and 233B are parts of branches and leaves that move, and since there is little change or movement in the skin-colored parts between dimensions 231B and 231C, it is inferred that captured image 220 is a human.

[0047] Figure 8 examines whether it represents a human arm. In the captured image 220D, dimension 232D moves very quickly. It moves smoothly and quickly between 8S1 and 8S5. In contrast, movements of 8S7 to 8S12 are also observed. This movement is more frequent and often faster than the movements of dimension 233E, which is assumed to represent a foot in Figure 9, 9S1 to 9S4.

[0048] This technology for judging people using an image capture device can be used to judge the balance and movement of the eyes, nose, mouth, ears, and other parts of the human face, as well as the color of the face, but it is only used to capture images in an easy-to-understand manner so that people can recognize the person in the image. However, when it comes to recognition for the purpose of preventing theft or accidents, there is no way to capture images in an easy-to-understand manner.

[0049] However, the work vehicle used in this invention is for agricultural use and is intended for identification in fields such as rice paddies. Therefore, human movement is often limited. When humans move in fields, they walk on two legs, and it is unlikely that they move while sitting or crawling on all fours. Inside homes, people move in a variety of ways, such as sitting, lying down, or using chairs or desks, resulting in a wide range of changing conditions, making it difficult to identify people. A difficult aspect of identifying people in fields is the wearing of hats, gloves, boots, etc. Furthermore, wearing a towel around the neck or a mask can also make it difficult to identify the head, but walking in fields is often distinctive.

[0050] In the field, people do not sit on the soil, nor do they walk while crouching. Since it is particularly difficult to maintain balance on soft fields, people tend to walk with their arms and legs stretched out, which makes it easy to convert to the images shown in Figures 5 to 7. Images of arm movements in Figure 8 and foot movements in Figure 9 are often captured at anticipated measurement positions, allowing for image analysis and matching with minimal computational effort. Thus, in the field where the present invention is implemented, human movement is limited, making it possible to perform analysis from captured images.

[0051] In this way, when the imaging device is equipped with a LIDAR device and a camera that recognizes color, and the position that can be determined to be skin color from the ground is at a certain height, it is clear that it is easy to match human shapes.

[0052] Figures 10, 11, 12, and 13 illustrate control for dealing with misidentification of non-human animals. As mentioned above, in farm fields, identifying the head can be difficult due to the wearing of hats, gloves, boots, towels around the neck, and masks. However, in the case of humans, due to movement, it is possible to recognize skin-colored areas by capturing an image from a certain direction. In Figure 10, the dimensional ratio of the position of the dimension 231F of the face area relative to the total height 230F and the ratio of the dimension 241 itself are calculated. Furthermore, because dimension 241 also moves, an acceptable range is also defined as dimension 242, which is also determined by the ratio. In this way, regardless of the actual dimensions of the total height 230F, the likelihood of it being a human can be confirmed by adjusting the ratio.

[0053] FIG. 11 shows subtle differences in the color of skin tones depending on the relationship between RGB color hues and total height 230F. The X-axis represents the red hue of RGB, and the Y-axis represents total height 230F. Although this differs depending on weather and field conditions, under the same conditions, when the total height 230F is small (S11-1), a short person may have a slightly stronger red color due to the reflected light from the field and the image from the imaging device being slightly downward. Conversely, when the person is tall (S11-2), blue colors are expressed slightly stronger, and these color corrections also allow for accurate analysis of the face and its position.

[0054] FIG. 12 shows the relationship between the height of an object and the amplitude of the object's movement by superimposing consecutively captured object images at object heights 230, 230A, 230B, 230C, 230D, 230E, and 230F. The X-axis represents the object's height, and the Y-axis represents the object's amplitude. In this embodiment, the image amplitude is large at low position 12S-1 and intermediate position 12S-2. Positions 12S-1 and 12S-2 correspond to the position of a person's feet and hands, respectively.

[0055] Furthermore, 12S-3 corresponds to the torso of a human, and 12S-4 corresponds to the head of a human, and the amplitude is low. By synthesizing the outline of an object in this way, the amplitude of a specific part is large, and if the position matches, it is possible to estimate that it is a human. In this embodiment, the solid line indicates the registered reference amplitude value, and the measured value is displayed as a two-dot chain line. The dashed line indicates a large quadrupedal animal, and it can be seen that the tendency is different. Note that the analysis is performed using mathematical means, and if the matching rate is above a predetermined level, it is determined to be a match.

[0056] As mentioned above, such human judgments are action judgments based on specific actions taken by humans in the field. In fields where the soil is soft, such as after plowing, plowing, or immediately after rice planting, bipedal walking often involves stretching out the arms and legs, which increases the accuracy of image judgment. However, on ordinary asphalt pavements, the footing is stable, and people often run with bent arms or carry luggage, making it difficult to determine whether they are humans. Control that primarily recognizes people as moving objects is more immediate. However, recognizing people in the field has different requirements. In emergency responses other than work-related entrapment accidents, the prevention of theft and vandalism requires not only accurate recognition of moving objects but also accurate recognition of people.

[0057] For human judgment, it is necessary to increase the degree of external shape matching. This is the analysis of the amount of movement of the limbs mentioned above. The degree of amplitude can be measured as the amount of movement in a simple external image, or by measuring the movement of a specific color. It is also possible to filter out everything except the moving parts, then use a convolutional neural network to match specific parts and calculate the degree of overlap. The human shapes that serve as the basis for comparison are registered as cloud data, and can be handled using ICT technology on mobile devices, reducing the load on the work vehicle's CPU.

[0058] Both methods extract the center and head positions of the humanoid shape from the image, calculate the relative and absolute movement amounts relative to the limbs, and perform image control to improve the recognition of the humanoid shape and thereby increase the accuracy of human identification. This is explained with reference to FIG. 13 . Continuously capturing images of the same humanoid shape, over a predetermined period of time, humanoid shape 220G moves to humanoid shape 220H. When measuring the movement amount at the center of the humanoid shape in these images, dimension 251 is detected. Furthermore, a movement amount of dimension 252 is detected for the part estimated to be the humanoid's head. In either case, the movement amount estimated as the main body of the humanoid shape is small. However, compared to this movement amount, the movement amounts estimated as the left and right arms are detected as dimensions 253 and 254. Furthermore, the movement amounts estimated as the left and right feet are dimensions 256 and 257. The movement amount of dimension 257 for the stepping out on the moving side of the human body is particularly large. Thus, dimensions 251 and 252 are considered to be the movement amount of the main body of the humanoid shape and are absolute movement amounts. In comparison, dimensions 253, 254, 256, and 257 are relative movement amounts, which are movement distances including the amount of movement of the main body. In this case, dimensions 253, 254, 256, and 257 are greater than dimensions 251 and 252, which are the amount of movement of the main body, and the parts of dimensions 253, 254, 256, and 257 are estimated to be human limbs. Furthermore, because main body center dimension 251 is much smaller than dimension 252, dimension 252 is estimated to be the head, and it is determined that there is a high possibility that humanoid shapes 220G and 220H are human.

[0059] Based on this technology, the fourth invention performs safety control for both the target object and the work vehicle. If it is suspected to be a human, it is extremely dangerous to approach the work vehicle. If the distance to the work vehicle falls below a predetermined distance α, the work equipment will be stopped if it is attached, and then the vehicle will be stopped from traveling. In addition, information on the emergency stop is immediately uploaded to the cloud, allowing users to share the information.

[0060] If the human-like figure approaches within a predetermined distance β even after the vehicle has stopped, this is considered a dangerous act and may be a mischief or theft attempt. In this case, manual operation of the vehicle is blocked. Simultaneous operation is blocked, and operation is only accepted via external communication, i.e., remote operation is the only option.

[0061] If the operating seat is a cabin type, it is also possible to lock the doors, lock the steering wheel, and fix the operating levers using mechanical locks.

[0062] The values ​​of α and β can be changed depending on the vehicle type and also on the working speed. α is about 4 m and β is about 2 m, and can be set arbitrarily, but upper and lower limits should be set in the settings, and even if the safety control can be turned off, there is an automatic return function after a certain time has passed.

[0063] In addition to color and movement amount (amplitude), the aforementioned amount of light reflection is also used as a criterion for determining whether a person has a humanoid shape. The amount of reflection is indicated by the relative sensitivity of the CCD camera 130 and is calculated by analyzing specific wavelengths of near-infrared light. The left lighting 140 and right lighting 150 are composed of LED lights, and the color is modulated by changing the frequency or wavelength, thereby correcting and controlling the relative sensitivity of the CCD camera 130. When working in a farm field, the desired light intensity or wavelength may not be obtained depending on the working time or weather. Therefore, the left lighting 140 and right lighting 150 control their LED lights to correspond to the light intensity and wavelength used as the reference for the CCD camera 130.

[0064] As mentioned above, captured image 201 detects a color tone corresponding to skin color, but the reflectivity is medium to slightly high, and the object surface has small irregularities that prevent the reflectivity from increasing significantly. In contrast, captured images 202, 203, and 206 are of colors other than skin color, and compared to captured image 201, they show areas with higher and lower reflectivity. The higher reflectivity areas include resin or enamel boots, raincoats, hats, etc. Conversely, the lower reflectivity areas are clothing made from materials such as cotton. Clothing made from synthetic fibers has a higher reflectivity than cotton, but is still low enough to be distinguishable compared to the reflectivity of skin.

[0065] The relative sensitivity of the CCD camera 130 corresponds to the reflectance at a specific wavelength. Therefore, it is possible to identify human skin by extracting the median value of the relative sensitivity. Identifying this median value of relative sensitivity is necessary for each wavelength, and it can be registered in advance in cloud data and identified in the work vehicle by performing a comparison operation.

[0066] Figure 14 shows the relationship between the reflectance of the light intensity of the lighting device and the height of the human figure. The X-axis represents the reflectance of the light intensity of the lighting device, which is similar to the relative sensitivity of the CCD camera 130. The Y-axis represents the height of the human figure. S14-1 is the shoe part. In the case of boots, the reflectance is extremely high. S14-2 is the lower body part, but the reflectance is low due to clothing. S14-3 corresponds to the upper body and face part. The face part has an intermediate reflectance, but if the clothing is open at the chest or short sleeves, it may show high reflectance. S14-4 is the hair part. If the hair is long and black, the reflectance is extremely low. Straw hats and other items have rough surfaces and are prone to similar trends. S14-5 is the top edge of the contour, which may be detected as high depending on the angle of sunlight.

[0067] While it is technically possible to identify humans using the reflectance of objects in this way, unlike the prior art in JP 2022-123742, it is not limited to identifying people wearing the same clothing on the premises of a factory or the like. Therefore, in a farm field, if a person is wearing sunglasses or a black mask, has long black hair, and is not wearing a hat, the reflectance will be reduced, which may result in false recognition. Therefore, if a person is recognized as possibly being human, issuing an audio warning is also a necessary safety control measure. For example, if a voice message saying "Danger! Do not approach the vehicle. It may suddenly accelerate," is played, it is expected that a human will respond to the voice.

[0068] In another embodiment, a radiation thermometer may be added to detect the temperature of an object, allowing estimation of the presence of a human.

[0069] Furthermore, as mentioned above, the lighting device can modulate the color by changing the frequency or wavelength, which makes it possible to detect even when language is not recognized, and even if a large animal other than a human is mistakenly recognized, safety control can be performed by adjusting the lighting device.

[0070] The control of identifying humans from images captured by an imaging device described above can be achieved using conventional analysis techniques, but doing so inside a work vehicle is difficult as it requires calculation and analysis solely by the CPU that controls the vehicle.

[0071] In other words, capturing a large number of images and searching and matching them on a large number of cloud servers places a burden on the CPU and takes time, making it impossible to use for urgent analysis.To solve this problem, it is necessary to control the work machine to make human estimation and judgment using only the work machine, and it is also necessary to have a function that can learn the criteria for this judgment.

[0072] Accuracy can be further improved by using a neural network, an artificial intelligence, as a means of identifying humans from image data. This process is explained in the block control diagram of human identification and work process changes through shape change analysis using a neural network in Figure 15.

[0073] In the first invention, the function of measuring the shape, movement, and position of an object based on the distribution of the object is the LIDAR 120, which is the first imaging device S15-1, and the function of recognizing color is the CCD camera 130, which is the second imaging device S15-7. By combining the images from both imaging devices, it is possible to determine whether the imaged object is moving or fixed, and to detect specific movement from the size, shape, and color of the object.

[0074] Although human detection is an important detection method, because humans wear a variety of clothing, human detection can be estimated based on color and shape, but is not conclusive. Therefore, in the present invention, limited control is performed in a specific location, namely, a farm field. As described above, when a human moves in a farm field, they are walking on two legs, and the positions of their face and limbs are within a limited range of positions. In reality, the work vehicle works in the farm field, and its position in the field is confirmed and controlled using a satellite positioning device. As described above, the image data from the LIDAR 120 and CCD camera 130 can detect the overall height and skin color position of a human-like shape from the position on the ground in the field. This allows the work vehicle to roughly determine whether it is a human.

[0075] The second invention further analyzes the human shape.

[0076] The first image capture device S15-1 performs analysis in the size detection unit S15-2 of the humanoid shape by combining images captured by the LIDAR 120, data from the obstacle sensor S15-23, and data from various vehicle sensors not shown in this block diagram. A first state change step S15-21 is performed to analyze the humanoid shape using a neural network based on the strength of the connections (synapses) between each piece of data. The analysis calculates the height S15-3 of the entire humanoid shape, the width S15-4 of the entire humanoid shape, the position and amplitude S15-5 of the limb-corresponding portion, and the position and amplitude S15-6 of the head-corresponding portion. This state corresponds to the captured image 210 in FIG. 5.

[0077] The second image capture device S15-7 is a CCD camera 130 that recognizes color and shape, and performs a humanoid shape color detection unit S15-8. The captured image and data from various vehicle sensors (not shown in this block diagram) are combined, and a first state change S15-21 is performed to analyze the humanoid shape through the strength of the connections (synapses) between the data using a neural network. This analysis process, which involves changing the image state, calculates the skin color position S15-9, the color corresponding to the limbs S15-10, the masking color determination S15-11, and the relative sensitivity detection S15-12. This state corresponds to the captured image 220 in Figure 5. The masking color determination S15-11 is initially set to a standard setting, but the third invention has a function that automatically changes the setting value while learning.

[0078] By combining the state-changed images from the humanoid shape size detection unit S15-2 and the humanoid shape color detection unit S15-8, the combined image is further analyzed in the second state change S15-22. This image analysis process is the image change process shown in Figures 6 and 7. At this stage, it is possible to recognize and infer a humanoid shape, but further image analysis of the limb amplitude is performed. The basic data in the data storage unit S15-20 is compared with the state-changed image to determine whether the person is a human S15-14.

[0079] This image analysis process is an analytical method for identifying a human by analyzing the image using a neural network to compare the movement of the humanoid shape, the movement of the limbs of the humanoid shape, and the amount of movement of the object itself with the amount of movement of the limbs, and by comparing the registered data with the state change data analyzed from the imaging device, and this is determined by the image analysis processes of Figures 8, 9, 10, 11, 12, 13, and 14.

[0080] In this way, the analysis is carried out by taking into account the data obtained from the imaging device and various sensor data, repeatedly changing the image state, comparing it with the data storage unit, registering the differences, and using them for the next data analysis.The analysis method used is an artificial intelligence circuit, or so-called neural network analysis method.

[0081] In using the neural network of the present invention, the series of steps is explained using the block diagram in Figure 15, and the analysis method based on the state change of each image is described above in Figures 5 to 14. This neural network makes it possible to identify people from captured images.

[0082] Furthermore, by utilizing the neural network data of the present invention, the aforementioned emergency stop determination S15-15 and theft prevention determination S15-16 are performed. Furthermore, there are cases where this human recognition is incorrect, and a false recognition estimation S15-17 is always performed in the background control. For example, if an image is recognized as a human, but image data captured a predetermined time after the emergency stop does not show any limb movement that would be considered human, and it is determined that another animal has been mistakenly detected, an operation restart plan S15-41 is made and operation is restarted.

[0083] The learning function of this neural network is to automatically determine the function of the work process change and determination unit S15-30. Although manual operation is also possible by receiving instructions from a mobile terminal or remote control device S15-31, in the present invention, the work process plan S15-37, operation resumption plan S15-41, and vehicle safety control S15-45 are performed by changing the weighting of information based on the learning level using data from the cloud S15-34, the satellite positioning device S15-32, and the status change information S15-14 to S15-17 described above.

[0084] In one example, in the driving route S15-38, a route is automatically created that does not pass through the location where a human is found, and this location is registered in the map control S15-40.From the next time onwards, a route is created that denies the robot from passing through this location, thereby making the user aware of this.

[0085] In addition, with the functions described above, the control timing and accuracy are further adjusted by the learning function in steps S15-42 to S15-49.

[0086] The basic functions of the third invention will now be described.

[0087] There are cases where the image data detected by the humanoid shape size detection unit S15-2 and the humanoid shape color detection unit S15-8 do not match. In the captured image 220 of FIG. 5, the image of the LIDAR 120 of the area corresponding to the face and the skin-colored area of ​​the CCD camera 130 are misaligned and do not match. This state can occur due to differences in the illuminance of external light, when the reference color for color recognition is misaligned due to reflected light in the field, when the background colors, such as the color of the soil in the field and the color of the sky, are similar and identification becomes unclear, or when the humanoid shape is moving quickly.

[0088] To address this issue, the imaging device is equipped with a lighting device near the device with the color recognition function, and the color tone emitted from the lighting device can be modulated. In this embodiment, the color tone of the left lighting device 140 and the right lighting device 150 can be modulated. When the image detection data from the humanoid shape size detection unit S15-2 and the humanoid shape color detection unit S15-8 cannot match, the modulation of the color tone emitted from the lighting device and the masking color determination S15-11 are changed to address this issue. Note that the masking color determination S15-11 has a learning function, and by registering data in the field map, if specific changes are required depending on the time and location, the lighting modulation and masking color determination are automatically changed, and this will automatically be addressed in future uses.

[0089] As described above, the function shown in FIG. 15 combines images from the imaging device and allows the calculation function in the work vehicle's CPU to perform analysis, enabling immediate processing and decision-making even if a person suddenly appears in front of the work vehicle. With conventional technology, even if a person suddenly appears, the only recognition possible is that of an obstacle. This allows for an emergency stop of the vehicle, but it is not possible to stop the work equipment or take other measures, such as theft prevention. Analysis and comparison of images via communication with the cloud takes time, making it difficult for conventional control systems to immediately anticipate human presence. By incorporating artificial intelligence functions, the present invention enables identification and decision-making within the vehicle, enabling emergency response.

[0090] However, this detection is limited to agricultural work and fields. However, since the images were not taken under conditions where the robot wanted to recognize itself, the conditions for analysis are not easy. To address this, the imaging device is equipped with a LIDAR device and a camera that recognizes color, the color emitted from the lighting device is modulated, and the walking state of people in the field is registered as a database for comparison, allowing for immediate analysis to determine whether they are human, although this is a guess under limited conditions.

[0091] However, even when matching skin color, there are differences in the imaging conditions and individuals, and even creating a database through extensive learning may not be enough to cover everything. Also, although there is a function to change the matching criteria and tolerance range, there is a high possibility of false recognition in situations where the filtering function is turned off.

[0092] Considering such condition analysis, increasing the accuracy of the judgment criteria for a specific element may result in an inability to make judgment analysis or may even result in misidentification. The invention that solves this problem is the content of the present invention, which makes a comprehensive judgment by measuring and analyzing various elements. In other words, rather than detecting a single condition with high accuracy, it is a method that increases accuracy by combining multiple conditions in a cumulative manner.

[0093] It is desirable to have technology that combines a large number of conditions and processes data in a way that resembles human recognition and judgment, and this is largely due to the unique technology of artificial intelligence, as in the present invention, and is made possible by incorporating control that processes images and responds to changes and outputs in the work process in a way that resembles human judgment.

[0094] This neural network's discrimination function also works outside of farm fields, which are outside the conditions of the first invention. However, when it comes to identifying people, people are in various states, and the accuracy of identifying them as humans decreases. However, while it cannot recognize people, it can recognize them as moving objects, so it can be dealt with by using an imaging device to recognize them as moving objects or obstacles and taking evasive action.

[0095] This function detects foreign objects and obstacles and stops the vehicle when it enters a farm road from a field. Because farm machinery is operating in the field, there is a risk of being caught in an accident, so the accuracy of human identification is necessary, but in other cases, accidents can often be avoided by stopping the vehicle.

[0096] Furthermore, while the speed of vehicles travelling on farm roads is high and theft while in motion is difficult, it is possible for someone to get into a work vehicle while driving at low speed in a field, so human recognition is necessary to deal with theft.

[0097] Next, the mounting positions of the imaging device and satellite positioning device of the present invention for different models will be described.

[0098] 16 shows a model in which the working unit is mounted in the center of the vehicle body, on the underside, and a work implement 282 such as a mower is located below the vehicle body. In this model, a stay extends from below the power frame in a position that is not affected by the movement of the work implement, and an image capture device 291 is mounted in front of the work implement and above its uppermost position, so that the area in which the work implement will be working can be captured in advance, and in the case of a mower, this can be used to obtain information such as the height and direction of the grass depending on the grass, mowing height, and power control.

[0099] Fig. 17 shows a different configuration from Fig. 16, in which the imaging device is disposed in the center of the vehicle body in front of the front axle 301. In Fig. 16, the imaging device is disposed at a low position that matches the height of the work machine, and images can be taken horizontally to the ground, but changing the height of the work machine can make it difficult to measure the distance accurately. In Fig. 17, however, the imaging device 302 is disposed as low as possible in a location where the position can be fixed, making it possible to take images horizontally to the ground under conditions where the distance can be measured accurately.

[0100] 18 shows the case of imaging from a high position. In order to mount the imaging device in a stable position on top, it is best to install it on top of the cabin 303, which is the operation room. Among these, when measuring while rotating 360 degrees like the LIDAR 304, it is best to install it in one place in the center of the cabin.

[0101] Furthermore, when taking continuous images, it is difficult to rotate the image capturing device, so by arranging the image capturing devices 305 at the four corners of the cabin as shown in FIG. 19, continuous image capturing in all directions is possible.

[0102] Figure 20 shows the safety frame specifications. To ensure safety in the event of a vehicle tipping over, there is a safety frame 306 that protects the operator up to the head, and this frame is equipped with an imaging device 307. Since the safety frame has a folding function, the folding shape must be taken into consideration.

[0103] Figure 21 shows the mounting position of the rear camera on a model without a cabin. There is a link to which the work equipment is attached, and by placing the camera on this link, it is possible to take images that match the movement of the work equipment. In Figure 21, the camera 309 is placed on the top link 308.

[0104] FIG. 22 shows an arrangement in which the millimeter-wave radar 260 is installed simultaneously with the imaging device 262. The use of millimeter waves in agricultural machinery allows for not only measuring the distance to an obstacle but also detecting the obstacle's speed with high accuracy. This makes it effective for detecting obstacle movement at the edge of a field, cooperative work, and detecting crops ahead. Since a lower, forward position is advantageous for crop detection, the imaging device 262 can also be installed using a member 263, such as a frame for attaching and detaching a front weight. When the millimeter wave radar is installed at a lower, forward position, the imaging device 262 should be installed at a higher position than the millimeter-wave radar 260, making it easier to use for detecting distant positions. The imaging device 262 can be accommodated within the eave-forming portion 261, which extends the cabin ceiling, to address the diffuse reflection of sunlight, rainwater, wind, and snow.

[0105] Figure 23 shows the relationship between the satellite positioning device 272 and the imaging device 271 in the transplanter. The imaging device 271 is supported by a stay 273 of the satellite positioning device 272 and is installed at a lower position than the satellite positioning device 272. Furthermore, if the transplanter has a control mechanism that detects ridges or obstacles and stops the vehicle, the imaging device 271 is installed below the spare seedling tray 274 and behind the stay 273 at the front end of the spare seedling tray. This makes it easy to remove seedling frames and allows the spare seedling tray 274 to protect the imaging device 271 in the event of a collision with an obstacle ahead.

[0106] 24 is a diagram showing the position of an imaging device 281 for checking the condition of planted seedlings in a rice transplanter. The imaging device 281 is placed above the work machine 282 that constitutes the planting section, at a position equivalent to the upper part of the seedling tank. This makes it possible to deal with mud splashed by the work machine 282 that constitutes the planting section, and also makes it easy to check the seedling tank.

[0107] 100 Tractor 120 Imaging device (LIDAR) 130 Imaging device (CCD camera) 140 Left lighting (LED light) 150 Right lighting (LED light) 200 Captured image (CCD camera) 210 Captured image (LIDAR) 220 Captured image (composite) S15-21 First state change (neural network) S15-22 Second state change (neural network)

Claims

1. A work vehicle in which an imaging device attached to a moving vehicle body has the functions of recognizing the shape of a moving object, measuring the position of said object, and recognizing the color of said object, said color recognition function being capable of recognizing human skin color among the colors of said object, and having the function of matching the image of said object with pre-registered human shape data when the height position of said human skin color is at or above a specified height from the ground.

2. An imaging device attached to a traveling vehicle body has a function of recognizing the shape of a moving object, a function of measuring the position of the object, and a function of recognizing the color of the object, the color recognition function being capable of recognizing human skin color among the colors of the object, and as a means for identifying that the imaged object has a limb equivalent and a head equivalent which are part of a human shape, a size detection unit is provided which detects the height and width of the object, the position and amount of movement of the limb equivalent and the position and amount of movement of the head equivalent, and a color tone detection unit which detects the height position of the human skin color and the color of the limb equivalent, and a work vehicle characterized in that the detection data of the size detection unit and the detection data of the color tone detection unit are matched to generate an image whose state has been changed.

3. A work vehicle as described in claim 2, characterized in that within the human-like shape, stably fixed parts and parts that partially oscillate are detected, and the limb-equivalent parts are the parts that partially oscillate.

4. A work vehicle as described in claim 2, further comprising a lighting device disposed adjacent to a color identification device having a color recognition function within said imaging device, the color tone emitted from said lighting device being modulated, and having a function of changing the modulation of the color tone emitted from said lighting device and the determination of the masking color when the image detection data of said size detection unit and said color tone detection unit cannot be matched.

5. A work vehicle according to claim 4, wherein said color recognition device and said lighting device are disposed in front of and below the operator's seat, and said color recognition device and said lighting device are disposed parallel to the ground or facing diagonally downward.

6. A work vehicle as described in any one of claims 1 to 5, wherein if the imaged object is determined to be a human, and if the distance between the object and the vehicle is equal to or less than a first predetermined distance (α), the work equipment is stopped or the vehicle stops moving, and if the distance between the object and the vehicle is equal to or less than a second predetermined distance (β) that is shorter than the first predetermined distance (α), the operating function of a vehicle operating unit equipped with a steering wheel is cut off and only operations are accepted via external communications.

Images