Determination system and determination method

Abstract

To properly judge the quality of a formed product formed in a field by a working device.SOLUTION: The determination system S includes an arithmetic processing unit 20c that determines the forming abnormality of the formed object based on the image of the formed object formed in the field by the working device 2.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a determination system and a determination method for determining molding abnormalities in molded products molded in a field. [Background technology]

[0002] The agricultural vehicle disclosed in Patent Document 1 includes a direction specifying unit that specifies the direction of the ridges, and a travel control unit that controls the vehicle to travel in the direction of the ridges specified by the direction specifying unit. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2021-153567 Summary of the Invention [Problem to be solved by the invention]

[0004] The agricultural work vehicle of Patent Document 1 can identify the direction of a structure (for example, a ridge) formed in a field, but cannot determine whether the structure (for example, a ridge) is abnormally formed.

[0005] The present invention has been made to solve the problems of the conventional technology, and aims to provide an evaluation system and method that can properly determine the quality of molded products formed in a field by a work device. [Means for solving the problem]

[0006] A determination system according to one aspect of the present invention includes a processing unit that determines a molding abnormality of a molded product based on an elevation image of the molded product molded in a field by a working device.

[0007] In a determination method according to one aspect of the present invention, a calculation processing unit determines whether a molding abnormality has occurred in a molded product formed in a field by a working device based on an elevation image of the molded product. [Effects of the Invention]

[0008] According to the present invention, it is possible to properly determine whether a molded product molded in a field by a work device is good or bad. [Brief explanation of the drawings]

[0009] [Figure 1] FIG. 1 is an overall view showing an example of a determination system. [Figure 2] FIG. 2 is a schematic side view showing the working machine. [Figure 3] FIG. 2 is a schematic plan view showing the working machine. [Figure 4] FIG. 2 is a rear perspective view of the position changing device. [Figure 5] FIG. 3 is a diagram illustrating an example of a sensing range of a sensing device provided in a work machine. [Figure 6] FIG. 10 is a diagram illustrating a worked area. [Figure 7] FIG. 2 is a diagram illustrating a planned driving route. [Figure 8] FIG. 2 is a diagram illustrating an example of a flow of a series of processes performed by a calculation processing unit. [Figure 9A] 10 is a flowchart showing an example of a process for determining detection conditions for a molded object. [Figure 9B] 10 is a flowchart illustrating an example of an elevation imaging and abnormality detection process. [Figure 9C] 10 is a flowchart showing an example of a visualization process for a molding and a control process based on molding abnormalities. [Figure 10A] FIG. 10 is a diagram showing an example of an image captured at the start of work. [Figure 10B] FIG. 10 is a diagram showing an example of an image captured when an arbitrary distance away from the work start point. [Figure 11] FIG. 10 is a diagram showing the relationship between all point cloud data obtained by the second sensing device and a region of interest. [Figure 12A] FIG. 10 is a diagram showing that the region of interest moves as the tractor moves, and showing the superposition of two regions of interest where the tractor is in a different position. [Figure 12B]FIG. 10 is a diagram illustrating an example of point cloud data when regions of interest are not superimposed. [Figure 12C] FIG. 10 is a diagram illustrating an example of point cloud data when regions of interest are superimposed. [Figure 13] FIG. 10 is a diagram for explaining how extracted point cloud data is incorporated into an elevation image. [Figure 14] FIG. 10 is a diagram for explaining grid median processing. [Figure 15] FIG. 10 is a diagram for explaining grid interpolation processing. [Figure 16] FIG. 10 is a diagram for explaining grid normalization processing. [Figure 17] FIG. 10 is a diagram showing an example of an elevation image subjected to an abnormality detection process. DETAILED DESCRIPTION OF THE INVENTION

[0010] Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

[0011] FIG. 1 is an overall view showing an example of a determination system S in this embodiment. The determination system S is a device that determines molding abnormalities in a molded object M (e.g., a ridge M1) in a farm field H1. In this embodiment, a work machine 1 is equipped with the determination system S. Note that a server 50, which will be described later, may be equipped with part of the determination system S (e.g., a position estimation unit 20a, an automatic driving control unit 20b, and / or an arithmetic processing unit 20c). The determination system S may also be configured by the work machine 1 and the server 50 working together.

[0012] First, we will explain the work machine 1. The work machine 1 is a vehicle that can perform work while traveling, and in this embodiment is a tractor to which a work device 2 (implement) can be attached to a traveling body 3 (machine body). Note that the work machine 1 is not limited to a tractor, and can be any vehicle that can perform work while traveling. For example, the work machine 1 may be an agricultural machine such as a combine harvester or rice transplanter, or a construction machine such as a compact track loader or backhoe.

[0013] Fig. 2 is a schematic side view of the work machine 1. Fig. 3 is a schematic plan view of the work machine 1. In the description of this embodiment, the direction toward which an operator seated in the driver's seat 10 of the work machine 1 faces (the left side in Figs. 2 and 3) is referred to as the front, and the opposite direction (the right side in Figs. 2 and 3) is referred to as the rear. The left side of the operator (the front side in Fig. 2, the bottom side in Fig. 3) is referred to as the left side, and the right side of the operator (the back side in Fig. 2, the top side in Fig. 3) is referred to as the right side. In addition, the horizontal direction perpendicular to the front-to-rear direction is referred to as the width direction.

[0014] As shown in Figures 2 and 3, the work machine 1 is equipped with a traveling body 3 having a traveling device 7, a prime mover 4, and a transmission 5. The traveling device 7 is driven to provide propulsion force to the traveling body 3. The traveling device 7 is a wheeled traveling device 7 in which the front wheels 7F and the rear wheels 7R are configured as tires. The front wheels 7F and the rear wheels 7R are each provided as a pair, spaced apart in the width direction. As another example, a traveling device 7 in which the front wheels 7F and / or the rear wheels 7R are configured as crawlers may be used. The traveling body 3 is capable of traveling forward and backward by being driven by the traveling device 7.

[0015] A prime mover 4 is built into the front of the traveling vehicle body 3. The prime mover 4 is configured as, for example, a diesel engine. As another example, the prime mover 4 may be configured as another internal combustion engine such as a gasoline engine, an electric motor, or the like.

[0016] The transmission 5 changes the speed of the power output by the prime mover 4 by switching between gear positions, making it possible to switch the propulsive force of the traveling device 7 and change the switching state of the traveling device 7 (switching the traveling device 7 to forward or reverse). The transmission 5 also transmits the power of the prime mover 4 to the PTO shaft 6. The PTO shaft 6 is an output shaft that is connected to the working device 2 and drives the working device 2.

[0017] A protection mechanism 9 for protecting the driver's seat 10 is provided on the upper part of the traveling vehicle body 3. The protection mechanism 9 is, for example, a cabin surrounding the driver's seat 10, and the driver's seat 10 is provided inside the cabin 9A. Note that the protection mechanism 9 is not limited to the cabin 9A, and may be a canopy or a rope erected behind the driver's seat 10.

[0018] The working implement 2 is attached to the traveling body 3. In the tractor of this embodiment, the working implement 2 is detachably attached to the traveling body 3. Specifically, a coupling device 8 to which the working implement 2 can be detachably attached is provided at the front and / or rear of the traveling body 3. In the example shown in Figures 2 and 3, the coupling device 8 is provided at the rear of the traveling body 3. Therefore, the working machine 1 can couple the working implement 2 to the coupling device 8 and tow the coupled working implement 2 by driving the traveling device 7.

[0019] 2 and 3, a position changing device 8A configured with a three-point link mechanism is shown as an example of the coupling device 8. This position changing device 8A is a lifting device that changes the relative position between the traveling body 3 and the working device 2 by raising and lowering the working device 2 relative to the traveling body 3. Below, the position changing device 8A configured with a three-point link mechanism will be described in detail.

[0020] 4 is a perspective view of the position changing device 8A as seen from the rear. The position changing device 8A has a lift arm 8a, a lower link 8b, a top link 8c, a lift rod 8d, and a lift cylinder 8e.

[0021] The front end of the lift arm 8a is supported at the upper rear part of the case (transmission case) that houses the transmission 5 so that it can swing upward or downward. The lift arm 8a swings (lifts and lowers) when driven by a lift cylinder 8e. The lift cylinder 8e is composed of a hydraulic cylinder. As shown in FIG. 1, the lift cylinder 8e is connected to a hydraulic pump via a control valve 34. The control valve 34 is an electromagnetic valve or the like, and extends and retracts the lift cylinder 8e.

[0022] The front end of lower link 8b is supported on the rear lower part of transmission 5 so as to be swingable upward or downward. The front end of top link 8c is supported on the rear part of transmission 5 above lower link 8b so as to be swingable upward or downward. Lift rod 8d connects lift arm 8a and lower link 8b. The rear part of lower link 8b and the rear part of top link 8c are formed in a hook shape.

[0023] When the lift cylinder 8e is driven (extends and retracts), the lift arm 8a moves up and down, and the lower link 8b connected to the lift arm 8a via the lift rod 8d also moves up and down, causing the working device 2 to swing (lift and lower) upward or downward with the front part of the lower link 8b as a fulcrum.

[0024] In the above explanation, the position changing device 8A configured as a three-point link mechanism has been described as an example of the coupling device 8, but the coupling device 8 may be any device that is at least capable of coupling the working device 2 to the traveling body 3. For example, the coupling device 8 may be configured as a swing drawbar or the like that couples the working device 2 to the traveling body 3 without changing the relative positions of the working device 2 and the traveling body 3.

[0025] The work device 2 is a device that performs work on a work field H (e.g., a field H1) or a work object in the work field H (e.g., crops planted in the field H1, etc.). The work device 2 is a tillage device that performs tillage work, a ridge forming device that forms ridges, a furrow cutting device that cuts furrows, a harvesting device that harvests crops, a reaping device that cuts grass etc., a spreading device that spreads grass etc., a grass collecting device that collects grass etc., a shaping device that shapes grass etc., a fertilizer spreading device that spreads fertilizer, a pesticide spreading device that sprays pesticides, a separating device that separates crops, etc. In this embodiment, the work device 2 is a ridge forming device that forms ridges in the field H1.

[0026] Although the above description is of a case where the work machine 1 is a tractor and the work implement 2 is coupled to the coupling device 8, the work implement 2 is not limited to an implement coupled to the traveling body 3 by the coupling device 8. For example, the work implement 2 may be a front loader attached to the front of the traveling body 3.

[0027] Furthermore, the working device 2 may be any device that is provided on the working machine 1 and performs work at the work site H, and does not have to be a device that can be attached to and detached from the traveling body 3 like an implement. For example, if the working machine 1 is a combine harvester, the working device 2 includes a harvesting device that harvests crops. If the working machine 1 is a rice transplanter, the working device 2 includes a planting device that plants seedlings. If the working machine 1 is a backhoe or compact track loader, the working device 2 can be an attachment that can be attached to the position changing device 8A (such as an arm or boom).

[0028] 1, the work machine 1 is equipped with a steering device 11. The steering device 11 has a handle 11a (steering wheel), a steering shaft 11b (rotating shaft) that rotates in conjunction with the rotation of the handle 11a, and an assist mechanism 11c (power steering mechanism) that assists in steering the handle 11a.

[0029] The assist mechanism 11c includes a control valve 35 and a steering cylinder 32. The control valve 35 is, for example, a three-position switching valve that can be switched by moving a spool or the like. The control valve 35 can also be switched by steering the steering shaft 11b. The steering cylinder 32 is connected to an arm 36 (knuckle arm) that changes the direction of the front wheels 7F. Therefore, by rotating the steering wheel 11a, the switching position and opening degree of the control valve 35 are switched in response to the operation, and the steering cylinder 32 extends or retracts to the left or right depending on the switching position and opening degree of the control valve 35, making it possible to change the steering direction of the front wheels 7F.

[0030] The above-described steering device 11 is an example and is not limited to the above-described configuration. For example, if the traveling device 7 can change the rudder angle by differentiating the propulsive force in one direction and the propulsive force in the other direction in the width direction, the traveling device 7 may also be configured to function as the steering device 11.

[0031] As shown in FIG. 1, the work implement 1 is equipped with a control device 20. The control device 20 includes one or more processors. The control device 20 is a controller for the work implement 1 and performs various controls related to the work implement 1. The control device 20 is communicably connected to each device and apparatus mounted on the work implement 1 via an in-vehicle network such as CAN, ISOBUS, LIN, or FlexRay. For example, the control device 20 performs control processing (operations) of the work implement 2, prime mover 4, transmission 5, position change device 8A, steering device 11, etc., based on a signal (operation signal) input from an operation device.

[0032] The control device 20 includes one or more memories, various analog circuits, various digital circuits, etc. The one or more memories store (memorize) software programs and various data to be executed by one or more processors. The control device 20 can read software programs from one or more memories using one or more processors and execute various processes based on the software programs. Note that the control device 20 may also be able to execute various processes based on predetermined logic circuits using one or more processors.

[0033] The processor may be, for example, a central processing unit (CPU), a graphics processing unit (GPU), a digital signal processor (DSP), a field programmable gate array (FPGA), or an application specific integrated circuit (ASIC).

[0034] The control device 20 may execute various processes by having multiple physically separated processors cooperate with each other, and the configuration is not limited to the configuration described above. In such a case, the multiple processors are mounted on one or more computers that are physically separated from the work machine 1, and these processors are connected to each other so as to be able to communicate with each other via a network such as an in-vehicle network, a LAN, a WAN, or the Internet.

[0035] In addition, the software program may be stored in a storage device 21 (non-volatile memory such as HDD, SSD, CD-ROM, DVD-ROM, etc.) communicatively connected to the control device 20, or in an external server 50 connected via the network, and installed from there into the memory.

[0036] As shown in FIG. 1, the work machine 1 is equipped with one or more sensing devices 25. The sensing devices 25 sense the surroundings of the work machine 1. Specifically, the sensing devices 25 perform sensing by measuring the distance to the environment (surrounding objects) surrounding the work machine 1. The sensing devices 25 are distance measurement sensors that measure the distance to at least a portion of the surroundings of the work machine 1. The sensing devices 25 can measure the distance to at least a portion of the surroundings of the work machine 1 and detect point cloud data of the environment surrounding the work machine 1.

[0037] The sensing device 25 is connected to the control device 20 via wire or wireless communication so as to be able to communicate with the control device 20, and outputs sensing results to the control device 20. The sensing device 25 includes an optical distance measuring sensor, a signal processing circuit, etc. The optical distance measuring sensor of the sensing device 25 can be, for example, a LiDAR (Light Detection And Ranging).

[0038] A LIDAR (laser sensor) emits pulsed measurement light (laser light) millions of times per second from a light source such as a laser diode, and reflects the measurement light with a rotating mirror to scan horizontally or vertically and project it over a predetermined detection range (sensing range, e.g., 360°). The LIDAR then receives the measurement light reflected by an object with a light-receiving element. A signal processing circuit detects the distance to the object based on the time between when the LIDAR emits the measurement light and when it receives the reflected light (ToF (Time of Flight) method).

[0039] In addition to LIDAR, a ToF camera can be exemplified as the optical distance measuring sensor of the sensing device 25. In the above example, the sensing device 25 has an optical distance measuring sensor, but instead of the optical distance measuring sensor, an acoustic distance measuring sensor (for example, an airborne ultrasonic sensor such as a sonar) may be used.

[0040] FIG. 5 is a diagram showing an example of a sensing range Es of a sensing device 25 provided on the work machine 1. One or more sensing devices 25 are provided on the work machine 1, and the sensing range Es is sensed by the one or more sensing devices 25. The sensing range Es includes at least the worked area Ea where the work machine 1 (work machine 2) has performed work. The sensing device 25 also senses a position estimation range Eb, which is a range necessary to estimate the position of the work machine 1 on which the sensing device 25 is provided. An example of the position estimation range Eb is the range in the direction of travel of the work machine 1.

[0041] Note that Figure 5 is merely for the purpose of explaining the sensing range Es, and the sensing range Es, worked area Ea, and position estimation range Eb are not limited to the examples shown in Figure 5. The distance that the sensing device 25 can sense also varies depending on the distance measuring sensor adopted as the sensing device 25.

[0042] The work implement 1 performs work while traveling. For this reason, a work execution range Ea1, which is the area in which the work implement 1 can perform work (the area in which the work implement 2 of the work implement 1 performs work at a predetermined position), moves as the work implement 1 travels. The work execution range Ea1 is the range in which the work implement 2 performs work at a predetermined position, in other words, at a predetermined time, when the work implement 1 performs work while traveling. In other words, the work execution range Ea1 means the range in which the work implement 1 at a predetermined position (time) acts on an object (the field H1, crops planted in the field H1, weeds in the field H1, etc.).

[0043] Therefore, as the work implement 1 moves, the range to which the work execution range Ea1 moves can be said to be formed as a worked area Ea in the work site H. Furthermore, the work execution range Ea1 corresponding to the current position of the work implement 1 is included as part of the worked area Ea. In other words, the work execution range Ea1 does not mean the entire range in which the work implement 1 performs a series of tasks in the work site H (field H1).

[0044] In the example shown in FIG. 5, for convenience of explanation, the work execution range Ea1 is shown as a substantially rectangular range in a plan view, but this is not limited thereto. The work execution range Ea1 varies depending on the type of work implement 2 and the work content, and may be substantially circular or irregularly shaped. The work execution range Ea1 also varies depending on the work target (the location where work is performed) of the work implement 2. That is, when the work implement 2 performs work on the ground of the field H1, such as with a tilling implement, ridge-making implement, or harvesting implement, the work execution range Ea1 is the ground of the field H1, and therefore the worked area Ea is the part of the ground of the field H1 where the work implement 2 has performed work. Furthermore, when the work implement 2 performs work on areas other than the ground of the field H1, such as with a fruit harvesting implement, the work execution range Ea1 is fruit trees and the like other than the ground, and therefore the worked area Ea is the part of these work targets where the work implement 2 has performed work. In this embodiment, the working device 2 is a ridge-forming device, and therefore the worked area Ea includes the ridges M1 formed by the ridge-forming device.

[0045] 6 is a diagram illustrating the completed work area Ea. Here, regardless of the arrangement of the work device 2 relative to the work machine 1, when the work machine 1 performs work while traveling in a predetermined direction of travel, the work machine 1 moves away from the location where work has already been performed as it travels in the direction of travel. Therefore, when the work machine 1 performs work at a predetermined first position P1 and then moves from the first position P1 in the direction of travel, the sensing device 25 attached to the work machine 1 moves away from the work execution range Ea1 at the first position P1.

[0046] At this time, as viewed from the sensing device 25, the work machine 1 and / or work device 2 passes through at least a portion of the work execution range Ea1 (t=1) at the first position P1, and at a predetermined second position P2 on the traveling direction side of the first position P1, at least a portion of the work execution range Ea1 (t=1) appears from the work machine 1 and / or work device 2. The example shown in FIG. 6 shows a state in which, when the work machine 1 moves to the second position P2 on the traveling direction side (front side), the work execution range Ea1 (t=1) appears behind the work device 2 as viewed from the sensing device 25. For this reason, it is preferable that one or more sensing devices 25 be able to sense a range that includes the opposite side of the traveling direction as the sensing range Es.

[0047] 5 shows a case where the work implement 2 is attached to the rear of the traveling body 3 and the sensing device 25 senses the area behind the work machine 1 and the work implement 2, but the range of the sensing range Es that includes the side opposite the direction of travel is not limited to the rear of the work implement 1 and the work implement 2. For example, if the work implement 2 is attached offset in the width direction from the traveling body 3, in other words, if the work execution range Ea1 is offset in the width direction from the traveling body 3, the range of the sensing range Es that includes the side opposite the direction of travel will include the work execution range Ea1 that is offset in the width direction from the traveling body 3.

[0048] In this embodiment, the direction of travel of the work implement 1 is either forward or backward. Therefore, the sensing device 25 can sense an area around the work implement 1 that includes at least the front and rear of the work implement 1. In the example shown in FIGS. 2 and 3, two sensing devices 25 are provided on the work implement 1, one sensing device 25 (first sensing device 25a) senses the front, and the other sensing device 25 (second sensing device 25b) senses the rear. For example, the first sensing device 25a is provided in the front part of the roof 9a of the cabin 9A. The second sensing device 25b is provided in the rear part of the roof 9a.

[0049] The first sensing device 25a masks an area for detecting devices and equipment, such as the cabin 9A including the roof 9a, provided on the work machine 1. For this reason, the first sensing device 25a senses a range (for example, 180°) substantially in front of the work machine 1, and detects point cloud data of the sensing range Es.

[0050] The second sensing device 25b masks an area in which to detect devices and equipment provided on the work implement 1, such as the cabin 9A including the roof 9a. At this time, the second sensing device 25b may acquire the position of the work implement 2 connected to the position change device 8A and mask the area in which to detect the work implement 2. For this reason, the second sensing device 25b senses a range (for example, 180°) approximately behind the work implement 1 and detects point cloud data of the sensing range Es.

[0051] With the above configuration, in this embodiment, the first sensing device 25a and the second sensing device 25b can sense the area around the work machine 1 in approximately 360°. One or more sensing devices 25 may be provided on the work machine 1, and the area around the work machine 1 may be sensed by one or more sensing devices 25. The sensing range Es is not limited to approximately 360° around the work machine 1. The mounting position of the sensing device 25 is not limited to the above-described position; for example, the sensing device 25 may be mounted in front of or on top of the hood. In FIG. 5, the sensing range Es may include blind spots and is approximately 360° around the work machine 1, but is not limited thereto. In this embodiment, the work machine 2 is a ridge-forming device. Therefore, the sensing range Es is sufficient as long as it is at least a range in which the molded object M (e.g., ridge M1) can be detected. Here, the sensing range Es is the range on the side of the work implement 1 where the work tool 2 is placed, and is, for example, approximately 180° behind the periphery of the work implement 1, but it may also be 90° or the like and is not limited to these numerical values. The second sensing device 25b acquires point cloud data that indicates the topography of at least the periphery of the work implement 2. More specifically, the second sensing device 25b acquires point cloud data that indicates the topography of the field H1 including the ridges M1 formed by the work implement 2 (ridge-forming device).

[0052] As shown in FIG. 1, the work implement 1 is equipped with an imaging device 26. The imaging device 26 may be a CCD camera equipped with a CCD (Charge Coupled Device) image sensor, a CMOS camera equipped with a CMOS (Complementary Metal Oxide Semiconductor) image sensor, or the like. The imaging device 26 is provided at the rear of the roof 9a. The imaging device 26 captures an image behind the work implement 1, and the captured image includes ridges M1 formed in the field H1 by the work implement 2 (ridge-forming device). The second sensing device 25b and the imaging device 26 are arranged closely adjacent to each other, either vertically or horizontally, at the rear of the roof 9a. Therefore, the point cloud data of the sensing range Es captured by the second sensing device 25b and the captured image captured by the imaging device 26 are from approximately the same measurement point (viewpoint).

[0053] Furthermore, when a rope is provided as the protection mechanism 9, a single sensing device 25 may be provided on top of the rope. Alternatively, a sensing device 25 may be provided on mounting structures extending outward in the width direction of the traveling body 3 at the front and rear of the traveling body 3, with a pair of sensing devices 25 at each of the front and rear of the traveling body 3, positioned at a distance outward in the width direction from the traveling body 3. Furthermore, one or more sensing devices 25 may be provided on the working device 2 that is detachable from the traveling body 3. Furthermore, the second sensing device 25b and the imaging device 26 are arranged adjacent to each other vertically or horizontally above the rope.

[0054] As shown in Fig. 1, the determination system S for the work machine 1 includes a position estimation unit 20a that estimates the position of the work machine 1 based on the sensing results of the sensing device 25. The position estimation unit 20a is, for example, a software program implemented in the control device 20. As another example, if the work machine 1 is connected to an information processing device such as an external server 50 so that it can communicate directly or indirectly with the work machine 1, the position estimation unit 20a may be provided in the server 50 or the like external to the work machine 1. In the following explanation, an example will be described in which the position estimation unit 20a is provided in the control device 20 (work machine 1), and detailed explanations of other examples will be omitted.

[0055] The position estimation unit 20a estimates the position of the work machine 1 based on the sensing results of the sensing device 25 and environmental map information. The position estimation unit 20a estimates the position based on the sensing results of the sensing device 25 (ranging signals obtained from the ranging sensor), environmental map information, and a SLAM (Simultaneous Localization and Mapping) algorithm.

[0056] The environmental map information is map information that shows objects in the environment around the work field H, including the work field H where the work machine 1 performs work, and is generated from point cloud data. Taking an example where the work field H is a field H1 and the environmental map information shows the environment around the field H1, including the field H1, the environmental map information shows the ground around the field H1, the crops planted in the field H1, the ridges M1 formed in the field H1, the ridges around the field H1, the fences around the field H1, the weeds on the ground around the field H1, the barns around the field H1, etc. as a three-dimensional point cloud. The environmental map information is generated in advance based on the sensing results of the sensing device 25 and stored in the storage device 21. Note that the environmental map information stored in the storage device 21 may be generated based on the sensing results of the sensing device 25 of another work machine 1, etc.

[0057] In estimating the position of the work machine 1, the position estimation unit 20a acquires point cloud data (detected point cloud data) from the sensing results of the sensing device 25 of the work machine 1 and aligns (matches) the acquired detected point cloud data with the point cloud data of the environmental map information, thereby estimating the position of the work machine 1. The position estimation unit 20a estimates a predetermined position of the work machine 1 as the position estimation of the work machine 1.

[0058] Furthermore, the position estimation unit 20a may estimate (position estimate) the position (estimated position EP) of the work machine 1 (traveling body 3) based on its own position detected by a position detection device 27 attached to the work machine 1 using a satellite positioning system (positioning satellite) such as D-GPS, GPS, GLONASS, Beidou, Galileo, or Michibiki, that is, the position (e.g., latitude, longitude) of the GPS antenna. In this case, the own position (e.g., latitude, longitude) detected by the position detection device 27 may be used, and the sensing results of the sensing device 25 and environmental map information may not be used.

[0059] 1, the control device 20 has an automatic driving control unit 20b. The automatic driving control unit 20b is composed of electric and electronic circuits, a CPU, and programs stored in a memory, which are provided in the control device 20.

[0060] The automatic driving control unit 20b controls the automatic driving of the work machine 1 (hereinafter referred to as automatic driving control). The automatic driving control unit 20b can execute line-type automatic driving control and / or autonomous-type automatic driving control. To explain automatic driving using line-type automatic driving control as an example, the automatic driving control unit 20b controls the devices and apparatuses provided in the work machine 1 based on the estimated position EP and a predefined planned driving route L so that the traveling vehicle body 3 travels along the planned driving route L. For example, as automatic driving control, the automatic driving control unit 20b controls the steering angle and driving speed (vehicle speed) of the traveling vehicle body 3.

[0061] The planned travel route L may be stored in advance in the storage device 21, or may be created (defined) based on an estimated position EP estimated by the position estimation unit 20a when the work machine 1 actually travels. The planned travel route L may also be created based on information input via an input interface.

[0062] The input interface is, for example, a display device 15 that is provided in the work machine 1 and that allows input operations. The display device 15 has, in addition to a display screen that displays an image, for example, a touchpad or hardware switches. The input interface is sufficient as long as it allows at least the input operation of information and the input information can be acquired by the control device 20, and may be an operable terminal such as a smartphone that is communicatively connected to the control device 20. The input interface may also be a communication device that can communicate with an external server 50 or the like, and the communication device may receive the planned travel route L managed by the external server 50 or the like.

[0063] During automatic driving control, the automatic driving control unit 20b controls the steering angle so that the positional deviation between the estimated position EP and the planned traveling route L is less than a threshold value. In other words, when the positional deviation between the estimated position EP and the planned traveling route L is less than the threshold value, the automatic driving control unit 20b controls the control valve 35 of the steering device 11 to maintain the steering angle. On the other hand, when the positional deviation between the estimated position EP and the planned traveling route L is equal to or greater than the threshold value, the automatic driving control unit 20b controls the control valve 35 of the steering device 11 to change the steering angle in a direction that reduces the positional deviation.

[0064] Further, regarding the automatic driving control when the work implement 1 works in the field H1, the automatic driving control unit 20b performs automatic driving control, for example, so that the work implement 1 travels back and forth between one end and the other end of the work field H (field H1). Fig. 7 is a diagram illustrating a planned travel route L. As shown in Fig. 7, the planned travel route L in the field H1 includes a straight section L1 that travels from one end of the field H1 to the other, and a turning section L2 that connects one straight section L1 with the other straight section L1.

[0065] The automatic driving control unit 20b may control the work device 2, the position change device 8A, etc., depending on the position of the work implement 1 on the planned travel route L, etc., and control the work performed by the work device 2. The automatic driving control unit 20b can control the execution and stopping of work by the work device 2. The automatic driving control unit 20b can control the drive of the position change device 8A (lifting device) and the PTO shaft 6, and switch between a working state in which the work device 2 performs work and a non-working state in which the work device 2 does not perform work.

[0066] Taking the example of the working device 2 being a tilling device or a ridge-making device that is towed by the working machine 1 and performs work while in contact with or embedded in the ground, the automatic driving control unit 20b can use the position change device 8A to lower the working device 2 to the ground to switch it to a working state, or use the position change device 8A to raise the working device 2 from the ground to switch it to a non-working state.

[0067] Furthermore, when the working device 2 is a working device 2 that is driven by power transmitted from the PTO shaft 6 or by a built-in actuator (e.g., an electric actuator), such as a rotary tiller or a molding device, the automatic driving control unit 20b can switch between a working state and a non-working state by controlling these power sources (PTO shaft 6, actuator, etc.).

[0068] For example, the automatic driving control unit 20b switches to the working state when the estimated position EP is located on the straight section L1, and switches to the non-working state when the estimated position EP is located on the turning section L2.

[0069] The automatic driving control unit 20b may switch between the working state and the non-working state according to the areas defined in the field map, regardless of the position of the estimated position EP on the planned travel route L. For example, the area where work is performed (working area Ha) is defined as the area inside the headland of the field H1. Furthermore, the area where work is not performed (non-working area Hb) is defined as the headland, entrances and exits of the field H1, and places where work has already been performed. Note that the above-mentioned working area Ha and non-working area Hb are merely examples, and the working area Ha may, for example, include the headland.

[0070] In the above-described embodiment, automatic driving has been explained using line-type automatic driving control as an example, but in autonomous automatic driving control, the automatic driving control unit 20b controls each device and apparatus provided in the work machine 1 to perform work within the field H1 based on the estimated position and sensing results, regardless of the planned driving route L.

[0071] The work machine 1 may also employ a display device 15 that displays the current position of the work machine 1 on a field map based on the estimated position EP estimated by the position estimation unit 20a and a field map showing the field H1. The display device 15 may be a display placed near the driver's seat 10 of the work machine 1, or may be a mobile terminal carried by the worker or a manager's terminal that monitors the work of the work machine 1. Examples of mobile terminals and manager's terminals include terminals such as smartphones (multi-function mobile phones), tablets, and PDAs, as well as fixed computers such as personal computers.

[0072] The work machine 1 has a communication device 29. The communication device 29 is a communication module that performs either direct communication or indirect communication with the server 50, and can perform wireless communication using, for example, the IEEE802.11 series of communication standards, such as Wi-Fi (Wireless Fidelity, registered trademark), BLE (Bluetooth (registered trademark) Low Energy), LPWA (Low Power, Wide Area), and LPWAN (Low-Power Wide-Area Network). The communication device 29 can also perform wireless communication using, for example, a mobile phone communication network or a data communication network.

[0073] The server 50 has a communication device 51 and a storage device 52. Similar to the communication device 29, the communication device 51 is a communication module that performs either direct communication or indirect communication with the work machine 1. The communication device 51 can also perform wireless communication via, for example, a mobile phone communication network or a data communication network.

[0074] As shown in FIG. 1, the judgment system S of the work implement 1 includes a calculation processing unit 20c. The calculation processing unit 20c judges whether or not there is a molding abnormality in the formed object M (e.g., ridge M1) formed in the field H1 by the work implement 2 based on an image C (e.g., an elevation image C) of the formed object M (e.g., ridge M1). For example, the control device 20 functions as the calculation processing unit 20c when a processor of the control device 20 executes a judgment program. The calculation processing by the calculation processing unit 20c and the functioning of the control device 20 are synonymous. The calculation processing unit 20c is, for example, a software program implemented in the control device 20. Furthermore, if the work implement 1 is directly or indirectly connected to an external server 50 or the like so as to be able to communicate with the external server 50, the calculation processing unit 20c may be provided in the server 50 or the like external to the work implement 1. In the following explanation, a case in which the calculation processing unit 20c is provided in the control device 20 (work implement 1) will be described as an example, and detailed explanations of other examples will be omitted.

[0075] 8 is a diagram illustrating an example of the flow of a series of processes performed by the calculation processing unit 20c. This series of processes includes acquisition of point cloud data, elevation imaging, determination of malformation, and visualization processing (see FIG. 8), as well as control processing based on the malformation (see the control processing in FIG. 9C described later).

[0076] First, when the detection conditions for the molded product M are met, the processing unit 20c acquires point cloud data including the molded product M in the field H1. Specifically, the processing unit 20c acquires point cloud data within the region of interest K along with the position information of the work machine 1. This corresponds to "point cloud + GNSS" in FIG. 8. The processing unit 20c then generates an elevation image C including the molded product M from the acquired point cloud data. This corresponds to "accumulating point cloud in world coordinates" and "elevation imaging" in FIG. 8. The processing unit 20c then determines whether the molded product M has a molding abnormality based on the elevation image C. This corresponds to "applying PatchCore" and "elevation image + abnormality score" in FIG. 8. The processing unit 20c then performs visualization processing of the molded product M determined to have a molding abnormality (see the visualization processing in FIG. 8). The processing unit 20c also performs control processing based on the molding abnormality (see the control in FIG. 9C, which will be described later). The visualization process of the molded object M and the control process based on the molding abnormality may be performed simultaneously in parallel, or may be performed serially in order. The calculation processing unit 20c may perform the visualization process of the molded object M and then the control process based on the molding abnormality, or may perform the processes in the reverse order. Furthermore, the image C generated by the calculation processing unit 20c may be a color image or a near-infrared image including the molded object M, in addition to the elevation image C.

[0077] The series of processes shown in Fig. 8 by the arithmetic processing unit 20c will be described in more detail with reference to Figs. 9A to 9C. Fig. 9A is a flowchart showing an example of a process for determining the detection conditions for the molded object M. That is, the process for determining the detection conditions for the molded object M shown in Fig. 9A is a process that serves as a starting point for a series of processes by the arithmetic processing unit 20c.

[0078] The calculation processing unit 20c determines that the detection condition is met when the work by the work device 2 is started and the molded object M (for example, the ridge M1) is included in the region of interest K (ROI).

[0079] Specifically, as shown in Fig. 9A, the arithmetic processing unit 20c determines whether the working device 2 is working (S11). Based on a signal from the CAN of the in-vehicle network, the arithmetic processing unit 20c (control device 20) determines that the working device 2 is working if the rotation speed of the PTO (power take-off) (the rotation speed of the PTO shaft 6) is equal to or greater than a specified rotation speed and the detection value of the sensor of the lift arm 8a is equal to or greater than a threshold value (for example, a value indicating that the working device 2 is in a working position (a lowered position for ridge formation)). (S11: Yes) Note that the arithmetic processing unit 20c may determine that the working device 2 is working based on only one of the rotation speed of the PTO shaft 6 or the sensor of the lift arm 8a. Alternatively, the arithmetic processing unit 20c may determine that the working device 2 is working if the detection value of the sensor of the lift arm 8a is equal to or greater than a threshold value (for example, a value indicating that the working device 2 is in a working position (a lowered position for ridge formation)) and the working machine 1 is traveling (for example, if the vehicle speed of the traveling vehicle body 3 is equal to or greater than a predetermined speed). Alternatively, the determination may be made by other methods. The specified rotation speed and threshold value differ depending on the types of the work machine 1 and the work device 2. If the work device 2 is not in operation (S11: No), the calculation processing unit 20c returns to S11.

[0080] If the working device 2 is working (S11: Yes), the calculation processing unit 20c determines whether or not a molding M (e.g., ridge M1) is included in a region of interest K (ROI) of all point cloud data acquired by the second sensing device 25b (S12). If the region of interest K (ROI) includes the molding M (e.g., ridge M1), the calculation processing unit 20c determines that the detection condition is met (S12: Yes). If the region of interest K (ROI) does not include the molding M (e.g., ridge M1) (S12: No), the calculation processing unit 20c returns to S11. Alternatively, the calculation processing unit 20c may determine that the detection condition is met if the working machine 1 has moved a predetermined distance since the start of work by the working device 2 (S12: Yes). Specifically, when the working device 2 is working (S11: Yes) and the working machine 1 has moved a predetermined distance since the working device 2 started working, the calculation processing unit 20c determines that the region of interest K (ROI) contains the molded object M (e.g., ridge M1), that is, that the detection condition is met (S12: Yes). The predetermined distance is, for example, one to several meters, but may be any other value. When the working machine 1 has not moved the predetermined distance (S12: No), the calculation processing unit 20c returns to S11.

[0081] In addition, when determining whether or not the ridge M1 is included in the point cloud data within the region of interest K, in other words, the extracted point cloud data extracted from the region of interest K, the calculation processing unit 20c can determine whether or not the ridge M1 is included, for example, by performing a matching process between the extracted point cloud data and a point cloud pattern indicating the ridge M1.

[0082] FIG. 10A is a diagram showing an example of an image captured at the start of work. When the work device 2 starts work, no ridges M1 have been formed in the field H1. Therefore, as shown in FIG. 10A, the imaging device 26 captures an image showing the field H1 where no ridges M1 have been formed. No ridges M1 exist in the region of interest K. FIG. 10B is a diagram showing an example of an image captured at an arbitrary distance from the work start point. The work machine 1 has advanced a predetermined distance since the start of work by the work device 2, so ridges M1 have been formed in the field H1. As shown in FIG. 10B, the imaging device 26 captures an image showing the field H1 where, for example, trapezoidal ridges M1 have been formed. Note that the ridges M1 are not limited to a trapezoidal shape and may be semicircular, for example. A ridge M1 exists in the region of interest K.

[0083] The second sensing device 25b acquires point cloud data including the molded object M in the field H1. FIG. 11 is a diagram showing the relationship between all point cloud data acquired by the second sensing device 25b and the region of interest K. For all sensing ranges ES shown in FIG. 11, the amount of point cloud data is very large (e.g., 450,000 points / second), so processing all of the point cloud data takes a long calculation time, raising concerns that real-time performance may be impaired. Therefore, in order to exclude unnecessary point cloud data, the calculation processing unit 20c sets a region of interest K in a range including the ridge M1 and extracts the point cloud data for the region of interest K. As a result, the amount of point cloud data for the region of interest K is reduced to approximately 1 / 2 to 1 / 3 of the original amount of all point cloud data, but it may be reduced further.

[0084] The arithmetic processing unit 20c performs image processing on the extracted point cloud data extracted from the region of interest K of the point cloud data to generate an elevation image C. FIG. 9B is a flowchart showing an example of the elevation imaging and abnormality detection processing. Specifically, when the detection condition for the molding M is met (S12: Yes), the arithmetic processing unit 20c starts the image processing shown in FIG. 9B and extracts only the point cloud data of the region of interest K (S21). In other words, only the point cloud of the ROI is extracted from the point cloud of the LiDAR. Note that the image processing may include predetermined processing on the extracted point cloud data, or predetermined processing on the extracted point cloud data may be performed before or after the image processing.

[0085] The calculation processing unit 20c stores the position information indicating the position estimated by the position estimation unit 20a in association with the extracted point cloud data of the region of interest K. The calculation processing unit 20c performs coordinate conversion of the extracted point cloud data of the region of interest K from the coordinate system of the sensing device 25 to the world coordinate system based on the position information for the extracted point cloud data of the plurality of regions of interest K that are located at different positions on the traveling vehicle body 3, thereby superimposing the extracted point cloud data of the plurality of regions of interest K on the world coordinate system (S22).

[0086] FIG. 12A shows the movement of the region of interest K as the tractor moves, and illustrates the superposition of two regions of interest K where the tractor is positioned at different locations. As shown on the left side of FIG. 12A, the position of the region of interest K moves as the work machine 1 moves (travels straight ahead). As shown on the right side of FIG. 12A, the region of interest K of the work machine 1 located at a certain time (region K2 shown by a solid line) partially overlaps with the region of interest K of the work machine 1 located, for example, 0.1 seconds earlier (region K1 shown by a dashed line) (the overlapping portion is indicated by hatching). FIG. 12B shows an example of point cloud data without the superposition of the region of interest K. In the point cloud data without the superposition of the region of interest K shown in FIG. 12B, i.e., the point cloud data of the region of interest K at the above-mentioned time or 0.1 seconds earlier, there are many horizontal streaks and the shape of the ridge M1 is blurred. FIG. 12C shows an example of point cloud data with the superposition of the region of interest K. In the point cloud data when the region of interest K is superimposed as shown in Fig. 12C, the horizontal streaks are reduced and the shape of the ridge M1 appears. That is, the superimposition of the region of interest K shown in Fig. 12C is performed in S22.

[0087] The arithmetic processing unit 20c generates an elevation image C by projecting the extracted point cloud data superimposed in the world coordinate system onto a predetermined plane of the world coordinate system (S23 to S25). Specifically, the arithmetic processing unit 20c sequentially performs imaging processing using a median process of the in-grid point cloud (S23), a grid interpolation process (S24), and a grid normalization process (S25).

[0088] FIG. 13 is a diagram for explaining how extracted point cloud data is converted into an elevation image C. The arithmetic processing unit 20c divides the point cloud stored in the world coordinate system into square grids G on a predetermined plane (xy plane) shown on the left side of FIG. 13. These grids G correspond to pixels of the image. The grids G are, for example, 1 cm long and 1 cm wide, but are not limited to these dimensions and may be rectangular. The arithmetic processing unit 20c then uses the median z value when there are multiple points (point cloud data) for one grid G ​​(S23). The arithmetic processing unit 20c then applies interpolation to grids G that do not have any points (point cloud data) (S24). Once there is one z (elevation value) for one grid G, the arithmetic processing unit 20c then normalizes the z value to an integer between 0 and 255 for conversion into an image (S25). The arithmetic processing unit 20c then performs the processes of S23 to S25 on all remaining grids G. By performing the processes of S23 to S25, the point cloud data shown on the left side of FIG. 13 is visualized as a monochrome elevation image C shown on the right side of FIG.

[0089] FIG. 14 is a diagram for explaining median processing of grid G. On the right side of FIG. 14, voxels are shown in which one grid G ​​is enlarged. Voxels like these result from division by grid G ​​on the xy plane of three-dimensional space. As shown in FIG. 14, the arithmetic processing unit 20c determines the median value of the z values ​​of the points (point cloud data) within a voxel as the value of that voxel (grid G) (S23). Note that the arithmetic processing unit 20c may also determine the average value of the z values ​​of the points (point cloud data) within a voxel as the value of that voxel (grid G) (S23).

[0090] FIG. 15 is a diagram illustrating the interpolation process for a grid G. The left side of FIG. 15 illustrates a case where no point (point cloud data) exists within a voxel (grid G). The arithmetic processing unit 20c performs, for example, interpolation using an average value (S24). That is, as shown on the right side of FIG. 15, the arithmetic processing unit 20c interpolates using the average value of a 5×5 range centered on the missing grid G1. Note that the 5×5 range may be a 3×3, 7×7, or other range. The arithmetic processing unit 20c may also perform linear interpolation (S24). That is, the arithmetic processing unit 20c interpolates using the average value of the four grids G above, below, left, and right of the missing value. Note that if all four grids G above, below, left, and right of the missing value are missing, the arithmetic processing unit 20c performs interpolation using the average value because linear interpolation cannot accurately perform the interpolation. Note that, in addition to average value interpolation and linear interpolation, spline interpolation and nearest neighbor interpolation may also be used for interpolation.

[0091] 16 is a diagram for explaining the normalization process of the grid G. As shown in Fig. 16, the calculation processing unit 20c normalizes the pixel values ​​of the grid G ​​to 0 to 255 in accordance with the pixel values ​​of the grid G ​​(S25). That is, the calculation processing unit 20c normalizes the pixel values ​​of the grid G ​​to integers from 0 to 255.

[0092] As shown in FIG. 9B, the arithmetic processing unit 20c performs an abnormality detection process on the elevation image C (S26). As the abnormality detection process, the arithmetic processing unit 20c performs a process of calculating the degree of deviation of the elevation image C from a normal image C1. The normal image C1 is an image that shows the shape of a normal molding. Furthermore, as the abnormality detection process, the arithmetic processing unit 20c performs a process of calculating an abnormal region that deviates from the normal image C1 on the elevation image C, and generates an elevation image C with the abnormal region added.

[0093] For example, anomaly detection AI (artificial intelligence) can be used for the anomaly detection process. Here, the anomaly detection AI used is, for example, PatchCore shown in Figure 8. PatchCore is a method for detecting abnormal parts in an image using a feature map of a pre-trained model.

[0094] FIG. 17 is a diagram showing an example of anomaly detection processing of elevation image C. As shown in FIG. 17, an anomaly detection AI (e.g., PatchCore) learns from a normal image C1, which represents an elevation image C of a normal (clean) ridge M1, and extracts features of the normal pattern. The anomaly detection AI compares the features of the test image of ridge M1 with the features of the normal pattern and calculates the deviation of the test image of ridge M1 from the features of the normal pattern. The test image shown in FIG. 17 is the elevation image C generated in S25 of FIGS. 8 and 9B. In this way, the anomaly detection AI outputs an anomaly score and anomaly segmentation AS (anomaly distribution) results according to the deviation from the features of the normal pattern. The anomaly score is proportional to the deviation. Furthermore, the anomaly segmentation AS (anomaly distribution) indicates the degree (classification) of anomaly by classifying the deviation into multiple categories. FIG. 17 shows an elevation image C on which anomaly segmentation AS is displayed.

[0095] The anomaly detection AI is not limited to PatchCore, but may be any of EfficientAD, PUAD (Picturable and Unpicturable Anomaly Detection), SLSG (Self-supervised Learning and Selfattentive Graph convolution), ReConPatch, DDAD (Denoising Diffusion Anomaly Detection), and PaDiM.The anomaly detection process is not limited to the anomaly detection AI, and various types of anomaly detection process such as image recognition process and image pattern matching process can be used.

[0096] The calculation processing unit 20c determines whether the molding M has an abnormality based on the processing result (S26) obtained by performing an abnormality detection processing on the elevation image C. The calculation processing unit 20c determines whether the molding M has an abnormality when the deviation degree, which is the processing result, exceeds a threshold. FIG. 9C is a flowchart showing an example of the visualization processing of the molding M and the control processing based on the abnormality. For example, as shown in FIG. 9C, the calculation processing unit 20c determines whether the molding M has an abnormality when the abnormality score corresponding to the deviation degree, which is the processing result, exceeds a threshold (S31 to S35). The thresholds may include a first threshold (e.g., 1.5) and a second threshold (e.g., 2.0) that is greater than the first threshold. The thresholds (first threshold and second threshold) may be numerical values ​​other than those described above. There may be only one threshold, or three or more thresholds. The thresholds differ depending on the type of the working device 2. That is, the thresholds may have different values ​​depending on the type of the working device 2.

[0097] As shown in Fig. 9C, if the deviation (abnormality score) is equal to or less than the first threshold (S31: No), the calculation processing unit 20c determines that there is no abnormality in the molded object M (S32), continues the work (ridge-forming work) by the working device 2 (S32), and proceeds to S21 shown in Fig. 9B. In other words, the working machine 1 continues the ridge-forming work because the formation of the ridge M1 is normal.

[0098] On the other hand, as shown in Fig. 9C, when the deviation (abnormality score) exceeds the first threshold (S31: Yes), the calculation processing unit 20c performs two processes in parallel: S33 of the control process and S41 of the visualization process. Here, S33 of the control process will be described first, and then S41 of the visualization process will be described.

[0099] If the deviation (abnormality score) exceeds the first threshold (S31: Yes) and is equal to or less than the second threshold (S33: No), the calculation processing unit 20c determines that there is an abnormality (minor abnormality) below the specified level in the molded product M (S35), ignores the abnormality below the specified level, continues the work (ridge formation work) by the working device 2 (S34), and proceeds to S21 shown in Figure 9B. In other words, although there is an abnormality (minor abnormality) below the specified level in the formation of the ridge M1, it continues the ridge formation work as is because it does not need to be redone.

[0100] The work machine 1 can be manually operated, with at least one of the work device 2 and the traveling vehicle body 3 being manually operated. The work machine 1 is equipped with an alarm device 28 that notifies the driver of a molding abnormality (abnormality below a specified level) when a molding abnormality (abnormality below a specified level) is detected in the molded product M (e.g., ridge M1) during manual operation. The alarm device 28 is, for example, a speaker or buzzer that outputs an alarm sound (audio guidance, warning sound, etc.) indicating a molding abnormality (abnormality below a specified level) in the molded product M (e.g., ridge M1). The alarm device 28 may also be the display device 15. For example, the display device 15 may display an alarm to notify the driver of a molding abnormality (abnormality below a specified level) instead of or in addition to outputting an alarm sound from a speaker or buzzer. In other words, the alarm device 28 warns the driver during manual operation. The driver can take action such as slowing down the vehicle speed or lowering the work device 2 (lowering the working height of the work device 2).

[0101] The control device 20 controls at least one of the working device 2 and the traveling vehicle body 3 with predetermined control content. The working machine 1 is capable of automatic operation in which at least one of the working device 2 and the traveling vehicle body 3 is automatically controlled. When the control device 20 determines that the molded product M (e.g., ridge M1) has a molding abnormality (abnormality below a specified level) during automatic operation, the control device 20 changes the control content to a different one from the predetermined control content and performs automatic operation. For example, when the control device 20 determines that the molded product M has a molding abnormality (abnormality below a specified level) during automatic operation, the control device 20 performs automatic operation by at least one of changing the speed of the traveling vehicle body 3 when the molding abnormality (abnormality below a specified level) occurs and changing the working position of the working device 2 when the molding abnormality (abnormality below a specified level) occurs. In other words, during automatic operation, the control device 20 takes measures such as slowing down the vehicle speed or deepening the working device 2 (lowering the working height of the working device 2).

[0102] Specifically, when a molding abnormality (abnormality below a specified level) is determined in the molded product M during automatic operation, the control device 20 performs at least one of changing the speed of the traveling vehicle body 3 to a speed slower than that in the case of the molding abnormality (abnormality below a specified level) and changing the working device 2 to a second working position that is lower than the first working position in the case of the molding abnormality (abnormality below a specified level). Note that when a molding abnormality (abnormality below a specified level) is determined in the molded product M during automatic operation, the control device 20 may perform at least one of changing the speed of the traveling vehicle body 3 to a speed faster than that in the case of the molding abnormality (abnormality below a specified level) and changing the working device 2 to a third working position that is higher than the first working position in the case of the molding abnormality (abnormality below a specified level).

[0103] On the other hand, if the deviation (abnormality score) exceeds the second threshold (S33: Yes), the calculation processing unit 20c determines that there is an abnormality in the formation of the molded product M (S35), returns to the beginning of the ridge row and starts the work over again (S35), and returns to S11 shown in Figure 9A. In other words, since an abnormality that exceeds the specified limit has occurred in the formation of the ridge M1, the work machine 1 starts the ridge formation work over from the beginning of the ridge M1.

[0104] When a molding abnormality (abnormality exceeding specifications) is determined in the molded product M (e.g., ridge M1) during manual operation, the notification device 28 notifies the driver of the molding abnormality (abnormality exceeding specifications). The notification device 28 notifies the driver of a message recommending redoing. Based on the redo message, the driver redoes the ridge formation work from the beginning of the ridge M1. On the other hand, in the case of automatic operation, the control device 20 raises the working device 2 so that it is not in contact with the field H1, then backs up (reverses) the traveling vehicle body 3 to the beginning of the ridge M1, and redoes the ridge formation work from the beginning of the ridge M1. Note that when redoing the ridge formation work, it is sufficient to return to the beginning of the ridge M1, and the method for doing so is not limited to backing up (reverses) the traveling vehicle body 3.

[0105] Here, S41 of the visualization process shown in Fig. 9C will be described. As shown in Fig. 9C, when the degree of deviation (abnormality score) exceeds the first threshold (S31: Yes), the calculation processing unit 20c performs the visualization process (S41 to S43).

[0106] The calculation processing unit 20c converts the abnormal segmentation AS (see the upper right of FIG. 8) of the elevation image C that has been subjected to the abnormality detection process (abnormality detection AI) in S26 into an abnormality label AL (see the lower right of FIG. 8), and generates an elevation image C with this abnormality label AL (S41). The abnormality label AL is an image that is an integer value, for example, between 0 and 255, according to the abnormality segmentation AS (abnormality distribution).

[0107] The arithmetic processing unit 20c restores the elevation image C with the anomaly label AL (abnormal area) attached to it into point cloud data with abnormal point cloud data in the coordinate system of the sensing device 25. For example, the arithmetic processing unit 20c restores the elevation image C with the anomaly label AL generated in S41 to point cloud data with the anomaly label AL attached (see the point cloud data image PD shown in the lower center of FIG. 8) (S42). This restoration uses the parameters used for creating the elevation image in S23 to S25. Note that although the point cloud data image PD shown in the lower center of FIG. 8 shows semicircular ridges M1, this is merely an example, and it should be understood that to be precise, trapezoidal ridges M1 are shown.

[0108] The calculation processing unit 20c calibrates (LCC: LiDAR camera calibration) the captured image by the imaging device 26 and the point cloud data with the abnormal point cloud data to align them, and generates a composite image SI by superimposing the abnormal area image AE (an image showing an abnormal area generated based on the abnormal label AL) based on the abnormal point cloud data on the captured image (S43).Then, the control device 20 causes the display device 15 to display the composite image SI by superimposing only the abnormal area image AE (an abnormal label AL) on the captured image.

[0109] The control device 20 transmits the composite image SI to the server 50 shown in Fig. 1 (S44). The storage device 52 of the server 50 stores the composite image SI. The control device 20 may associate the composite image SI with position information of the molding M included in the composite image SI and transmit the image to the server 50. The storage device 52 of the server 50 stores the composite image SI and position information of the molding M included in the composite image SI in association with each other.

[0110] As shown in Figures 8 and 9B above, the judgment method involves the calculation processing unit 20c judging molding abnormalities in the molding M based on an elevation image C of the molding M formed in the field H1 by the working device 2 (S25, S26).

[0111] The main characteristic features and effects of the determination system S and the determination method in the above-described embodiments are as follows.

[0112] (Item A1) A judgment system S including a calculation processing unit 20c that judges molding abnormalities of a molded product M based on an image C (for example, an elevation image C) of the molded product M molded in a field H1 by a working device 2.

[0113] According to this configuration, it is possible to properly determine whether the molded object M molded in the field H1 by the work implement 2 is good or bad.

[0114] (Item A2) The determination system S according to item A1, wherein the calculation processing unit 20c determines a molding abnormality of the molded product M based on the processing result of abnormality detection processing of the image C.

[0115] This configuration makes it possible to find abnormalities in formation that are difficult to see or that are easily overlooked by the naked eye.

[0116] (Item A3) The calculation processing unit 20c performs the abnormality detection process by calculating the degree of deviation of the image C from the normal shape of the molding (e.g., normal image C1), and determines that the molding M has a molding abnormality if the deviation result exceeds a threshold value.

[0117] According to this configuration, the degree of molding abnormality (deviation) of the molded product M can be used to judge molding abnormality based on objective criteria.

[0118] (Item A4) The calculation processing unit 20c performs the abnormality detection process by calculating an abnormal area in the image C that deviates from the shape of the normal molding, and generates the image C with the abnormal area added. This is the judgment system S described in item A2 or A3.

[0119] According to this configuration, the position, size, and / or range of the abnormal area in the image C can be grasped.

[0120] (Item A5) A judgment system S described in any one of items A1 to A4, which has the working device 2 and a traveling body 3 on which the working device 2 is mounted, and which is equipped with a working machine 1 that can be manually operated so that at least one of the working device 2 and the traveling body 3 is operated manually, and the working machine 1 is equipped with an alarm device 28 that alerts the operator to a molding abnormality when a molding abnormality is determined in the molded product M during the manual operation.

[0121] According to this configuration, it is possible to notify the user of the occurrence of molding abnormalities in the molded product M during manual operation, thereby reducing the occurrence of molding abnormalities.

[0122] (Item A6) A judgment system S described in any one of items A1 to A5, which comprises a work machine 1 having the work device 2, a traveling vehicle body 3 to which the work device 2 is attached, and a control device 20 that controls at least one of the work device 2 and the traveling vehicle body 3 with predetermined control content, and which is capable of automatic operation in which at least one of the work device 2 and the traveling vehicle body 3 is automatically controlled, and when a molding abnormality is determined in the molded product M during automatic operation, the control device 20 changes the control content to a different one from when no molding abnormality is determined in the molded product and performs automatic operation.

[0123] According to this configuration, automatic operation is performed with control contents different from those when no molding abnormality is determined in the molded product, thereby reducing the occurrence of molding abnormalities during automatic operation.

[0124] (Item A7) The control device 20 performs automatic driving of the judgment system S described in item A6 by, when a molding abnormality is determined in the molded product M during automatic driving, at least one of changing the speed of the traveling vehicle body 3 when a molding abnormality occurs and changing the working position of the working device 2 when a molding abnormality occurs.

[0125] According to this configuration, when there is a molding abnormality in the molded product M, the speed (vehicle speed) of the traveling vehicle body 3 and / or the working position of the work machine 1 are changed to perform automatic operation, thereby reducing the occurrence of molding abnormalities in the case of automatic operation.

[0126] (Item A8) The control device 20, when a molding abnormality is determined in the molded product M during automatic operation, performs at least one of changing the speed of the traveling vehicle body 3 to a speed slower than that in the case of a molding abnormality, and changing the working device 2 to a second working position that is lower than the first working position in the case of a molding abnormality, in the judgment system S described in Item A7, which performs automatic operation.

[0127] According to this configuration, automatic driving is performed by changing the speed (vehicle speed) of the traveling vehicle body 3 to a speed (vehicle speed) slower than the speed (vehicle speed) when a molding abnormality occurs in the molded product M and / or by changing the working position of the working machine 1 to a second working position which is lower than the first working position, thereby reducing the occurrence of molding abnormalities during automatic driving.

[0128] (Item A9) A judgment system S described in any one of items A1 to A8, which includes a sensing device 25 that acquires point cloud data showing the topography around the work device 2, and the calculation processing unit 20c that performs image processing on extracted point cloud data extracted from a region of interest K of the point cloud data to generate the image C.

[0129] According to this configuration, point cloud data for the region of interest K is extracted from the point cloud data showing the surrounding topography and image processing is performed, so the number of data points can be reduced, and the image processing load can be reduced.

[0130] (Item A10) The determination system S according to Item A9, wherein the arithmetic processing unit 20c starts the image processing when the detection condition for the molded object M is met.

[0131] According to this configuration, image processing can be started at a timing suitable for detecting the molding M, and unnecessary image processing can be prevented.

[0132] (Item A11) The determination system S according to Item A10, wherein the calculation processing unit 20c determines that the detection condition is met when the work by the work device 2 is started and the work has moved a predetermined distance since the start of the work.

[0133] According to this configuration, image processing can be started in time with the start of molding of the molded product M, and unnecessary image processing can be prevented.

[0134] (Item A12) A determination system S described in Item A9, which is provided with a work machine 1 having the work device 2, a traveling vehicle body 3 to which the work device 2 is attached, a position estimation unit 20a that estimates the position of the traveling vehicle body 3, and the sensing device 25, wherein the calculation processing unit 20c converts the extracted point cloud data of the area of ​​interest K acquired by the sensing device into a world coordinate system based on position information indicating the position of the traveling vehicle body estimated by the position estimation unit, performs this conversion on the extracted point cloud data of multiple areas of interest K that have different position information, and overlays the extracted point cloud data of the multiple areas of interest K in the world coordinate system.

[0135] According to this configuration, the point cloud data of the regions of interest K, which are located at different positions, can be properly aligned and superimposed, so that the point cloud data can be made denser in the superimposed areas, thereby improving the image quality of the image C.

[0136] (Item A13) The determination system S according to item A12, wherein the calculation processing unit 20c generates the image C by projecting the extracted point cloud data superimposed in the world coordinate system onto a predetermined plane of the world coordinate system.

[0137] According to this configuration, the image C can be generated suitably.

[0138] (Item A14) The determination system S according to item A4, wherein the calculation processing unit 20c restores the image C with the abnormal area added thereto into point cloud data with abnormal point cloud data in the coordinate system of the sensing device 25.

[0139] According to this configuration, abnormal point cloud data can be distinguished from the point cloud data.

[0140] (Item A15) A judgment system S described in Item A14, which is equipped with a display device 15 and an imaging device 26 that captures images of the terrain around the work device 2, and the calculation processing unit 20c calibrates and aligns the image captured by the imaging device 26 with the point cloud data with the abnormal point cloud data, and generates a composite image SI by overlaying an abnormal area image AE based on the abnormal point cloud data on the captured image, and displays it on the display device 15.

[0141] According to this configuration, the captured image is aligned with the point cloud data with the abnormal point cloud data, and a composite image SI is generated and displayed by superimposing the abnormal area image AE based on the abnormal point cloud data on the captured image, so that the position, size, range, etc. of the abnormal area in the captured image can be grasped.

[0142] (Item A16) A determination system S according to item A15, comprising a server 50, the server 50 comprising a storage device 52 that stores the composite image SI and the positional information of the molding M contained in the composite image SI in association with each other.

[0143] According to this configuration, the composite image SI obtained by superimposing the abnormal region image AE and the position information of the molding M included in the composite image SI can be managed in association with each other.

[0144] (Item A17) The threshold values ​​include a first threshold value and a second threshold value greater than the first threshold value, and the calculation processing unit 20c determines that there is no abnormality in the molded object M if the deviation degree is less than the first threshold value, determines that there is an abnormality in the molded object M that is below a specified level if the deviation degree exceeds the first threshold value and is less than the second threshold value, and determines that there is a molding abnormality in the molded object M if the deviation degree exceeds the second threshold value.

[0145] According to this configuration, the molding of the molded product M can be appropriately judged into three types: normal, minor abnormality, and molding abnormality that cannot be overlooked.

[0146] (Item A18) The determination system S according to Item A3, wherein the threshold value varies depending on the type of the working device 2.

[0147] According to this configuration, the threshold value differs depending on the type of working device 2, so that molding abnormalities can be appropriately determined even when molding a molded object M whose shape and / or size differ depending on the type of working device 2.

[0148] (Item A19) A determination method in which the calculation processing unit 20c determines a molding abnormality of the molded product M based on an image C of the molded product M molded in the field H1 by the working device 2.

[0149] According to this configuration, it is possible to properly determine whether the molded object M molded in the field H1 by the work implement 2 is good or bad.

[0150] Although the present invention has been described above, the embodiments disclosed herein should be considered to be illustrative and not restrictive in all respects. The scope of the present invention is defined by the claims, not by the above description, and is intended to include all modifications within the meaning and scope of the claims. [Explanation of symbols]

[0151] 1: Work equipment 2: Work equipment 3: Running vehicle 15:Display device 20: Control device 20a: Position estimation part 20c: arithmetic processing unit 25: Sensing device 26: Imaging device 28: Alarm device 50: Server 52: Storage device H1: Field S: Judgment system

Claims

1. A judgment system including a calculation processing unit that judges molding abnormalities of a molded product based on an image of the molded product molded in a field by a working device.

2. The determination system according to claim 1 , wherein the arithmetic processing unit determines a molding abnormality of the molded product based on a processing result obtained by performing an abnormality detection processing on the image.

3. The judgment system described in claim 2, wherein the calculation processing unit performs the abnormality detection process by calculating the degree of deviation of the image from the normal shape of the molding, and judges that the molding has an abnormal molding if the deviation result exceeds a threshold value.

4. The determination system according to claim 3 , wherein the calculation processing unit performs the abnormality detection process by calculating an abnormal area in the image that deviates from the shape of the normal molding, and generates the image with the abnormal area marked.

5. a manually operable work machine having the work device and a traveling vehicle body to which the work device is attached, wherein at least one of the work device and the traveling vehicle body is manually operated; The determination system according to any one of claims 1 to 4, wherein the work machine is provided with an alarm device that notifies of molding abnormality when molding abnormality is determined in the molded product during the manual operation.

6. a working machine capable of automatic operation, which includes the working device, a traveling vehicle body to which the working device is attached, and a control device that controls at least one of the working device and the traveling vehicle body with predetermined control content, and in which at least one of the working device and the traveling vehicle body is automatically controlled; The control device, when a molding abnormality of the molded product is determined during automatic operation, changes the control content to a different one from when a molding abnormality of the molded product is not determined, and performs automatic operation. A judgment system described in any one of claims 1 to 4.

7. The judgment system described in claim 6, wherein when a molding abnormality is determined in the molded product during automatic operation, the control device performs automatic operation by at least one of changing the speed of the traveling vehicle body when a molding abnormality occurs and changing the working position of the working device when a molding abnormality occurs.

8. The judgment system described in claim 7, wherein when a molding abnormality is determined in the molded product during automatic driving, the control device performs automatic driving by at least one of changing the speed of the traveling vehicle body to a speed slower than the speed at which the molding abnormality occurs, and changing the working device to a second working position that is lower than the first working position at which the molding abnormality occurs.

9. a sensing device that acquires point cloud data indicating the topography around the work implement; 5. The determination system according to claim 1, wherein the arithmetic processing unit generates the image by performing image processing on extracted point cloud data extracted from a region of interest in the point cloud data.

10. The determination system according to claim 9 , wherein the arithmetic processing unit starts the image processing when the detection condition for the molded object is met.

11. The determination system according to claim 10 , wherein the calculation processing unit determines that the detection condition is met when the work tool starts working and has moved a predetermined distance since the start of the work tool.

12. a work machine including the work device, a traveling vehicle body to which the work device is attached, a position estimation unit that estimates the position of the traveling vehicle body, and the sensing device; The determination system of claim 9, wherein the calculation processing unit converts the extracted point cloud data of the region of interest acquired by the sensing device from the coordinate system of the sensing device to a world coordinate system based on position information indicating the position of the traveling vehicle body estimated by the position estimation unit, performs this conversion on the extracted point cloud data of multiple regions of interest having different position information, and overlays the extracted point cloud data of the multiple regions of interest in the world coordinate system.

13. The determination system according to claim 12 , wherein the arithmetic processing unit generates the image by projecting the extracted point cloud data superimposed in the world coordinate system onto a predetermined plane of the world coordinate system.

14. The determination system according to claim 4 , wherein the arithmetic processing unit restores the image with the abnormal region added thereto into point cloud data with abnormal point cloud data in a coordinate system of a sensing device.

15. a display device; an imaging device that captures images of the terrain around the work implement; 15. The determination system according to claim 14, wherein the calculation processing unit calibrates and aligns the image captured by the imaging device and the point cloud data with the abnormal point cloud data, generates a composite image by overlaying an abnormal area image based on the abnormal point cloud data on the captured image, and displays the composite image on the display device.

16. Equipped with a server, The determination system according to claim 15 , wherein the server includes a storage device that stores the composite image and position information of the molding included in the composite image in association with each other.

17. The threshold values ​​include a first threshold value and a second threshold value that is greater than the first threshold value, The judgment system described in claim 3, wherein the calculation processing unit judges that there is no abnormality in the molded product if the deviation degree is less than the first threshold value, judges that there is an abnormality in the molded product below a specified level if the deviation degree exceeds the first threshold value and is less than the second threshold value, and judges that there is a molding abnormality in the molded product if the deviation degree exceeds the second threshold value.

18. The determination system according to claim 3 , wherein the threshold value differs depending on the type of the work tool.

19. A method for determining whether a molding abnormality has occurred in a molded product based on an image of the molded product molded in a field by a working device using an arithmetic processing unit.

Images