Information processing system, information processing method and construction machine

Abstract

The present invention is made in view of the problems of existing working machines, and one of its purposes is to provide an information processing system, an information processing method and a construction machine for a working machine that can estimate the posture of a working part with a simple structure. The information processing system (10) includes a storage unit (30) and an image information acquisition unit (12). The storage unit (30) stores data (Gs) of a reference image of a working part (40) of the working machine (100) and data (Ks) of the posture of the working part (40) of the working machine (100) in correspondence. The image information acquisition unit (12) acquires data (Gj) of an image of the working part (40).

Description

Technical Field

[0001] The present invention relates to an information processing system, an information processing method and a construction machine. Background Art

[0002] A construction machine is known that includes a camera for capturing images of a working device to determine its posture. For example, Patent Document 1 describes a construction machine comprising: a camera mounted on a rotating structure to capture images of the working device; an angle detection unit for detecting relative angles; and a posture determination unit for determining the posture. This construction machine detects the relative angles between links based on the edges of each link extracted from the camera image, and determines the posture of the working device relative to the rotating structure based on these relative angles.

[0003] Prior art literature

[0004] Patent Literature

[0005] Patent Document 1: Japanese Patent Application Laid-Open No. 2017-053627 Summary of the Invention

[0006] Problems to be solved by the invention

[0007] The present inventors have found the following regarding a construction machine including a boom, an arm, and attachments driven by power such as hydraulic pressure.

[0008] A certain construction machine uses power to drive an arm mechanism consisting of a boom and an arm, thereby operating an attachment, such as a bucket, attached to the arm mechanism to perform a desired task. To prevent collisions between the arm mechanism and attachments (hereinafter referred to as the "working unit") or to provide feedback on the working unit's position to control its operation, it is considered possible to determine and use information on the three-dimensional position of each working unit component (hereinafter referred to as the "working unit posture" or simply "posture"), such as the link angles between the boom, arm, and bucket, or the stroke length of each cylinder.

[0009] To determine the posture of the working unit, one approach is to install a stroke sensor on each cylinder driving each component and calculate and determine the posture of the working unit based on the detection results of the sensor. However, this approach requires installing a stroke sensor for each component and wiring between the stroke sensor and the control device, complicating the structure and being cost-effective.

[0010] From the perspective of estimating the posture of the working unit with a simple structure, it cannot be said that the construction machine described in Patent Document 1 has fully addressed this problem. This problem is not limited to construction machines, but also occurs in other types of working machines.

[0011] The present invention has been made in view of the above-mentioned problems, and one object of the present invention is to provide an information processing system for a working machine capable of estimating the posture of a working unit with a simple configuration.

[0012] Solutions for solving problems

[0013] In order to solve the above-mentioned problems, an information processing system of a certain embodiment of the present invention comprises: a storage unit, which stores data of a reference image of a working part of a working machine and data of a posture of the working part of the working machine in correspondence with each other; and an acquisition unit, which acquires data of an image of the working part of the working machine for comparison with the data of the reference image.

[0014] In addition, any combination of the above, or forms obtained by replacing the structural elements of the present invention or the expressions in methods, devices, programs, transient or non-transient storage media recording the programs, and systems, etc., are also valid as forms of the present invention.

[0015] Effects of the Invention

[0016] According to the present invention, it is possible to provide an information processing system for a working machine capable of estimating the posture of a working unit with a simple configuration. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 It is a side view schematically showing a working machine including the information processing system according to the first embodiment.

[0018] Figure 2 It is a schematic representation Figure 1 Block diagram of an information processing system.

[0019] Figure 3 It is an explanation Figure 1 An illustration showing the posture of the working part of a working machine.

[0020] Figure 4 It shows Figure 1 A graph of learning data for an information processing system.

[0021] Figure 5 It is an explanation Figure 1 An explanatory diagram of a posture estimation process of an information processing system.

[0022] Figure 6 It shows Figure 1 Flowchart of the actions of the information processing system.

[0023] Figure 7 It shows Figure 1 Flowchart of the actions of the information processing system.

[0024] Description of Reference Numerals

[0025] 10: Information processing system; 20d: Model generation unit; 20e: Posture estimation unit; 20k: Background information removal unit; 20n: Color information removal unit; 30: Storage unit; 32: Model storage unit; 40: Working unit; 42: Boom; 44: Arm; 46: Bucket; 100: Working machine; 1000: Construction machine. DETAILED DESCRIPTION

[0026] The present inventors have studied a method for estimating the posture of a working unit based on image information from a camera, and have come to the following conclusions. In construction machinery, the following structure is considered: the edges of each part are extracted from the image information of a camera used to photograph the working unit, the relative angles of each part are detected based on the edges, and the posture is determined based on the relative angles. However, in this structure, multiple stages of information processing are to be performed successively and continuously, so an expensive, high-performance CPU is required to perform a large amount of information processing. If a CPU with low processing power is used, the processing cannot keep up, the estimation accuracy is reduced, and collision avoidance and motion control will not function properly. In addition, this information processing has the following problem: a corresponding information processing program needs to be developed for each of the various shapes of construction machinery, which consumes a lot of development time.

[0027] Against this backdrop, the present inventors have developed a technology for estimating the posture of a work unit using camera image information, reference image data of the work unit, and a database of posture data. This technology enables posture estimation using a simple configuration while minimizing the loss of estimation accuracy. An example of this technology will be described below based on an implementation example.

[0028] The present invention is described below based on preferred embodiments with reference to the accompanying drawings. In the embodiments and modifications, identical or equivalent structural elements and components are denoted by the same reference numerals, and duplicate descriptions are omitted as appropriate. Furthermore, for ease of understanding, the dimensions of the components in the drawings are appropriately enlarged or reduced. Furthermore, in the drawings, some components that are not essential for describing the embodiments are omitted.

[0029] In addition, terms including ordinal numbers such as first and second are used to describe a plurality of structural elements. These terms are only used to distinguish one structural element from other structural elements, and the structural elements are not limited by these terms.

[0030] [First embodiment]

[0031] The configuration of an information processing system 10 for a working machine according to a first embodiment of the present invention will be described with reference to the drawings. Figure 1It is a side view schematically showing a working machine 100 including the information processing system 10 according to the first embodiment. Figure 2 1 is a block diagram schematically showing the information processing system 10 .

[0032] The information processing system 10 includes an image information acquisition unit 12, an environment information acquisition unit 14, a posture information acquisition unit 16, a control unit 20, and a storage unit 30. The information processing system 10 has a normal operation time (hereinafter referred to as "non-learning operation time") during machine learning and non-machine learning. During machine learning, the information processing system 10 can generate a posture estimation model based on the learning image information and learning posture information of each part of the working part 40 of the working machine 100. During non-learning operation, the information processing system 10 can estimate the posture of the working part 40 based on real-time image information and the posture estimation model. The working machine 100 can control the operation of the working part 40 based on the posture of the working part 40 estimated by the information processing system 10.

[0033] The image information acquisition unit 12 acquires image data of the working unit 40. The environmental information acquisition unit 14 acquires information related to the surrounding environment of the working machine 100. The posture information acquisition unit 16 acquires data related to the posture of the working unit 40 (hereinafter referred to as "posture data"). The control unit 20 performs various data processing related to the generation of the posture estimation model and the posture estimation of the working unit 40. The storage unit 30 is used to store data referenced or updated by the control unit 20. First, the structure of the working machine 100 is explained, and other structures are described later.

[0034] The working machine 100 of this embodiment is a construction machine that moves a bucket 46 to perform work, and functions as a so-called power shovel. The working machine 100 includes a lower travel portion 36, an upper vehicle body portion 34, an arm mechanism 48, and a bucket 46. In this embodiment, the arm mechanism 48 and the bucket 46 constitute a working portion 40. The lower travel portion 36 is configured to be able to travel in a predetermined direction via an infinite track or the like. The upper vehicle body portion 34 is mounted on the lower travel portion 36. The upper vehicle body portion 34 and the working portion 40 are configured to be able to rotate around a rotation axis La relative to the lower travel portion 36 via a rotation drive portion 60. The rotation drive portion 60 can be composed of, for example, a rotation motor (not shown) and a rotation gear (not shown). An operator's cabin 38 is provided on the upper vehicle body portion 34.

[0035] The control room 38 is equipped with an operating unit 54 for operating the working unit 40. When an operation is input from the operating unit 54, multiple hydraulic valves 58 open and close in response to the operation. In response to the opening and closing of the hydraulic valves 58, hydraulic oil supplied by a hydraulic pump (not shown) is delivered to multiple hydraulic cylinders 56. The hydraulic cylinders 56 include hydraulic cylinders 56a, 56b, and 56c, which are arranged in order from the base end to the tip end of the arm mechanism 48. The hydraulic cylinders 56a, 56b, and 56c extend and retract in response to the amount of hydraulic oil delivered.

[0036] For example, the base end of the arm mechanism 48 is located on the right side of the operator's cab 38 in the upper vehicle body 34. The arm mechanism 48 includes, for example, a boom 42 and an arm 44 extending forward from the upper vehicle body 34. A bucket 46 is attached to the front end of the arm mechanism 48. In this manner, the work machine 100 can change the position of the working unit 40 in response to the operator's control, thereby driving the bucket 46 to perform the intended work. Furthermore, the work machine 100 can rotate the upper vehicle body 34 and the working unit 40 to move the bucket 46 three-dimensionally.

[0037] Figure 3 This is an explanatory diagram illustrating the posture of the working unit 40. The hydraulic cylinders 56a, 56b, and 56c can change their extension and retraction lengths (L1, L2, and L3) in accordance with the hydraulic pressure. The boom 42 is configured so that the front end thereof rotates vertically about the base end thereof on the upper vehicle body 34 side by extension and retraction of the hydraulic cylinder 56a. The arm 44 is configured so that the front end thereof rotates forward and backward about the base end thereof on the boom 42 side by extension and retraction of the hydraulic cylinder 56b. The bucket 46 is configured so that the front end thereof rotates forward and backward or vertically about the base end thereof on the arm 44 side by extension and retraction of the hydraulic cylinder 56c.

[0038] The working unit 40 can change the bending angles θ1, θ2, and θ3 of the joints connecting the boom 42, the arm 44, and the bucket 46 by changing the extension and contraction lengths L1, L2, and L3 of the hydraulic cylinders 56a, 56b, and 56c. As an example, the angle θ1 is the angle of the boom 42 relative to the horizontal plane, the angle θ2 is the bending angle of the joint connecting the boom 42 and the arm 44, and the angle θ3 is the bending angle of the joint connecting the arm 44 and the bucket 46.

[0039] The posture of the working unit 40 can be defined by the positions and relative angles of the boom 42, arm 44, and bucket 46. The shapes of the boom 42, arm 44, and bucket 46 are fixed, and the posture of the working unit 40 can be determined through geometric calculations based on the sizes of the boom 42, arm 44, and bucket 46, as well as the extension and contraction lengths L1, L2, and L3 or bending angles θ1, θ2, and θ3 of the hydraulic cylinders 56a, 56b, and 56c.

[0040] Return to Figure 2 . Figure 2 Each block shown can be implemented in hardware by components such as a computer processor, CPU, and memory, electronic circuits, and mechanical devices, and in software by computer programs, etc. However, functional blocks implemented through the cooperation of these components are depicted here. Therefore, those skilled in the art will understand that these functional blocks can be implemented in various forms through a combination of hardware and software.

[0041] The image information acquisition unit 12 will be described. The image information acquisition unit 12 of this embodiment has an image sensor for photographing the operation unit 40. During machine learning described later, the image information acquisition unit 12 provides the photographing result obtained by photographing the operation unit 40 to the control unit 20 as image data. Hereinafter, the image data acquired during machine learning will be referred to as "reference image data Gs." During non-learning operations, the image information acquisition unit 12 also provides the photographing result obtained by photographing the operation unit 40 as image data to the control unit 20. Hereinafter, the image data acquired during non-learning operations will be simply referred to as "image data Gj." The image data Gj may be real-time image data.

[0042] The image information acquisition unit 12 is configured to rotate integrally with the working unit 40. Specifically, the image information acquisition unit 12 is positioned on the ceiling of the control room 38 so as to be able to capture images of the working unit 40. As the working unit 40 rotates, the image information acquisition unit 12 moves integrally with the working unit 40. Therefore, even with the rotation, the relative positional relationship between the image information acquisition unit 12 and the working unit 40 does not change.

[0043] The posture information acquisition unit 16 will be described. It includes stroke sensors 16a, 16b, and 16c that acquire the extension and contraction lengths L1, L2, and L3 of the hydraulic cylinders 56a, 56b, and 56c of this embodiment. The posture information acquisition unit 16 is installed during machine learning and removed during non-learning operations. During machine learning, the posture information acquisition unit 16 provides data on the extension and contraction lengths L1, L2, and L3 to the control unit 20.

[0044] The environmental information acquisition unit 14 will be described. When acquiring image data, if the surrounding environment, such as the weather, varies, the brightness and color temperature of the image will also vary, which can be a major factor in increasing errors in posture estimation. Therefore, in this embodiment, the environmental information acquisition unit 14 acquires information related to the surrounding environment and corrects the image data based on the acquisition results. The environmental information acquisition unit 14 of this embodiment includes an illuminance sensor for acquiring the surrounding brightness and a color temperature sensor for acquiring the color temperature. The environmental information acquisition unit 14 provides the acquisition results to the control unit 20 as environmental information Mp. In this example, the environmental information acquisition unit 14 is located on the ceiling of the control room 38.

[0045] The storage unit 30 will be described. The storage unit 30 includes a model storage unit 32. The model storage unit 32 stores a posture estimation model for estimating the posture of the working unit 40. This posture estimation model is generated through well-known machine learning based on the reference image data Gs and posture data. The posture estimation model can also be said to be a function whose input and output data formats are predetermined. The posture estimation model of the embodiment receives image data Gj as input. The posture estimation model of the embodiment also outputs information related to the estimated posture corresponding to the image data. The method for generating the posture estimation model will be described later.

[0046] The control unit 20 will be described. The control unit 20 includes an image information receiving unit 20a, a posture information receiving unit 20b, a model generating unit 20d, a posture estimating unit 20e, an estimated posture presenting unit 20f, an individual information storage unit 20h, an environmental information receiving unit 20g, an image information correcting unit 20j, a background information removing unit 20k, and a color information removing unit 20n. An application program containing multiple modules corresponding to these multiple functional blocks can be installed in the storage device (e.g., storage unit 30) of the information processing system 10. The processor (e.g., CPU) of the information processing system 10 can load the application program into the main memory and execute it, thereby enabling the functions of each functional block to be realized.

[0047] The image information receiving unit 20a receives input of imaging results from the working unit 40 from the image information acquiring unit 12. In particular, during machine learning, the image information receiving unit 20a receives reference image data Gs for learning from the image information acquiring unit 12. Furthermore, during non-learning operations, the image information receiving unit 20a receives image data Gj from the image information acquiring unit 12.

[0048] The posture information receiving unit 20b receives input of posture information of the working unit 40 from the posture information acquiring unit 16. Specifically, during machine learning, the posture information receiving unit 20b receives data on the extension and contraction lengths L1, L2, and L3 of the stroke sensors 16a, 16b, and 16c. The data on the extension and contraction lengths L1, L2, and L3 are collectively referred to as posture data Ks.

[0049] Figure 4: is a diagram showing the learning data of the information processing system 10. In this diagram, for ease of understanding, the reference image data Gs and the posture data Ks from the image information acquisition unit 12 are replaced with a diagram on a plane for explanation. During machine learning, the control unit 20 stores the received reference image data Gs and posture data Ks in the storage unit 30 in correspondence. During machine learning, the control unit 20 causes the posture of the working unit 40 to change significantly within the movable range, and each time it changes, the reference image data Gs and the posture data Ks are stored in the storage unit 30 in correspondence. The reference image data Gs and the posture data Ks stored in the storage unit 30 are referred to as learning data Sd. It is expected that the learning data Sd covers the postures that the working unit 40 can take. Therefore, as Figure 4 As shown, the learning data Sd includes a large amount of reference image data Gs and posture data Ks that correspond to each other.

[0050] The model generation unit 20d uses the corresponding reference image data Gs and posture data Ks of the learning data Sd as teacher data to generate a posture estimation model. The model generation unit 20d of this embodiment uses the reference image data Gs and posture data Ks as teacher data to generate a posture estimation model through machine learning (supervised learning). The model generation unit 20d can use well-known machine learning methods such as support vector machines, neural networks (including deep learning), and random forests to generate the posture estimation model. The model generation unit 20d stores the generated posture estimation model in the model storage unit 32.

[0051] During non-learning operations, the posture estimation unit 20e estimates the posture of the working unit 40 based on the image data Gj and the information stored in the storage unit 30. For example, the posture estimation unit 20e may compare the image data Gj with the reference image data Gs of the learning data Sd and use the posture data Ks corresponding to the reference image data Gs with the highest degree of similarity as the estimation result. In this case, since a large amount of reference image data Gs is referenced, obtaining the result may take time. Therefore, the posture estimation unit 20e of this embodiment uses the posture estimation model generated by the model generation unit 20d to derive the estimated posture based on the image data Gj.

[0052] Figure 5 This diagram illustrates the posture estimation process performed by the posture estimation model generated by the model generation unit 20d. When image data Gj is input, the posture estimation model outputs estimated posture information Ke corresponding to the image data. In this embodiment, the estimated posture information Ke includes the bending angles θ1, θ2, and θ3 of the joints of the boom 42, arm 44, and bucket 46 of the working unit 40.

[0053] The estimated posture presentation unit 20f transmits the estimated posture information Ke estimated by the posture estimation unit 20e to the outside of the information processing system 10. In this embodiment, the estimated posture presentation unit 20f transmits the estimated posture information Ke to the work machine control unit 62. The work machine control unit 62 controls the operation of the work machine 100. For example, the work machine control unit 62 controls the opening and closing of the hydraulic valve 58 to extend and retract the hydraulic cylinders 56a, 56b, and 56c, thereby controlling the operation of the boom 42, arm 44, and bucket 46.

[0054] The image information correction unit 20j of this embodiment corrects the image data Gj based on information about the surrounding environment at the time the image data Gj was acquired, the age of the work machine, or individual differences between the work machines. Furthermore, the environmental information receiving unit 20g and the individual information storage unit 20h provide correction information to the image information correction unit 20j.

[0055] The environmental information receiving unit 20g receives the acquisition result from the environmental information acquiring unit 14. In particular, the environmental information receiving unit 20g receives environmental information Mp from the environmental information acquiring unit 14. The image information correcting unit 20j corrects the brightness and color temperature of the image data Gj based on the environmental information Mp. The image information correcting unit 20j corrects the brightness of the image data Gj so that it matches the brightness of the reference image data Gs. The image information correcting unit 20j corrects the color temperature of the image data Gj so that it matches the color temperature of the reference image data Gs. This configuration can reduce estimation errors caused by the surrounding environment.

[0056] The appearance of the working unit 40 varies from one work machine 100 to another. These individual differences can be a major source of error in posture estimation. Therefore, the individual information storage unit 20h of this embodiment stores individual information Me for each individual. This individual information Me includes information related to individual differences in appearance caused by the age of the work machine 100, damage to the working unit 40, attachments, and deformation. The image information correction unit 20j corrects the image data Gj based on the individual information Me. This configuration can reduce estimation errors caused by individual differences.

[0057] The image data Gj includes a background image that varies for each location where the work machine 100 operates. Therefore, the background image included in the image data Gj can potentially cause errors in posture estimation. Therefore, the background information removal unit 20k of this embodiment removes information related to the background image from the image data Gj. This configuration can reduce estimation errors caused by the background image.

[0058] When the reference image data Gs and the image data Gj are stored and processed as full-color image data, the amount of data processed increases, which is disadvantageous in terms of processing speed and storage capacity. Therefore, the color information removal unit 20n of this embodiment removes color information from the reference image data Gs and the image data Gj to create grayscale image data. This configuration reduces the amount of data processed, which is advantageous in terms of processing speed and storage capacity.

[0059] The operation of the information processing system 10 configured as described above will be described. Figure 6 1 is a flowchart showing the operation of the information processing system 10. This figure shows operation S70 for generating a posture estimation model by machine learning during machine learning. The posture information acquisition unit 16 is previously installed in the working machine 100, and operation S70 is started when the manager inputs an instruction to generate a model.

[0060] If the model generation timing has arrived ("Yes" in step S71), the control unit 20 of the information processing system 10 receives the reference image data Gs and the posture data Ks from the image information acquisition unit 12 and the posture information acquisition unit 16 (step S72). In this step, the control unit 20 changes the posture of the working unit 40 widely within the movable range. Each time the posture changes, the control unit 20 receives the reference image data Gs and the posture data Ks and stores them in the storage unit 30.

[0061] The background information removal unit 20k removes information related to the background image from the reference image data Gs (step S73). The color information removal unit 20n removes color information from the reference image data Gs (step S74). The background image removal and color information removal can be performed on each received reference image data Gs, or on the reference image data Gs stored in the storage unit 30.

[0062] The model generation unit 20d generates a posture estimation model using machine learning based on the reference image data Gs from which the background image and color information have been removed and the posture data Ks. The model generation unit 20d then stores the model in the model storage unit 32 (step S75). Once the posture estimation model has been stored, step S70 ends. After step S70 ends, the posture information acquisition unit 16 can be removed from the work machine 100.

[0063] If the model generation timing has not arrived (step S71: No), steps S72 to S75 are skipped. This operation S70 is just an example, and the order of the steps may be changed, or some steps may be added, deleted, or modified.

[0064] Figure 7This is also a flowchart showing the operation of the information processing system 10. This figure shows operation S80 of estimating the posture of the working unit 40 from the image data Gj using the posture estimation model. In the non-learning operation, operation S80 is started when the administrator inputs an instruction to perform posture estimation.

[0065] If the timing for posture estimation has arrived (Yes in step S81 ), the control unit 20 of the information processing system 10 receives the image data Gj from the image information acquisition unit 12 (step S82 ). In this step, the received image data Gj is stored in the storage unit 30 .

[0066] The background information removal unit 20k removes information related to the background image from the image data Gj (step S83). The color information removal unit 20n removes color information from the image data Gj (step S84).

[0067] The image information correction unit 20j corrects the image data Gj based on the individual information Me stored in the individual information storage unit 20h (step S85). The image data Gj stored in the storage unit 30 is subjected to background image removal, color information removal, and image correction.

[0068] The posture estimation unit 20e estimates the posture of the working unit 40 based on the posture estimation model from the image data Gj after background image removal, color information removal, and image correction (step S86). In this step, the posture estimation model outputs estimated posture information Ke.

[0069] The estimated posture presentation unit 20f transmits the estimated posture information Ke output from the posture estimation model to the outside of the information processing system 10 (step S87). For example, the estimated posture presentation unit 20f transmits the estimated posture information Ke to the work machine control unit 62. If the estimated posture information Ke is transmitted, act S80 ends. Act S80 is repeated until the instruction for posture estimation is no longer given.

[0070] If the timing for posture estimation has not arrived (step S81: No), steps S82 to S87 are skipped. This operation S80 is just an example, and the order of the steps may be changed, or some steps may be added, deleted, or modified.

[0071] The features of the information processing system 10 of this embodiment, configured as described above, will now be described. This information processing system 10 includes a storage unit 30 that stores reference image data Gs of the working unit 40 of the working machine 100 and posture data Ks of the working unit 40 of the working machine 100 in association with each other; and an image information acquisition unit 12 that acquires image data Gj of the working unit 40 of the working machine 100 and compares it with the reference image data Gs. This configuration enables estimation of the posture of the working unit 40 based on the image data Gj, the reference image data Gs, and the posture data Ks acquired by the image information acquisition unit 12.

[0072] The image information acquisition unit 12 may be configured to rotate integrally with the working unit 40 of the working machine 100. In this case, the operation of the working unit 40 of the working machine 100 can be easily identified.

[0073] The information processing system 10 may include a posture estimation unit 20e that estimates the posture of the working unit 40 based on the image data Gj and the information stored in the storage unit 30. In this case, the posture of the working unit 40 can be estimated based on the image data Gj.

[0074] The storage unit 30 may store a posture estimation model generated by machine learning based on the reference image data Gs and the posture data Ks. In this case, the posture estimation model can be generated by machine learning.

[0075] The information processing system 10 may further include an image information correction unit 20j that corrects the image data Gj based on at least one of information related to the surrounding environment when the image data Gj was acquired, the age of the work machine 100, and individual differences among the work machines 100. In this case, the image data Gj can be corrected to improve posture estimation accuracy.

[0076] Alternatively, the posture estimation unit 20e may estimate the posture of the working unit 40 based on at least one of information about the surrounding environment when the image data Gj was acquired, the age of the working machine, and individual differences between working machines. In this case, the image data Gj can be corrected to improve the accuracy of posture estimation.

[0077] The information processing system 10 may further include a color information removal unit 20n for compressing or removing color information from the reference image data Gs. In this case, there are advantages in terms of the storage capacity and processing speed of the storage unit 30.

[0078] The information processing system 10 may further include a background information removal unit 20k for removing the background image from the reference image data Gs. In this case, it is possible to suppress a decrease in estimation accuracy due to the background image.

[0079] Next, the second to fourth embodiments of the present invention will be described. In the drawings and descriptions of the second to fourth embodiments, the same or equivalent structural elements and components as those of the first embodiment are marked with the same reference numerals. The descriptions that overlap with the first embodiment will be omitted as appropriate, and the descriptions of the structures that differ from the first embodiment will focus on.

[0080] [Second embodiment]

[0081] A second embodiment of the present invention is an information processing method for a work machine. The information processing method includes the following steps: storing reference image data Gs of a work unit 40 of the work machine 100 and posture data Ks of the work unit 40 of the work machine 100 in a storage unit 30 in a corresponding manner (S72-S75); acquiring image data Gj of the work unit 40 for comparison with the reference image data Gs (S82); and estimating the posture of the work unit 40 of the work machine 100 based on the image data Gj and information stored in the storage unit 30 (S86).

[0082] Alternatively, the estimation step (S86) may include a step of referring to a posture estimation model generated by machine learning based on the reference image data Gs and the posture data Ks. The second embodiment achieves the same effects as the first embodiment.

[0083] [Third embodiment]

[0084] A third embodiment of the present invention is a construction machine 1000. The construction machine 1000 includes: a working unit 40; a storage unit 30 that stores reference image data Gs of the working unit 40 and posture data Ks of the working unit 40 of the working machine 100 in association with each other; an image information acquisition unit 12 that acquires image data Gj of the working unit 40 for comparison with the reference image data Gs; and a posture estimation unit 20e that estimates the posture of the working unit 40 of the working machine 100 based on the image data Gj and information stored in the storage unit 30.

[0085] Alternatively, the construction machine 1000 may be a machine that performs construction work by moving a bucket 46 mounted on an arm mechanism 48. Alternatively, various attachments such as a fork, a hammer, or a crusher may be mounted on the arm mechanism 48 of the construction machine 1000 in place of the bucket. The structure of the third embodiment provides the same operational advantages as the first embodiment.

[0086] [Fourth embodiment]

[0087] A fourth embodiment of the present invention is a computer program P100. This computer program P100 causes a computer to implement the following functions: storing reference image data Gs of the working unit 40 of the working machine 100 and posture data Ks of the working unit 40 of the working machine 100 in the storage unit 30 in correspondence; acquiring image data Gj of the working unit 40 of the working machine 100 for comparison with the reference image data Gs; and estimating the posture of the working unit 40 of the working machine 100 based on the image data Gj and information stored in the storage unit.

[0088] The computer program P100 can be an application program that implements multiple modules corresponding to the functional blocks of the control unit 20 and can be installed in a storage device (e.g., storage unit 30) of the information processing system 10. The computer program P100 can be read into the main memory of a processor (e.g., a CPU) of the information processing system 10 and executed. The structure of the fourth embodiment achieves the same functions and effects as the first embodiment.

[0089] The above describes in detail examples of embodiments of the present invention. The above embodiments are only used to illustrate specific examples of implementing the present invention. The contents of the embodiments are not intended to limit the technical scope of the present invention, and various design changes such as changes, additions, and deletions of structural elements can be made within the scope of the idea of ​​the invention specified in the claims. In the above embodiments, the contents that can be subjected to such design changes are described by marking expressions such as "embodiment" and "in the embodiment", but this does not mean that design changes to contents that do not have such expressions are not allowed.

[0090] [Modification]

[0091] Next, a modified example will be described. In the drawings and description of the modified example, the same or equivalent structural elements and components as those in the embodiment are marked with the same reference numerals. The description that overlaps with the embodiment will be appropriately omitted, and the structure that differs from the first embodiment will be described in detail.

[0092] While the first embodiment illustrates an example in which a posture estimation model is generated by each work machine 100 and stored in the model storage unit 32 of that work machine 100, the present invention is not limited to this. Alternatively, a posture estimation model may be generated by a reference work machine and pre-stored in the model storage unit 32 of each work machine 100. Furthermore, the posture estimation model may be updated at appropriate timings.

[0093] In the first embodiment, the color information removal unit 20n completely removes color information from the reference image data Gs and the image data Gj. However, the present invention is not limited to this embodiment. Alternatively, the color information removal unit 20n may compress the color information of the reference image data Gs and the image data Gj by color reduction or the like.

[0094] While the first embodiment illustrates an example in which the image information acquisition unit 12 is composed of a single image sensor, the present invention is not limited thereto. The image information acquisition unit 12 may also be composed of multiple image sensors. For example, the image information acquisition unit 12 may include a so-called stereo camera.

[0095] While the first embodiment illustrates an example in which the image information acquisition unit 12 is installed on the ceiling of the operator's cab 38, the present invention is not limited thereto. For example, the image information acquisition unit 12 may be located on the side of the operator's cab 38 or on the lid of the upper vehicle body 34. Furthermore, the image information acquisition unit 12 may be located on the working unit 40.

[0096] In the description of the first embodiment, an example is shown in which the working machine 100 is a construction machine that moves the bucket 46 to perform construction work. However, the present invention is not limited to this and is applicable to working machines other than construction machines.

[0097] In the description of the first embodiment, the arm mechanism 48 is shown as an example of being installed on the right side of the control room 38, but the present invention is not limited to this. For example, the arm mechanism may be installed on the left side of the control room or in front of the control room.

[0098] In the description of the first embodiment, an example is shown in which the operator operates the working machine 100 from the operating room 38, but the present invention is not limited thereto. For example, the working machine may be an automatically operated or remotely operated machine.

[0099] The above-described modified example achieves the same operations and effects as those of the first embodiment.

[0100] Any combination of the above-described embodiments and modifications is also useful as an embodiment of the present invention. A new embodiment created by the combination has the effects of the combined embodiments and modifications.

Claims

1. An information processing system comprising: a storage unit that stores data of a reference image of a working unit of the working machine and data of a posture of the working unit of the working machine in association with each other; an acquisition unit that acquires data of an image of a working part of the working machine to compare with data of the reference image; as well as a correction unit that corrects the image data based on at least one of information about the surrounding environment when the image data was acquired, the number of years of the working machine, and individual differences of the working machine, The working unit is defined by changing the extension and contraction length of a hydraulic cylinder, and the data of the posture includes the extension and contraction length of the hydraulic cylinder.

2. An information processing system comprising: a storage unit that stores data of a reference image of a working unit of the working machine and data of a posture of the working unit of the working machine in association with each other; an acquisition unit that acquires data of an image of a working part of the working machine to compare with data of the reference image; as well as an estimating unit that estimates a posture of a working unit of the working machine based on the image data and the storage information of the storage unit, The estimation unit makes the estimation based on at least one of information on the surrounding environment when the image data is acquired, the number of years of the working machine, and individual differences of the working machine. The working unit is defined by changing the extension and contraction length of a hydraulic cylinder, and the data of the posture includes the extension and contraction length of the hydraulic cylinder.

3. The information processing system according to claim 1 or 2, characterized in that: The acquisition unit is configured to rotate integrally with a working unit of the working machine.

4. The information processing system according to claim 1 or 2, characterized in that: The storage unit stores a posture estimation model generated by machine learning based on the data of the reference image and the data of the posture.

5. The information processing system according to claim 1 or 2, characterized in that: The device further includes a color information removing unit configured to compress or remove color information of the reference image data.

6. The information processing system according to claim 1 or 2, characterized in that: The device further includes a background information removing unit configured to remove a background image from the data of the reference image.

7. An information processing system comprising: a storage unit that stores data of a reference image of a working unit of the working machine and data of a posture of the working unit of the working machine in association with each other; an acquisition unit configured to rotate integrally with a working unit of the working machine and acquire image data of the working unit of the working machine to compare with the reference image data; and an estimating unit that estimates a posture of a working unit of the working machine based on the image data and the storage information of the storage unit, The estimation unit makes the estimation based on at least one of information on the surrounding environment when the image data is acquired, the number of years of the working machine, and individual differences of the working machine. The working unit is defined by changing the extension and contraction length of a hydraulic cylinder, and the data of the posture includes the extension and contraction length of the hydraulic cylinder.

8. An information processing method comprising the following steps: storing data of a reference image of a working unit of the working machine and data of a posture of the working unit of the working machine in a storage unit in correspondence with each other; acquiring data of the image of the working part to compare with data of the reference image; as well as estimating a posture of a working unit of the working machine based on the image data and the storage information of the storage unit, In the step of estimating, the estimation is performed based on at least one of information on the surrounding environment when the image data is acquired, the number of years of the working machine, and individual differences of the working machine. The working unit is defined by changing the extension and contraction length of a hydraulic cylinder, and the data of the posture includes the extension and contraction length of the hydraulic cylinder.

9. The information processing method according to claim 8, characterized in that: The estimating step includes a step of referring to a posture estimation model, wherein the posture estimation model is generated by machine learning based on the data of the reference image and the data of the posture.

10. A computer program product for causing a computer to implement the following functions: storing data of a reference image of a working unit of the working machine and data of a posture of the working unit of the working machine in a storage unit in correspondence with each other; acquiring data of an image of a working part of the working machine to compare with data of the reference image; and estimating a posture of a working unit of the working machine based on the image data and the storage information of the storage unit, In the estimating function, the estimation is performed based on at least one of information on the surrounding environment when the image data is acquired, the number of years of the working machine, and individual differences of the working machine. The working unit is defined by changing the extension and contraction length of a hydraulic cylinder, and the data of the posture includes the extension and contraction length of the hydraulic cylinder.

11. A construction machine comprising: The operating part of the operating machine; a storage unit that stores data of a reference image of a working unit of the working machine and data of a posture of the working unit of the working machine in association with each other; an acquisition unit that acquires data of an image of a working part of the working machine to compare with data of the reference image; as well as an estimating unit that estimates a posture of a working unit of the working machine based on the image data and the storage information of the storage unit, The estimation unit makes the estimation based on at least one of information on the surrounding environment when the image data is acquired, the number of years of the working machine, and individual differences of the working machine. The working unit is defined by changing the extension and contraction length of a hydraulic cylinder, and the data of the posture includes the extension and contraction length of the hydraulic cylinder.

Images