Action mastering system

Abstract

An action grasping system includes a control device that time-divisionally operates a feature vector that features information of a machine operating in time series, classifies a plurality of feature vectors into a plurality of clusters and assigns an identification ID, thereby creating a classification model, assigns the identification ID to the plurality of feature vectors operated time-divisionally based on the classification model, establishes a correspondence between a pattern of the identification ID that changes in time series corresponding to a prescribed action of the machine and identification information of the prescribed action, thereby creating an ID pattern model, generates output information for grasping the prescribed action based on the time series of the identification ID and the ID pattern model, and outputs the output information using a display device.

Description

Technical Field

[0001] This invention relates to a motion control system. Background Technology

[0002] In order for machinery to operate efficiently, it is effective to properly control the machinery's movements. As a system for controlling movements, a movement recognition system based on information from multiple acceleration sensors is known (see Patent Document 1).

[0003] In the system described in Patent Document 1, data from multiple acceleration sensors are divided into specified time intervals. Feature vectors are calculated, with similarity to multiple pre-selected standard variation patterns as a factor. The resulting vector, synthesized from these feature vectors, is designated as feature data. The similarity between the feature data and data represented by feature data from sample observations of actions belonging to a motion dictionary is calculated. Labels are assigned to the representative data with high similarity. In the system described in Patent Document 1, each time the feature data is labeled, the label column assigned to the actual observed feature data up to that point is matched with the label column of some actions in the motion dictionary. Sample actions with high likelihood in the motion dictionary are recognized as actions in the current environment.

[0004] Existing technical documents

[0005] Patent documents

[0006] Patent Document 1: Japanese Patent Application Publication No. 2007-187555 Summary of the Invention

[0007] In the system described in Patent Document 1, partial actions contained in the sample actions are extracted from the column of feature data of sample observations, and the extracted partial actions are labeled with natural language names (move, sit down, leave, etc.). In the system described in Patent Document 1, a table for pre-creating labels for assigning natural language names to partial actions needs to be created. However, if a label is set for each partial action, the creation of the table is time-consuming, which may increase the manufacturing cost of the system.

[0008] The purpose of this invention is to provide a motion control system that can reduce manufacturing costs.

[0009] An operation grasping system according to one aspect of the present invention includes a display device that outputs information, and a control device that acquires information of a machine that operates in time series, generates output information for grasping an operation of the machine based on the information of the machine, and outputs the output information using the display device. The control device can execute: classification model creation processing that classifies feature vectors each of which is a feature quantity of the information of the machine that changes in time series, classifies a plurality of the feature vectors each of which is calculated in time series into a plurality of clusters, and assigns an identification ID to the plurality of clusters, thereby creating a classification model; ID pattern model creation processing that classifies feature vectors each of which is a feature quantity of the information of the machine that changes in time series, assigns the identification ID to a plurality of the feature vectors each of which is calculated in time series based on the classification model, and stores a pattern of the identification ID that changes in time series in correspondence with a prescribed operation of the machine in association with identification information of the prescribed operation, thereby creating an ID pattern model; and information output processing that classifies feature vectors each of which is a feature quantity of the information of the machine that changes in time series, assigns the identification ID to a plurality of the feature vectors each of which is calculated in time series based on the classification model, and generates output information for grasping the prescribed operation based on a time series of the identification ID and the ID pattern model, and outputs the output information using the display device.

[0010] Effects of the Invention

[0011] According to the present invention, it is possible to provide an operation grasping system that can reduce manufacturing costs. BRIEF DESCRIPTION OF DRAWINGS

[0012] Figure 1 is a diagram showing the structure of a management system of a work machine.

[0013] Figure 2 is a functional block diagram of an operation grasping system according to a first embodiment of the present invention.

[0014] Figure 3 is a diagram showing an example of a multi-dimensional collection data table.

[0015] Figure 4 is a flowchart showing classification model creation processing executed by a management server.

[0016] Figure 5 is a diagram showing an example of a feature quantity data table.

[0017] Figure 6 is a diagram showing an example of a classification model.

[0018] Figure 7 is a flowchart showing ID pattern model creation processing executed by a management server.

[0019] Figure 8 FIG. 1 is a diagram showing an example of a recognition ID table.

[0020] Figure 9 FIG. 2 is a diagram showing an example of a motion data table.

[0021] Figure 10 FIG. 3 is a diagram explaining ID pattern extraction processing.

[0022] Figure 11 FIG. 4 is a graph showing time variation of a feature quantity and a recognition ID.

[0023] Figure 12 FIG. 5 is a flowchart showing information output processing performed by a management server.

[0024] Figure 13 FIG. 6 is a diagram explaining a feature of DP matching.

[0025] Figure 14 FIG. 7 is a diagram showing a cost between an entry ID pattern corresponding to a motion A ("203" -> "70" -> "203") and a recognition ID column of a recognition ID table.

[0026] Figure 15 FIG. 8 is a diagram showing an example of a cost image displayed on a display screen of a display device.

[0027] Figure 16 FIG. 9 is a diagram showing an output control section of a motion grasping system according to a second embodiment of the present application and data used by the output control section.

[0028] Figure 17 FIG. 10 is a diagram showing an example of an image showing a cumulative occurrence number of a motion displayed on a display screen of a display device.

[0029] Figure 18 FIG. 11 is a diagram showing an example of an image showing a frequency of occurrence of a motion displayed on a display screen of a display device.

[0030] Figure 19 FIG. 12 is a diagram showing an output control section of a motion grasping system according to a third embodiment of the present application and data used by the output control section.

[0031] Figure 20 FIG. 13 is a diagram showing an example of an image displayed on a display screen of a display device of the motion grasping system according to the third embodiment of the present application.

[0032] Figure 21 FIG. 14 is a diagram showing an output control section of a motion grasping system according to a first modification of the third embodiment of the present application and data used by the output control section.

[0033] Figure 22 is a graph showing motion data generated by interpolation when the recognition ID changes in the order of "5"→"7"→"10".

[0034] Figure 23 is a graph showing an example of an image displayed on a display screen of a display device of the action grasping system of Modification 2 of the 3rd embodiment of the present application.

[0035] Figure 24 is a graph showing an output control section of the action grasping system of the 4th embodiment of the present application and data used by the output control section.

[0036] Figure 25 is a functional block diagram of the action grasping system of Modification 1-1.

[0037] Figure 26 is a functional block diagram of the action grasping system of Modification 1-2. DETAILED DESCRIPTION

[0038] An action grasping system of an embodiment of the present application will be described with reference to the drawings.

[0039] <1st Embodiment>

[0040] Referring to Figures 1-15 , an action grasping system 10 of the 1st embodiment of the present application will be described. Figure 1 is a graph showing the structure of a management system 1 of a work machine. The management system 1 includes a management device 105 that manages a work machine 101 that performs work at a work site, and the work machine 101 such as a hydraulic excavator 101A, a dump truck 101B, and a work robot 101C that perform work at the work site.

[0041] The hydraulic excavator 101A is a work machine 101 that is provided with a traveling body 2, a rotating body 3 rotatably provided on the traveling body 2, and a multi-joint type work device 4 mounted on the rotating body 3. The work device 4 has a boom 11, a stick 12, and a bucket 13, and hydraulic cylinders (a boom cylinder 11a, a stick cylinder 12a, and a bucket cylinder 13a) that drive them. The dump truck 101B is a work machine 101 that is provided with a traveling body, and a work device having a vessel provided on the traveling body and loaded with cargo, and a hydraulic cylinder that turns (pivots) the vessel in the up-down direction. The work robot 101C is a work machine 101 that is provided with a work device having a multi-joint type arm and a gripping portion that grips an object.

[0042] The management device 105 is an external device that remotely manages (grasps, monitors) the state of the working machine 101. The management device 105 is provided, for example, at a facility of a head office, a branch, a factory, or the like of a maker of the working machine 101, a rental company of the working machine 101, a data center that exclusively operates a server, a facility of an owner of the working machine 101, or the like.

[0043] In the first embodiment, the management device 105 constitutes an action grasping system 10 that grasps the action of the working machine 101 on the basis of information from the working machine 101. Hereinafter, a case in which the working machine 101 whose action is grasped by the action grasping system 10 is a hydraulic excavator 101A will be mainly described.

[0044] The management system 1 is configured to enable bidirectional communication between the hydraulic excavator 101A that performs work at a work site and the management device 105 provided at a place away from the work site via a communication line 109 of a wide area network. That is, the hydraulic excavator 101A and the management device 105 can transmit and receive information (data) via the communication line 109. The communication line 109 is a mobile phone communication network (mobile communication network) developed by a mobile phone operator or the like, the Internet, or the like. For example, as illustrated, in a case in which the hydraulic excavator 101A and a wireless base station office 108 are connected using a mobile phone communication network (mobile communication network), the wireless base station office 108 transmits received information to the management device 105 via the Internet if the wireless base station office 108 receives predetermined information from the hydraulic excavator 101A.

[0045] The hydraulic excavator 101A is provided with a vehicle body controller 110 that controls each part of the hydraulic excavator 101A as a control device, an input device 120 that inputs predetermined information to the vehicle body controller 110, a plurality of sensors 121 to 125, and a communication device 128. The communication device 128 is a wireless communication device that can perform wireless communication with the wireless base station office 108 connected to the communication line 109, and has a communication interface including, for example, a communication antenna that sets a bandwidth of a 2.1 GHz band or the like as a sensitive bandwidth. The communication device 128 exchanges information with the management server 150 or the like via the wireless base station office 108 and the communication line 109.

[0046] The management device 105 is provided with the management server 150, an input device 161 for inputting prescribed information to the management server 150, a communication device 162 that receives information transmitted from the hydraulic excavator 101A (information on the hydraulic excavator 101A that changes in time series along with operation), and a display device 165 as an output device that outputs prescribed information based on a control signal from the management server 150. The display device 165 is, for example, a liquid crystal display device that displays a display image of information on the state of the hydraulic excavator 101A on a display screen 165a based on a display control signal from the management server 150.

[0047] The management server 150 functions as a control device that acquires information on the hydraulic excavator 101A that changes in time series, generates output information for grasping the action of the hydraulic excavator 101A based on the information on the hydraulic excavator 101A, and causes the display device (output device) 165 to output the output information by controlling the display device (output device) 165.

[0048] The vehicle body controller 110 and the management server 150 are constituted by a computer, and are provided with a CPU (Central Processing Unit) 111, 151 as a working circuit, a ROM (Read Only Memory) 112, 152 as a storage device, a RAM (Random Access Memory) 113, 153 as a storage device, an input interface 114, 154, an output interface 115, 155, and other peripheral circuits. In addition, the management server 150 and the vehicle body controller 110 can each be constituted by one computer, or can be constituted by a plurality of computers.

[0049] The ROM 112, 152 is a nonvolatile memory such as an EEPROM, and stores programs that enable various operations. That is, the ROM 112, 152 is a storage medium that can read programs that realize the functions of the present embodiment. The RAM 113, 153 is a volatile memory, and is a work memory that directly performs input and output of data between the CPU 111, 151. The RAM 113, 153 temporarily stores necessary data during the operation execution of the programs by the CPU 111, 151. The management server 150 and the vehicle body controller 110 also have an HDD (Hard Disk Drive) 116, 156 as a storage device. The HDD 116, 156 is a nonvolatile large-capacity storage device that stores various information (data). Furthermore, in the following description, an example in which various data (multidimensional collection data table 117, action data table 118, classification model 175, ID pattern model 184) is stored in the HDD 116, 156 is described, but various storage devices such as an SSD (Solid State Drive), a flash memory, or the like can be used instead of the HDD 116, 156.

[0050] The CPU 111, 151 is a processing device that expands and performs operations on the control programs stored in the ROM 112, 152 in the RAM 113, 153, and performs prescribed operation processing on information (data) taken in from the input interface 114, 154 and the ROM 112, 152, RAM 113, 153, HDD 115, 156, and the like in accordance with the control programs. Signals from various devices are input to the input interface 114, 154. The input interface 114, 154 converts the input signals so that they can be operated by the CPU 111, 151. The output interface 115, 155 generates signals for output corresponding to the operation results in the CPU 111, 151, and outputs the signals to various devices.

[0051] Figure 2 is a functional block diagram of the action grasping system 10 of the first embodiment of the present application. As described above, in the first embodiment, the management device 105 functions as the action grasping system 10. The action grasping system 10 acquires information collected by the hydraulic excavator 101A (hereinafter, also referred to as collection information), and performs processing for grasping the action of the hydraulic excavator 101A that changes in time series.

[0052] The collection information of the hydraulic excavator 101A includes sensor information detected by various sensors 121 to 125 possessed by the hydraulic excavator 101A, and control information of the body controller 110 of the hydraulic excavator 101A. The body controller 110 repeatedly acquires the sensor information and the control information of the hydraulic excavator 101A at a prescribed sampling cycle, stores them to the HDD 116, and creates a multi-dimensional collection data table 117. The multi-dimensional collection data table 117 is composed of a plurality of kinds of data (multi-dimensional collection information) that change in time series.

[0053] Figure 3 is a view that shows an example of the multi-dimensional collection data table 117. As shown in Figure 3 , the multi-dimensional collection data table 117 is composed of the collection information of the hydraulic excavator 101A that changes in time series. Specifically, the multi-dimensional collection data table 117 is created by storing the multi-dimensional collection information acquired at the same time, at the same sampling cycle, to the HDD 116 together with the time of acquisition in time series. Further, for the sake of easy understanding, the sampling cycle is set to 1 [sec], but in reality, it is several [msec] to several tens [msec] or so.

[0054] As shown in Figure 3 , among the plurality of collection information that constitutes the multi-dimensional collection data table 117, as the sensor information, for example, the following explained boom cylinder 11a's cylinder bottom pressure Pb, boom cylinder 11a's cylinder rod pressure Pr, boom operation amount Lb, boom angle a, and outside air temperature To are included.

[0055] As shown in Figure 1 , the boom cylinder 11a is a hydraulic cylinder that drives the boom 11. The hydraulic cylinder is provided with a bottomed cylindrical cylinder tube whose one end is plugged, a head cover that seals the opening of the other end of the cylinder tube, a cylinder rod that penetrates the head cover and is inserted into the cylinder tube, and a piston that is provided at the top end of the cylinder rod and divides the inside of the cylinder tube into a cylinder rod side chamber and a cylinder bottom side chamber. The one end side of the boom cylinder 11a is connected to the boom 11, and the other end side is connected to the frame of the swing body 3.

[0056] The hydraulic excavator 101A is provided with a pressure sensor (cylinder bottom pressure sensor) 121 that detects the pressure of the cylinder bottom side chamber of the boom cylinder 11a. The body controller 110 acquires the cylinder bottom pressure Pb of the boom cylinder 11a detected by the cylinder bottom pressure sensor 121. The hydraulic excavator 101A is provided with a pressure sensor (cylinder rod pressure sensor) 122 that detects the pressure of the cylinder rod side chamber of the boom cylinder 11a. The body controller 110 acquires the cylinder rod pressure Pr of the boom cylinder 11a detected by the cylinder rod pressure sensor 122. Further, although not shown, the body controller 110 likewise acquires the cylinder rod pressure and the cylinder bottom pressure of the arm cylinder 12a and the bucket cylinder 13a.

[0057] The hydraulic excavator 101A has a boom operation lever that operates the boom 11 (boom cylinder 11a), and a boom operation sensor 123 that detects an operation amount of the boom operation lever. The body controller 110 acquires the boom operation amount detected by the boom operation sensor 123. Further, although not shown, the body controller 110 likewise acquires operation amounts of the arm 12 (arm cylinder 12a) and the bucket 13 (bucket cylinder 13a).

[0058] The hydraulic excavator 101A has a boom angle sensor 124 that detects a rotation angle of the boom 11 with respect to the swing body 3. The body controller 110 acquires the boom angle a detected by the boom angle sensor 124. Further, although not shown, the body controller 110 likewise acquires a rotation angle of the arm 12 with respect to the boom 11 (arm angle) and a rotation angle of the bucket 13 with respect to the arm 12 (bucket angle). The sensors that detect the boom angle a, the arm angle, and the bucket angle constitute a posture sensor for detecting a posture of the working device 4.

[0059] The hydraulic excavator 101A has a temperature sensor 125 that detects an outside air temperature To. The body controller 110 acquires the outside air temperature To detected by the temperature sensor 125. Further, although not shown, the body controller 110 acquires a temperature of engine cooling water and a temperature of working oil.

[0060] Furthermore, although not shown, among the plurality of information that constitute the multi-dimensional collection data table 117, control information for controlling the hydraulic excavator 101A is included. In the control information, for example, a target value of a discharge capacity of a hydraulic pump (a control signal for controlling the hydraulic pump) is included. The hydraulic excavator 101A is provided with a hydraulic pump that is driven by an engine and supplies working oil as a working fluid to hydraulic actuators such as a hydraulic cylinder of the working device 4, a hydraulic motor for rotation of the swing body 3, and a hydraulic motor for travel of the traveling body 2, and a flow control valve that controls a direction and a flow rate of the working oil supplied from the hydraulic pump to the hydraulic actuators.

[0061] The flow control valve acts in accordance with an operation of a lever. For example, a flow control valve that controls a flow of the working oil supplied to the boom cylinder 11a acts in accordance with an operation of the boom operation lever. By the action of the flow control valve, the working oil discharged from the hydraulic pump is introduced to a cylinder bottom side chamber (or a cylinder rod side chamber) of the boom cylinder 11a, and the working oil is discharged from the cylinder rod side chamber (or the cylinder bottom side chamber) to a tank, whereby the boom cylinder 11a is elongated (or contracted). Thus, the boom 11 is rotated upward (or downward).

[0062] The hydraulic pump includes a swashplate mechanism (not shown) with a pair of input / output ports, and a regulator that adjusts the discharge capacity (discharge volume) by adjusting the swashplate's tilt angle. The body controller 110 calculates a target value for the hydraulic pump's discharge capacity based on the pump's discharge pressure detected by a discharge pressure sensor. Furthermore, the discharge pressure sensor is located between the hydraulic pump and the flow control valve. The body controller 110 outputs a control signal to the regulator to make the hydraulic pump's discharge capacity reach the target value. Additionally, the body controller 110 stores the target value for the hydraulic pump's discharge capacity as part of the control information in the HDD 116.

[0063] As described above, the body controller 110 acquires sensor information and control information from sensors installed on the measuring parts of the hydraulic excavator 101A in a timing sequence and stores it in the HDD 116. Furthermore, the body controller 110 can also acquire sensor information from the hydraulic excavator 101A from other controllers connected via an onboard network such as CAN (Controller Area Network) and store it in the HDD 116. For example, the body controller 110 can also acquire information detected by the engine speed sensor from the engine controller via the onboard network and store it in the HDD 116.

[0064] like Figure 2 As shown, the body controller 110 of the hydraulic excavator 101A functions as a data transmission unit 129 by executing the program stored in the ROM 112. The data transmission unit 129 transmits the multidimensional data collection table 117 stored in the HDD 116 to the management unit 105 via the communication device 128.

[0065] The management server 150 uses the communication device 162 to acquire the multidimensional data collection table 117 from the hydraulic excavator 101A via the communication line 109 and stores it in the HDD 156. The management server 150 of the motion mastering system (management device) 10 processes the acquired multidimensional data collection table 117, the pre-created classification model 175, and the pre-created ID style model 184 for mastering (cognizing) motion, and displays the results on the display screen 165a of the display device 165.

[0066] The management server 150 of the motion control system (management device) 10 functions as a data acquisition unit 171, a feature vector generation unit 172, a classification unit 173, an identification ID assignment unit 174, an identification ID column generation unit 181, an ID pattern extraction unit 182, a similarity calculation unit 191, and a display control unit 192 by executing the program stored in the ROM 152. The creation process of the classification model 175, the creation process of the ID pattern model 184, and the motion control process will be described in detail below.

[0067] First, the creation processing of the classification model 175 based on the management server 150 will be described in detail with reference to Figures 2-6 The management server 150 can execute the classification model creation processing of classifying the feature vectors V, each of which is set as a feature amount with the information of the hydraulic excavator 101A in time series, into a plurality of clusters, and assigning an identification ID to the plurality of clusters, thereby creating the classification model 175.

[0068] As shown in Figure 2 , the data acquisition section 171 acquires the multi-dimensional collection data table 117 from the hydraulic excavator 101A with the communication device 162, and stores it to the HDD 156 as a multi-dimensional collection data table 117a for learning. The feature vector generation section 172 creates a feature amount data table 176 based on the multi-dimensional collection data table 117a for learning (refer to Figure 5 ). The feature amount data table 176 is a data table composed of a plurality of feature vectors V after time division. The classification section 173 classifies the plurality of feature vectors V after time division into a plurality of clusters. The identification ID assignment section 174 assigns an identification ID to a representative feature vector of each of the plurality of clusters classified by the classification section 173. Thus, the classification model 175 is created. That is, the feature vector generation section 172, the classification section 173, and the identification ID assignment section 174 function as a classification model creation section 170 that creates the classification model 175 based on the multi-dimensional collection data table 117a.

[0069] Figure 4 is a flowchart showing the classification model creation processing executed by the management server 150. Figure 4 The flowchart shown in

[0070] As shown in Figure 4 , in step S110, the data acquisition section 171 acquires the multi-dimensional collection data table 117 from the hydraulic excavator 101A via the communication line 109, stores it to the HDD 156 as a multi-dimensional collection data table 117a for learning, and proceeds to step S115.

[0071] The multi-dimensional collection data table 117 acquired as the multi-dimensional collection data table 117a for learning is desirably created in a manner capable of grasping information required for mastering the operation of the hydraulic excavator 101A assumed in the work site. For this reason, the multi-dimensional collection data table 117 is created using information collected by previously causing the hydraulic excavator 101A to operate various work contents in various situations assumed in a test field or the like.

[0072] In step S115, the feature vector generation unit 172 converts the specified information constituting the multidimensional collected data table 117a obtained in step S110 into feature quantities, and creates a feature quantity data table 176 (see reference). Figure 5 In sensor information (physical quantities), there are cases where converting it to other information (physical quantities) facilitates the classification process described later. Furthermore, when sensor information contains noise components, removing the noise components can improve the accuracy of the classification process. Therefore, for given collected information, it is preferable to convert it into other information and set the converted value as a feature quantity.

[0073] Figure 5 This is a diagram representing an example of feature data table 176. Specifically, the feature vector generation unit 172 is based on the multidimensional collected data table 117a (see reference). Figure 3 The feature vector generation unit 172 collects the specified information (physical quantity), calculates other information (physical quantity), sets the calculated information (physical quantity) as feature quantities, and stores it in HDD156. The feature vector generation unit 172 creates a feature quantity data table 176 by storing the feature quantity in correspondence with the time when the collection information used to calculate the feature quantity is acquired in the hydraulic excavator 101A.

[0074] For example, the feature vector generation unit 172 calculates the cylinder output (cylinder thrust) Fb of the boom cylinder 11a at the same time (time 12:25:32) based on the cylinder bottom pressure Pb and the cylinder rod pressure Pr of the boom cylinder 11a obtained at time 12:25:32. The cylinder output Fb is calculated by subtracting the value obtained by multiplying the cylinder bottom pressure Pb of the boom cylinder 11a by the pressure area of ​​the piston on the cylinder bottom side from the value obtained by multiplying the cylinder rod pressure Pr of the boom cylinder 11a by the pressure area of ​​the piston on the cylinder rod side. Furthermore, it is preferable that the feature vector generation unit 172 performs filtering processing to remove noise components from the cylinder output Fb.

[0075] Furthermore, the feature vector generation unit 172 performs frequency analysis based on the timing waveform of the cylinder output Fb, sets the dominant frequency f at that moment as a feature quantity, and stores it in the HDD 156. In addition, the boom angle α and the outside air temperature To are directly stored in the HDD 156 as feature quantities. That is, the feature vector generation unit 172 directly sets the specified collection information (e.g., boom angle α, outside air temperature To) constituting the multidimensional collection data table 117 as feature quantities and stores it in the HDD 156, and converts multiple specified collection information constituting the multidimensional collection data table 117 (e.g., cylinder bottom pressure Pb, cylinder rod pressure Pr) into other information (physical quantities), sets the converted values ​​as feature quantities, and stores them in the HDD 156, thereby creating the feature quantity data table 176.

[0076] Furthermore, if there is an imbalance in the magnitude of each feature quantity, the classification process described later (step S125) may not be performed appropriately. An imbalance occurs, for example, when feature quantity A varies within the range of 0 to 1, while feature quantity B varies within the range of 0 to 100. If such an imbalance exists, the weight of feature quantity B becomes larger compared to feature quantity A. As a result, in the classification process described later (step S125), the influence of feature quantity B becomes larger, and the influence of feature quantity A becomes smaller. Therefore, it is preferable that the feature vector generation unit 172 performs a scaling transformation process to make the weights of each feature quantity uniform. The scaling transformation process is, for example, a normalization process based on the minimum and maximum values ​​of the collected information constituting the multidimensional collected data table 117. Furthermore, to improve the accuracy of the classification process described later (step S125), the weights may be adjusted appropriately.

[0077] If the creation process of feature data table 176 (step S115) is completed, the feature vector generation unit 172 proceeds to step S120. In step S120, the feature vector generation unit 172 generates the feature data table (refer to step S115) from the feature data table generated in step S115. Figure 5 The eigenvector V is then separated. The eigenvector V is as follows: Figure 5 As shown by the dashed line t2a, this is obtained by separating the multidimensional feature quantities (Fb, f, α, To) that constitute the feature quantity data table 176 at a specified time. The feature vector generation unit 172 separates the multidimensional feature quantities horizontally according to time, thereby separating the feature vector V(Fb, f, α, To). The feature vector V(Fb, f, α, To) shows the number of elements (in Figure 5 In the example shown, there are 4 (these are the position coordinates of the feature space of dimension 4). Hereafter, these position coordinates will also be denoted as "feature position".

[0078] In this way, the feature vector generation unit 172 performs time-sharing calculations to set the information of the hydraulic excavator 101A that changes over time as the feature vector V of the feature quantity (steps S115, S120). If the separation process based on the feature vector V of the feature vector generation unit 172 (step S120) is completed, then step S125 is performed.

[0079] In step S125, the classification section 173 performs classification processing of classifying the plurality of feature vectors V calculated in the time division into a plurality of clusters. The classification processing of the feature vectors V is processing of classifying groups in which the feature amount positions are close into one set (cluster). The feature amount positions being close means that the elements (feature amounts) of the feature vectors V are similar values, the same state as the hydraulic excavator 101A. Among methods of classification, there are various methods, but in the present embodiment, the k-means method, which is one of clustering methods, is used. The k-means method is a method of classifying multidimensional data without teaching, and is a method of finding clusters (blocks) of data by regarding each input data as a point in a multidimensional space and taking the Euclidean distance of each point as a reference.

[0080] The classification section 173 performs clustering processing based on the k-means method on the feature vectors V of each time divided in step S120. The classification section 173 first randomly allocates all of the feature amount positions to the k clusters prescribed in advance, and calculates the barycentric coordinate values with respect to the feature amount positions allocated to each cluster.

[0081] Next, the classification section 173 calculates the distances between the calculated barycentric coordinate values and all of the feature amount positions, and allocates each feature amount position to the cluster of the barycentric coordinate value closest in distance. The classification section 173 calculates the barycentric coordinate values with respect to the feature amount positions allocated to each cluster. The classification section 173 repeatedly performs the allocation of the clusters and the calculation of the barycentric coordinate values until the barycentric coordinate values no longer change. If the barycentric coordinate values no longer change, the classification section 173 determines that the classification is complete, ends the clustering processing, and proceeds to step S130.

[0082] In step S130, the recognition ID assigning section 174 generates recognition IDs (i = 1, 2, 3,..., k) for recognizing the barycentric coordinate values of each cluster, and assigns the recognition IDs to the barycentric coordinate values of each cluster. The recognition ID assigning section 174 creates the classification model 175 by storing the barycentric coordinate values and the recognition IDs for recognizing the barycentric coordinate values in correspondence with each other in the HDD 156.

[0083] Figure 6 is a diagram showing an example of the classification model 175 created by the classification model creating section 170. The classification model 175 is constituted by the recognition IDs and the barycentric coordinate values (i.e., representative values of each feature amount) stored in correspondence with the recognition IDs. Further, the barycentric coordinate values are represented by representative feature vectors Vi (i = 1, 2, 3,..., k). For example, the barycentric coordinate value of the recognition ID "1" is represented by the representative feature vector V1 (3.0, 20.2, 45, 20.5). If the recognition ID assigning processing (step S130) is complete, the processing shown in the flowchart of Figure 4 is complete.

[0084] Referring to Figure 2 , Figures 6-11 , the creation processing of the ID pattern model 184 based on the management server 150 will be described in detail. The management server 150 can execute the ID pattern model creation processing of performing time division operation on a feature vector V that features information of the hydraulic excavator 101A that changes in time series, assigning an identification ID to the plurality of feature vectors V that are time division operated based on the classification model 175, and storing a pattern of the identification ID that changes in time series in correspondence with the prescribed action of the hydraulic excavator 101A in correspondence with the identification information of the prescribed action, thereby creating the ID pattern model 184.

[0085] As shown in Figure 2 , the data acquisition section 171 acquires the multidimensional collection data table 117 from the hydraulic excavator 101A and stores it to the HDD 156 as a learning-use multidimensional collection data table 117b. In addition, the data acquisition section 171 acquires the action data table 118 from the hydraulic excavator 101A and stores it to the HDD 116. The feature vector generation section 172 creates a feature quantity data table 176 based on the learning-use multidimensional collection data table 117b (refer to Figure 5 ). The identification ID column generation section 181 creates an identification ID table 185 that is a data table of an identification ID column in which identification IDs are arranged in time series based on the feature quantity data table 176 and the classification model 175 (refer to Figure 8 ). The ID pattern extraction section 182 extracts, from the identification ID table 185, identification ID columns within an interval (action start time and action end time) of a prescribed action as an identification ID pattern of the prescribed action based on the action data table 118 and assigns an action name that is identification information for identifying the action. The ID pattern extraction section 182 creates the ID pattern model 184 by storing the extracted identification ID pattern in correspondence with the action name. That is, the feature vector generation section 172, the identification ID column generation section 181, and the ID pattern extraction section 182 function as an ID pattern model creation section 180 that creates the ID pattern model 184 based on the multidimensional collection data table 117b, the classification model 175, and the action data table 118.

[0086] Figure 7 is a flowchart showing the ID pattern model creation processing executed by the management server 150. Figure 7 The flowchart shown in

[0087] Figure 7 The processing of steps S110 to S120 is processing of creating the ID pattern model 184 based on the management server 150. The processing of steps S130 to S140 is processing of creating the ID pattern model 184 based on the management server 150. Figure 4The processing in steps S110 to S120 is the same, so the explanation is omitted. Furthermore, in step S110, the multidimensional data collection table 117b obtained as the learning data collection table 117 can be set to be the same as in... Figure 2 The data obtained in step S110 for the multidimensional data collection table 117a used for learning can be the same as that obtained in multidimensional data collection table 117, but can also be different. For example, for the creation of classification model 175, multidimensional data collection table 117 collected by performing experimental actions of hydraulic excavator 101A in a test field can be used; for the creation of ID style model 184, multidimensional data collection table 117 collected by the actions of hydraulic excavator 101A actually performing operations in a work site can be used. Figure 7 As shown, if the separation of feature vector V (step S120) is completed, then proceed to step S240.

[0088] In step S240, the ID column generation unit 181 identifies the classification model 175 (referencing...). Figure 6 The process involves extracting a representative feature vector Vi (i = 1 to k) that is most similar to the feature vector V extracted in step S120, and storing the identification ID of the extracted representative feature vector Vi in correspondence with the time of feature vector V. Specifically, the identification ID column generation unit 181 stores the feature quantity position (refer to the feature vector V at a specified time ti) extracted in step S120. Figure 5 ) and all the centroid coordinates represented by the representative feature vector Vi of classification model 175 (refer to Figure 6 The distance between the two values ​​in the feature space is calculated, and the identification ID of the nearest centroid coordinate value is correspondingly stored with the specified time ti. In step S240, the identification ID column generation unit 181 calculates the distance between the two values ​​in the feature data table 176 (refer to...). Figure 5 Set the identification ID for all times, and create an identification ID table 185 consisting of identification ID columns arranged in chronological order (see reference). Figure 8 Then proceed to step S245. In this way, the identification ID column generation unit 181 assigns identification IDs to multiple feature vectors V calculated in time-division based on the classification model 175, thereby creating an identification ID table 185.

[0089] Figure 8 This is a diagram illustrating an example of ID table 185. (Example) Figure 8 As shown, the identification ID table 185 is a data table that establishes a correspondence between time and identification ID. Although not illustrated, the identification ID table 185 is stored in HDD 156.

[0090] like Figure 7As shown, if the creation process of ID table 185 (step S240) is completed, the process proceeds to step S245. In step S245, the data acquisition unit 171 acquires motion data table 118 (see reference) from the hydraulic excavator 101A via communication line 109. Figure 9 ), store it in HDD156, and proceed to step S250.

[0091] Figure 9 This is a diagram representing an example of motion data table 118. (For example...) Figure 9 As shown, action data table 118 is a data table that establishes a correspondence between action name and the start and end times of the action.

[0092] Action data table 118 is created together with multidimensional data collection table 117 (117b) using the body controller 110 of the hydraulic excavator 101A. The operator of the hydraulic excavator 101A inputs an "action name (e.g., action A)" into the body controller 110 by performing a prescribed operation (action name input operation) on the input device 120. Then, when starting the action corresponding to the "action name (e.g., action A)", the operator of the hydraulic excavator 101A performs a prescribed operation (start time input operation) on the input device 120. As a result, the body controller 110 sets this time as the "start time" and stores it in HDD 116 in correspondence with the "action name (e.g., action A)". When ending the action corresponding to the "action name (e.g., action A)", the operator of the hydraulic excavator 101A performs a prescribed operation (end time input operation) on the input device 120. Therefore, the body controller 110 sets the current time as the "end time" and stores it in the HDD 116 in correspondence with the "action name (e.g., action A)".

[0093] The action name input operation is, for example, selecting an action name from a plurality of action names displayed on the touch panel (input device 120), or inputting a string by operating the keyboard (input device 120). The start time input operation and the end time input operation are, for example, touching the touch panel (input device 120), or pressing a switch (input device 120). In this way, by the prescribed operation performed by the operator of the hydraulic excavator 101A, the body controller 110 establishes a corresponding storage of the "action name", "start time" and "end time", thereby creating the action data table 118.

[0094] Further, the action data table 118 can also not be created together with the creation of the multidimensional collection data table 117, but can be created after the creation of the multidimensional collection data table 117 is completed, while the multidimensional collection data table 117 is observed. The action name is a natural language name given to a series of state changes of the hydraulic excavator 101A. For example, as an action name corresponding to an action of excavating sand until the sand is loaded into a dump truck, "excavation and loading action" is input. Alternatively, the "excavation and loading action" can be split into "excavation action" and "loading action" and set as an action name corresponding to each action.

[0095] As shown in Figure 7 , if the action data table acquisition processing (step S245) is completed, step S250 is entered. In step S250, the ID pattern extraction section 182 refers to the action data table (refer to Figure 9 ) and performs ID pattern extraction processing, that is, extracts an identification ID column composed of a plurality of identification IDs within an interval in which a prescribed action is performed as an identification ID pattern, from an identification ID column (refer to Figure 8 ) composed of all identification IDs arranged in time series.

[0096] Figure 10 is a diagram illustrating the ID pattern extraction processing. As shown in Figure 10 , in the ID pattern extraction processing (step S250) of Figure 7 , the ID pattern extraction section 182 refers to the action data table 118 (refer to Figure 9 ) and extracts a plurality of identification IDs from the "start time" to the "end time" in time series by the "action name" from the identification ID table 185. Further, the ID pattern extraction section 182 leaves only one identification ID in a case where a plurality of the same identification ID is continuously extracted. This is because the repeated identification ID is ignored in the DP matching method described later.

[0097] For example, the ID pattern extraction section 182 refers to the action data table 118 and extracts a plurality of identification IDs contained in the action interval (start time 12:25:33 to end time 12:25:39) of the action A in time series. Further, one identification ID is extracted for a plurality of the same identification ID in succession. Therefore, the ID pattern extraction section 182 extracts "203", "70", "203" as the identification ID pattern within the action interval of the action A. The ID pattern extraction section 182 stores the identification ID pattern extracted in time series in association with "action A" as the action name.

[0098] The ID pattern extraction section 182 stores the recognition ID pattern corresponding to each of the action names (action A, action B, action C, ···) stored in the action data table 118 in the HDD 156, thereby creating an ID pattern model 184. When the ID pattern extraction processing (step S250) is completed, the processing shown in the flowchart is ended. In this way, the ID pattern extraction section 182 stores the pattern of the recognition ID that changes in time series corresponding to the prescribed action of the hydraulic excavator 101A in correspondence with the action name (recognition information) of the prescribed action, thereby creating the ID pattern model 184. In the following description, the recognition ID pattern stored (registered) in correspondence with the action name is also referred to as a registered ID pattern. Further, in the following description, the registered ID pattern is also referred to as the recognition ID pattern. In the following description, the recognition ID pattern is also referred to as the ID pattern. Figure 7 In the following description, the registered ID pattern of each action is described as an example in which the registered ID pattern is composed of three elements (for example, the registered ID pattern of the action A is composed of the three elements of "203", "70", and "203"). However, the registered ID pattern can be composed of two elements or four or more elements depending on the action. Figure 10

[0099] Figure 11 is a graph showing the time change of the feature amount and the recognition ID. As shown in Figure 11 , the recognition ID changes as the values of the respective feature amounts A to F change depending on the passage of time. The change in the recognition ID indicates the change in the state of the hydraulic excavator 101A. In the section in which the action A is performed, the recognition ID changes from "203" to "70" to "203".

[0100] The information output processing performed by the management server 150 is described in detail with reference to Figure 2 , Figures 12-15 . The management server 150 can perform the information output processing of calculating the feature vectors V in which the information of the hydraulic excavator 101A that changes in time series is set as the feature amount, assigning the recognition ID to the plurality of feature vectors V calculated on a time division basis on the basis of the classification model 175, generating the output information for grasping the prescribed action on the basis of the time series change in the recognition ID and the ID pattern model 184, and outputting the generated output information using the display device 165 as the output device.

[0101] As shown in Figure 2 , the data acquisition section 171 acquires the multi-dimensional collection data table 117 from the hydraulic excavator 101A and stores it as the action grasping use multi-dimensional collection data table 117c of the hydraulic excavator 101A in the HDD 156. The feature vector generation section 172 generates the feature amount data table 176 on the basis of the action grasping use multi-dimensional collection data table 117c (refer to Figure 5 ​). The recognition ID column generation section 181 generates a recognition ID table 185 (refer to FIG. 18) in which recognition IDs are arranged in time series, based on the feature amount data table 176 and the classification model 175. The recognition ID table 185 is a data table in which the recognition ID column is generated. Figure 14 The similarity operation section 191 operates the similarity between the recognition ID column of the recognition ID table 185 and the input ID pattern of the ID pattern model 184. The display control section 192 generates information (data) of a display image that represents the similarity operated by the similarity operation section 191, and outputs to the display device 165. The display device 165 causes the display screen 165a of the display device 165 to display a display image that represents the similarity, based on the input information (data) of the display image. That is, the display control section 192 controls the display device (output device) 165 to output, using the display device (output device) 165, output information (in the present embodiment, information of a display image that represents the similarity of the action) that is used for grasping the prescribed action of the hydraulic excavator 101A. In this way, the feature vector generation section 172, the recognition ID column generation section 181, the similarity operation section 191, and the display control section 192 function as an output control section 190 that generates output information for grasping the action of the hydraulic excavator 101A based on the multidimensional collected data table 117c, the classification model 175, and the ID pattern model 184, and outputs using the output device.

[0102] Figure 12 is a flowchart that represents the information output processing performed by the management server 150. Figure 12 The flowchart illustrated in

[0103] As illustrated in Figure 12 In step S110, the data acquisition section 171 acquires the multidimensional collected data table 117 from the hydraulic excavator 101A via the communication line 109, and stores it as the action-grasping multidimensional collected data table 117c to the HDD 156, and proceeds to step S115.

[0104] In step S110, the multidimensional collected data table 117 acquired as the action-grasping multidimensional collected data table 117c is the same as the one acquired in Figure 2 step S110 of Figure 7the multi-dimensional collection data table 117 acquired in step S110 as learning use multi-dimensional collection data tables 117a, 117b, and the multi-dimensional collection data table 117 acquired as the multi-dimensional collection data table 117c for action grasping is created in order to grasp the action of the hydraulic excavator 101A that actually performs various work in the work site. That is, the multi-dimensional collection data table 117c for action grasping is a data table created using information collected during the period in which work is performed in the work site.

[0105] Figure 12 The processing of steps S115, S120 is the same processing as the processing of steps S115 to S120 of Figure 4 The processing of step S240 is the same processing as the processing of step S240 of Figure 12 The processing of step S240 is the same processing as the processing of step S240 of Figure 7 The processing of step S240 is the same processing as the processing of step S240 of Figure 12 As shown in FIG. 36, if the creation processing of the recognition ID table 185 (step S240) is completed, the processing proceeds to step S360.

[0106] In step S360, the similarity degree calculation section 191 calculates, for each "action name" registered in the ID pattern model 184, a cost that indicates the similarity degree between the recognition ID column of the recognition ID table 185 and the registered ID pattern of the ID pattern model 184.

[0107] In the present embodiment, DP matching is performed between the registered ID pattern of the ID pattern model 184 and the recognition ID column of the recognition ID table 185, and the cost is calculated. DP matching is one of flexible matching techniques that match patterns by non-linearly stretching the patterns in order to consider the stretching of the patterns on the time axis. Further, since DP matching is a general technique, the detailed algorithm is not described here. In a machine operated by a person such as the hydraulic excavator 101A, there is a case where a part of a pattern is stretched on the time axis. Since DP matching is one of flexible matching techniques, it is possible to evaluate the similarity even in the case where a part of a pattern is stretched on the time axis, and thus is appropriate.

[0108] Figure 13 is a diagram that explains the characteristics of DP matching. As shown in (a) of Figure 13 is a diagram that explains the characteristics of DP matching. As shown in (a) of Figure 13 As shown in (b) of FIG. 33, in DP matching, the recognition ID column in which the recognition ID is continuously repeated (for example, the recognition ID column in which the recognition IDs "5" and "7" are continuously repeated) and the recognition ID column in which the recognition ID is not repeated are processed in the same manner.

[0109] Similarity is represented by the cost of moving from the start point to the end point of the style. Figure 13 (c)~ Figure 13 (e) shows an example of cost operation. Figure 13 In the example shown in (c), the identification ID column is "5" → "7" → "10", which is consistent with... Figure 13 The ID entry styles shown in (a) are completely identical, therefore the cost is the minimum value C0 (=0.0). Figure 13 In the example shown in (d), the identification ID column contains Figure 13 The ID "8" is not included in the input ID pattern shown in (a). Therefore, the cost of the operation becomes C1 (C1 > C0), which is greater than the minimum value C0. Figure 13 In the example shown in (e), the identification ID column is replaced Figure 13 The ID "7" of the input ID pattern "5" → "7" → "10" shown in (a) is set to ID "9". Therefore, the cost of the operation becomes C2 (C2 > C1 > C0) which is greater than the minimum values ​​C0 and C1.

[0110] In this way, regarding the cost, a low value indicates high similarity, and a high value indicates low similarity. In addition, the minimum value of the cost is 0 (zero), which means that the identification ID column is consistent with the identification ID style (entered ID style) entered in ID style model 184.

[0111] As a method for calculating the cost, for example, a cost calculation method based on comparing the identification ID column with the input ID pattern (hereinafter referred to as calculation method A) and a cost calculation method based on the distance between the feature positions of the identification ID (hereinafter referred to as calculation method B) can be adopted.

[0112] In calculation method A, the identification ID column and the entered ID pattern are compared. If they match, the cost is set to 0 (zero). If they do not match, the cost is set to a value greater than 0 (zero) based on a predefined condition. Furthermore, since the cumulative cost at the start and end points differs depending on the number of elements in the entered ID pattern, it can also be normalized based on the number of elements.

[0113] In calculation method A, the distance between identified IDs in the feature space is not considered. Therefore, for example, in Figure 13 In the example shown in (e), the identification ID "9" in the identification ID column and the identification ID "7" in the input ID pattern have the same cost whether they are close or far apart in the Euclidean distance in the feature space.

[0114] On the other hand, in the calculation method B, the cost is calculated based on the distance between the feature amount position corresponding to the recognition ID of the recognition ID table 185 and the feature amount position corresponding to the recognition ID of the input ID pattern. As indicated by the dotted arrow, in the case where the similarity is calculated using the calculation method B, the similarity calculation section 191 acquires the feature amount position corresponding to the recognition ID of the input ID pattern from the classification model 175. In the case where the distance is short in the feature amount space, the cost becomes low, and in the case where the distance is long, the cost becomes high. Further, in order to separate the close feature amounts, the cost representation using a function such as the logarithm value of the Euclidean distance, which emphasizes the close distance, can be performed. Figure 2

[0115] Figure 14 is a graph indicating the similarity (cost) between the input ID pattern corresponding to the action A ("203" → "70" → "203") and the recognition ID column of the recognition ID table 185. As indicated by the dotted arrow, the cost becomes large until the recognition ID "203" of the input ID pattern appears in the recognition ID column. In the recognition ID column, if the first recognition ID "203" of the input ID pattern appears at the time "13:00:02", the cost becomes small, if the recognition ID "70" of the input ID pattern appears at the time "13:00:04", the cost further becomes small, and if the last recognition ID "203" of the input ID pattern appears at the time "13:00:07", the cost becomes the minimum value. Then, if the recognition ID "16" not included in the input ID pattern appears at the time "13:00:09", the cost becomes large. Therefore, by detecting the interval (time) in which the cost becomes small, it is possible to perform the judgment whether the input ID pattern is included in the recognition ID column. Figure 14

[0116] As such, the similarity calculation section 191 calculates the similarity between the time-series change of the recognition ID and the pattern (input ID pattern) of the recognition ID of the ID pattern model 184. As indicated by the dotted arrow, in the case where the similarity is calculated using the calculation method B, the similarity calculation section 191 acquires the feature amount position corresponding to the recognition ID of the input ID pattern from the classification model 175. In the case where the distance is short in the feature amount space, the cost becomes low, and in the case where the distance is long, the cost becomes high. Further, in order to separate the close feature amounts, the cost representation using a function such as the logarithm value of the Euclidean distance, which emphasizes the close distance, can be performed. Figure 12

[0117] In step S370, the display control section 192 generates information (data) indicating the display image of the similarity (cost) of the time series calculated in step S360 as output information, and causes the output information to be output using the display device 165 (i.e., causes the display image indicating the similarity to be displayed using the display device 165), and ends the processing indicated by the flowchart of Figure 12 ​​​​

[0118] Figure 15 is a graph showing an example of a similarity image 166 displayed on a display screen 165a of the display device 165. The similarity image 166 is a graph image of a curve in which the vertical axis represents a cost and the horizontal axis represents time, and shows a time change in cost for a plurality of actions. The dotted line represents a time-series similarity 166a of the action name "action A", the dashed line represents a time-series similarity 166b of the action name "action B", and the thick solid line represents a time-series similarity 166c of the action name "action C". In addition, for each line type, a graph image 166f representing the action name is displayed on the display screen 165a. Furthermore, an image 166h having a reference line (threshold Ct) representing the cost is displayed on the display screen 165a as a criterion of whether or not a prescribed action is performed.

[0119] As such, according to the present embodiment, the similarity of each action that changes in correspondence with the passage of time is displayed on the display screen of the display device 165, and thus it is possible to easily grasp at which timing (time point) the hydraulic excavator 101A performs which action, and it is possible to achieve an improvement in work efficiency of the hydraulic excavator 101A.

[0120] Further, an example in which the multi-dimensional collection data table 117 is acquired from the hydraulic excavator 101A in step S110 of the process 100 has been described, but it is also possible to sequentially acquire data collected at each time point, calculate the similarity, and output it. Figure 12

[0121] According to the above-described embodiment, the following effects are exerted.

[0122] (1) The action grasping system (management device 105) 10 includes a display device (output device) 165 that outputs information, and a management server (control device) 150 that acquires information of the hydraulic excavator (machine) 101A that changes in time series along with operation, generates output information for grasping an action of the hydraulic excavator 101A on the basis of the information of the hydraulic excavator 101A, and causes the output information to be output by the display device 165. The management server 150 can execute the following classification model creation process (see Figure 4 ), that is, a feature vector V in which information of the hydraulic excavator 101A that changes in time series is set as a feature amount is calculated in time series, a plurality of feature vectors V calculated in time series are classified into a plurality of clusters, and an identification ID is assigned to the plurality of clusters, thereby creating a classification model 175. The management server 150 can execute the following ID pattern model creation process (see Figure 7 ​), that is, the time-series operation sets the feature vector V that features the information of the hydraulic excavator 101A that changes in time series as the characteristic quantity, and gives the recognition ID to the plurality of feature vectors V calculated by the time-series operation based on the classification model 175, and stores the pattern of the recognition ID that corresponds to the prescribed action (excavation action, loading action, etc.) of the hydraulic excavator 101A in association with the recognition information (action name) of the prescribed action, thereby creating the ID pattern model 184. The management server 150 can perform the following information output process (refer to Figure 12 ), that is, the time-series operation sets the feature vector V that features the information of the hydraulic excavator 101A that changes in time series as the characteristic quantity, and gives the recognition ID to the plurality of feature vectors V calculated by the time-series operation based on the classification model 175, and generates the output information for grasping the prescribed action based on the time-series change of the recognition ID and the ID pattern model 184, and causes the output information to be output by the display device 165.

[0123] According to this structure, by creating the classification model 175 and the ID pattern model 184 in advance, the output information (information indicating the degree of similarity) for grasping the prescribed action is displayed on the display device 165 in the case where the work based on the hydraulic excavator 101A is performed in the work site. Therefore, the operator of the hydraulic excavator 101A and the supervisor of the work site and the like who are the viewers of the display device 165 can easily grasp the action of the hydraulic excavator 101A from the output information (information indicating the degree of similarity) displayed on the display screen 165a of the display device 165, and the improvement of the work efficiency of the hydraulic excavator 101A can be achieved.

[0124] In addition, according to this structure, the recognition ID is automatically given to the plurality of micro behaviors (partial actions) that constitute the prescribed action (excavation action, loading action, etc.). Therefore, it is not necessary to set the recognition information (action name) for each partial action. In the case where the partial action name is set in natural language for each partial action, if the number of partial actions is large, the creation of the table in which the recognition information (partial action name) of the partial action is associated with the information of the hydraulic excavator 101A is troublesome, and there is a concern that the manufacturing cost of the action grasping system increases. In the present embodiment, as described above, since the recognition ID is automatically given to the partial action, the reduction of the manufacturing cost of the action grasping system 10 can be achieved.

[0125] (2) In the present embodiment, the recognition ID is given in the sampling period of the characteristic quantity. Therefore, the time resolution of the recognition ID column is the same as the sampling period of the characteristic quantity. Therefore, as compared with the case where the collected information is divided in the prescribed time range longer than the sampling period and the partial action is extracted, a high time resolution can be obtained.

[0126] (3) The management server 150 sets a physical quantity (e.g., cylinder output Fb) calculated from a plurality of different kinds of physical quantities (e.g., cylinder bottom pressure Pb and cylinder rod pressure Pr of the boom cylinder 11a) detected by the sensors of the hydraulic excavator 101A as a feature quantity. In a case where conversion from a plurality of information (physical quantities) to another one information (physical quantity) is performed, reduction in the number of feature quantities can be achieved.

[0127] <2nd Embodiment>

[0128] With reference to Figure 16 and Figure 17 , an operation grasping system of the 2nd embodiment will be described. Further, the same reference numerals are attached to the same or equivalent parts as those of the 1st embodiment, and only the difference points will be mainly described. The operation grasping system of the 2nd embodiment measures the number of times of a prescribed operation based on the similarity degree (cost), and causes the cumulative number of times to be displayed on the display screen 165a of the display device 165. Hereinafter, detailed description will be made.

[0129] Figure 16 is a view showing an output control section 290 of the operation grasping system of the 2nd embodiment of the present application and data used by the output control section 290. As shown in Figure 16 , the output control section 290 has an operation recognition section 293. The operation recognition section 293 judges whether or not the operation of the operation name registered in the ID pattern model 184 is performed based on the cost calculated by the similarity degree calculation section 191. The operation recognition section 293 judges that the operation is performed in a case where the cost C changes from a state where the cost C is larger than a threshold value Ct to a state where the cost C is equal to or smaller than the threshold value Ct according to the passage of time. Further, the threshold value Ct is stored in the ROM 152 in advance.

[0130] The operation recognition section 293 judges that the operation A is performed, for example, in a case where the cost C of the operation A changes from a state where the cost C is larger than the threshold value Ct to a state where the cost C is equal to or smaller than the threshold value Ct, and measures the number of times of the operation A. The operation recognition section 293 measures the number of times of the prescribed operation judged to be performed as the occurrence number of the operation, and stores the cumulative value (cumulative occurrence number) thereof to the HDD 156. The operation recognition section 293 creates a cumulative occurrence number table by storing the occurrence time of the operation and the cumulative occurrence number of the operation in correspondence with each other to the HDD 156. The cumulative occurrence number table is a data table showing the relationship between the cumulative occurrence number of the operation and the operation time of the hydraulic excavator 101A.

[0131] The display control section 192 generates a display image showing the cumulative occurrence number of the operation as output information based on the cumulative occurrence number table, and causes the generated output information to be output by the display device 165.

[0132] Figure 17is an example of a graph indicating an image of the cumulative number of occurrences of the operation displayed on the display screen 165a of the display device 165. The horizontal axis indicates the operation time of the hydraulic excavator 101A, and the vertical axis indicates the cumulative number of occurrences of the operation. In the graph, the solid line indicates the time variation of the cumulative number of occurrences of "operation A", the dashed line indicates the time variation of the cumulative number of occurrences of "operation B", and the one-dot chain line indicates the time variation of the cumulative number of occurrences of "operation C". As such, according to the present embodiment, the display of the cumulative number of occurrences of the operation of the hydraulic excavator 101A can be performed using the display device 165. Therefore, the service person can easily infer the deterioration of the parts related to the operation with a large cumulative number of occurrences, and thus can appropriately set the timing of replacement of the parts, the timing of adjustment of the parts, and the like. In addition, the operator can confirm the presence or absence of unnecessary operations or the insufficiency of the prescribed operation based on the information of the cumulative number of occurrences, and can achieve the improvement of the work efficiency by adjusting the number of operations in the work from then on.

[0133] As such, in the second embodiment, the management server 150, in the information output process, calculates the number of times of the prescribed operation based on the time-series variation of the identification ID and the ID pattern model 184, generates output information (information of a display image indicating the time variation of the cumulative number of occurrences of the operation) based on the calculation result thereof, and causes the output information to be output using the output device (display device 165). Therefore, according to the second embodiment, on the basis of the same effects as those of the above-described first embodiment, by analyzing the information of the cumulative number of occurrences of the operation displayed on the display device 165, the improvement of the efficiency of the maintenance work and the efficiency of the work based on the hydraulic excavator 101A can be achieved.

[0134] Further, the management server 150 can cause a display image related to the cumulative fatigue damage degree or the life consumption amount to be displayed on the display screen 165a of the display device 165 instead of the display image of the cumulative number of occurrences.

[0135] <Modification Example 1 of Second Embodiment>

[0136] In the above-described second embodiment, an example in which the operation recognition unit 293 calculates the cumulative number of occurrences of the operation and the display control unit 192 generates a display image indicating the calculation result thereof is described, but the present application is not limited thereto. It can also be that the operation recognition unit 293 calculates the frequency of occurrence of the prescribed operation on the time axis based on the time-series variation of the identification ID and the ID pattern model 184, the display control unit 192 generates output information based on the calculation result thereof, and causes the generated output information to be output using the display device 165.

[0137] The action recognition unit 293 counts the number of times the prescribed action is judged to have been performed as the number of occurrences of the action, and calculates the number of occurrences in each prescribed time band as the frequency of occurrence. The action recognition unit 293 divides the time axis into prescribed time bands, sets the number of occurrences of the action within the prescribed time band as the frequency of occurrence, and stores it to the HDD 156, thereby creating a frequency of occurrence table. The frequency of occurrence table is a data table that shows the frequency of occurrence of the action in relation to the operating time of the hydraulic excavator 101A.

[0138] The display control unit 192 generates a display image showing the frequency of occurrence of the action as output information based on the frequency of occurrence table, and causes the generated output information to be output using the display device 165.

[0139] Figure 18 is a drawing showing an example of an image showing the frequency of occurrence of the action displayed on the display screen 165a of the display device 165. The horizontal axis shows the operating time of the hydraulic excavator 101A, and the vertical axis shows the frequency of occurrence of the action. In the drawing, the solid line shows the time variation of the frequency of occurrence of "action A", the dashed line shows the time variation of the frequency of occurrence of "action B", and the one-dot chain line shows the time variation of the frequency of occurrence of "action C". As such, according to the present modification example, the display of the frequency of occurrence of the prescribed action of the hydraulic excavator 101A on the time axis can be performed using the display device 165. Therefore, the supervisor of the work performed by the hydraulic excavator 101A can confirm what action the hydraulic excavator 101A performed at which time, and adjust the work plan from then on, thereby enabling the improvement of work efficiency. In addition, the operator can confirm the presence or absence of unnecessary actions or the insufficiency of the prescribed action from the information on the frequency of occurrence, and can improve the work efficiency by adjusting the number of actions in the work from then on.

[0140] <Modification Example 2 of the 2nd Embodiment>

[0141] Alternatively, the action recognition unit 293 can calculate the frequency of occurrence of the prescribed action within the prescribed region based on the time-series variation of the recognition ID and the ID pattern model 184, the display control unit 192 can generate output information based on the calculation result, and the display device 165 can output the generated output information.

[0142] The hydraulic excavator 101A has a plurality of antennas for GNSS (Global Navigation Satellite System) (hereinafter referred to as GNSS antennas), and a positioning calculation device that calculates the position of the hydraulic excavator 101A in a geographic coordinate system (global coordinate system) based on satellite signals (GNSS waves) from a plurality of positioning satellites received by the GNSS antennas.

[0143] The action recognition unit 293 sets the position of the hydraulic excavator 101A calculated by the position calculation device as the place where the prescribed action has occurred and stores it to the HDD 156 if it is judged that the prescribed action has occurred. The action recognition unit 293 divides the map of the work site in the geographic coordinate system into a grid shape and generates a plurality of divided areas. The action recognition unit 293 creates a frequency-of-occurrence map by storing the divided areas and the frequency of occurrence of the action in the divided areas in correspondence with each other to the HDD 156. The frequency-of-occurrence map is a map that indicates the relationship between the place (divided area) where the action has occurred and the frequency of occurrence of the action.

[0144] As such, according to the present modification example, it is possible to perform display of the frequency of occurrence (occurrence distribution) of the prescribed action of the hydraulic excavator 101A in the prescribed area (in the work site) using the display device 165. Therefore, the supervisor of the work performed by the hydraulic excavator 101A can confirm in which place and what kind of action the hydraulic excavator 101A has performed, and can achieve improvement of work efficiency by adjusting the work plan from then on.

[0145] <Modification Example 3 of the 2nd Embodiment>

[0146] It can also be that the action recognition unit 293 calculates the degree of deterioration of a component used for performing each action (for example, a sealing component of a hydraulic cylinder used for performing a digging action) based on the cumulative number of occurrences of the action, the display control unit 192 generates a display image indicating the result of the calculation, and causes the display screen 165a of the display device 165 to display an image indicating the degree of deterioration of the component. Furthermore, the action recognition unit 293 can calculate the degree of deterioration of a prescribed component based on the cumulative number of occurrences of one kind of action (for example, action A) or based on the cumulative number of occurrences of each of a plurality of kinds of actions (for example, action A and action B).

[0147] <3rd Embodiment>

[0148] Reference Figure 19 and Figure 20 describe the action grasping system of the 3rd embodiment. Furthermore, in the drawings, the same reference numerals are attached to the same or equivalent parts as those of the 1st and 2nd embodiments, and only the points of difference are mainly described. The action grasping system of the 3rd embodiment displays the additional information attached to the action name and the animation 367 of the action together with the similarity image 166 to the display screen 165a of the display device 165. Hereinafter, detailed description is made.

[0149] Figure 19 is a diagram that indicates the output control unit 390 of the action grasping system of the 3rd embodiment of the present application and the data used by the output control unit 390. Figure 20is a view showing an example of an image displayed on a display screen 165 of the display device 165 of the action grasping system of the third embodiment of the present application. As shown in the figure, the output control section 390 has an additional information selection section 394 and an action display image generation section 395. In addition, in the HDD 156 of the management server 150, an additional information data table 396 and an action display database 397 are stored. Figure 19

[0150] As with the second embodiment, the action recognition section 293 judges whether or not the prescribed action has been performed based on the similarity (cost). If it is judged that the prescribed action has been performed, the action recognition section 293 stores the time when the prescribed action has been performed in the HDD 156 in correspondence with the prescribed action.

[0151] In addition, if it is judged by the action recognition section 293 that the prescribed action has been performed, the additional information selection section 394 selects additional information corresponding to the prescribed action from the additional information data table 396. The additional information data table 396 is a data table in which "action name" and "additional information" are stored in correspondence.

[0152] The action display image generation section 395 generates a moving image for displaying a three-dimensional animation (dynamic image) in the display device 165. In the action display database 397, data of component shape data including information of shapes and colors of each component of the hydraulic excavator 101A (the traveling body 2, the swing body 3, the boom 11, the stick 12, and the bucket 13), and data of connection portions (turning pivot points) of the component shape data, and the like are included. Further, the component shape data and the data of the connection portions, and the like can be recorded in the action display database 397 as data for each type of the hydraulic excavator 101A. In this case, by inputting the type of the hydraulic excavator 101A to the management server 150 using the input device 161, the action display image generation section 395 selects the component shape data and the data of the connection portions corresponding to the input type to generate the moving image.

[0153] In the present embodiment, the hydraulic excavator 101A is provided with a posture calculation device that calculates the boom angle α, the stick angle β, the bucket angle γ, the tilt angle θ of the traveling body 2 in the front-rear direction with respect to a reference surface (for example, a horizontal surface), the tilt angle φ of the traveling body 2 in the left-right direction with respect to the reference surface, the relative angle (rotation angle) ψ of the traveling body 2 and the swing body 3, and the like based on signals of the posture sensors such as the boom angle sensor 124, the stick angle sensor, the bucket angle sensor, the vehicle body front-rear tilt angle sensor, the vehicle body left-right tilt angle sensor, and the rotation angle sensor. The posture information of the traveling body 2 and the swing body 3.

[0154] ​In the present embodiment, the posture information (α, β, γ, θ, ψ) calculated by the posture calculation device is included in the multidimensional collection data table 117. The motion display image generation section 395 extracts the posture information (α, β, γ, θ, ψ) included in the multidimensional collection data table 117c as motion data, and generates a motion image of the hydraulic excavator 101A on the basis of the extracted motion data and the component shape data and the data of the connection portions of the motion display database 397. More specifically, the motion display image generation section 395 generates a motion image of the hydraulic excavator 101A using three-dimensional computer graphics (3DCG) for each sampling period on the basis of the component shape data and the data of the connection portions, and the posture information for each sampling period. The 3DCG is a technique of expressing the body shape in geometry and converting it to a two-dimensional image by projecting it to a projection plane on the basis of the shape data of the hydraulic excavator 101A. Thus, the appearance of the hydraulic excavator 101A can be visualized from various viewpoints. In addition, by causing each constituent element (the boom and the arm, etc.) of the hydraulic excavator 101A to rotate around the connection portions as the center of rotation on the basis of the posture information (motion data), various postures can also be expressed.

[0155] The display control section 392 causes the animation (moving image) 367 of the hydraulic excavator 101A to be displayed on the display screen 165a of the display device 165 by causing a plurality of motion images generated by the motion display image generation section 395 to change in time series, as shown in Figure 20 In addition, the display control section 392 causes the time mark 366d indicating the time at which the motion image corresponding to the animation 367 is displayed to be displayed on the display screen 165a of the display device 165 on the time axis (time axis) of the similarity image 166. Thus, the time mark 366d moves to the right along the time axis as the image constituting the animation 367 changes.

[0156] The display control section 392 causes an image (action name image) 366f indicating the action name of the prescribed action and an image (time image) 366h indicating the time at which the prescribed action is performed to be displayed on the display screen 165a of the display device 165 when the animation 367 of the time at which the prescribed action is performed is displayed. Furthermore, the display control section 392 causes an image (additional information image) 366g indicating the additional information selected by the additional information selection section 394 to be displayed on the display screen 165a of the display device 165.

[0157] In Figure 20In the illustrated example, a case is shown in which a lift-up operation is performed as a prescribed operation at time "13:24:26". The lift-up operation is an operation in which the working machine 4 is operated in such a manner that the bucket 13 presses the ground, and as a result, the hydraulic excavator 101A is lifted by the reaction force from the ground, and becomes a posture in which the front portion of the running body 2 is lifted from the ground with the rear portion of the running body 2 as a fulcrum. In the lift-up operation, a large load is applied to the bucket cylinder 13a. Therefore, in the illustrated example, a message image indicating the caution "damage to the bucket cylinder is likely" is displayed as the additional information image 366g on the display screen 165a. In addition, on the display screen 165a, the operation name "lift-up operation prohibited" is displayed as the operation name image 366f.

[0158] As described above, in the third embodiment, the management server 150 generates a moving image of the hydraulic excavator 101A on the basis of the information of the hydraulic excavator 101A (posture information included in the multidimensional collection data table 117c) used in the operation of the feature vector V and the shape data of the hydraulic excavator 101A (shape data of the running body 2, the swing body 3, the boom 11, the stick 12, and the bucket 13) in the information output process, and causes the animation 367 of the hydraulic excavator 101A to be displayed on the display screen 165a of the display device 165 on the basis of the moving image.

[0159] As a result, it is possible to visually and easily convey to the viewer of the display device 165 what operation is performed at what time. In particular, when it is intended to grasp the operation of a working machine having a plurality of joints like the hydraulic excavator 101A, there is a case in which it takes time to grasp what operation is performed even if only a graph like the similarity image 166 is observed. According to the present embodiment, the animation 367 of the hydraulic excavator 101A is displayed on the display screen 165a of the display device 165 together with the similarity image 166. Therefore, the viewer of the display device 165 can grasp what operation is performed by the hydraulic excavator 101A in a short time, and thus it is possible to improve the efficiency of adjustment of the work plan and the like.

[0160] In addition, the management device 105 is provided with a storage device (HDD 156) in which additional information corresponding to the operation name (identification information) is stored, and the management server 150 generates output information (information related to a message for calling attention to "damage to the bucket cylinder is likely") on the basis of the operation name (identification information) and the additional information in the information output process, and causes the output information to be output by the display device 165. As a result, it is possible to notify the viewer of the additional information related to the operation, such as the operation performed at a prescribed time being an operation likely to cause damage to a prescribed component, and to call attention to reduce the number of prescribed operations.

[0161] <Modification 1 of the Third Embodiment>

[0162] In the third embodiment, an example in which the output control section 390 generates an animation based on the posture information included in the multi-dimensional collection data table 117 is described, but the present application is not limited thereto. The output control section 390B can also generate an animation based on the time-series change of the recognition ID.

[0163] Figure 21 is a diagram showing the output control section 390B of the action grasping system of the first modification of the third embodiment of the present application and data used by the output control section 390B. The action display image generation section 395B selects a representative feature vector Vi corresponding to the recognition ID at the time series of the recognition ID table 185 generated by the recognition ID column generation section 181 from the classification model 175. The action display image generation section 395B extracts the posture information (a, β, γ, θ, ψ) included in the feature amounts constituting the selected representative feature vector Vi, generates an image of the hydraulic excavator 101A based on the extracted feature amounts (posture information) and the component shape data and the data of the connection sections of the action display database 397, and causes the generated image to be displayed in the time series on the display device 165.

[0164] However, since one representative feature vector Vi is selected from the recognition ID, the movement is more dramatic than in the case where the animation 367 is generated based on the posture information of the multi-dimensional collection data table 117 (the third embodiment). In the case where the animation 367 is generated using the multi-dimensional collection data table 117, since the time resolution is high, smooth movement can be expressed by the animation 367. On the other hand, in the case where the images corresponding to the posture information constituting the representative feature vector Vi corresponding to the recognition ID at the time series are sequentially displayed, the movement becomes unsmooth. Therefore, in the present modification, the key frame method used in the field of computer animation is used.

[0165] The key frame method is a method of interpolating the posture between the postures that become keys to generate (also referred to as key frame interpolation). By interpolating the posture information (i.e., the motion data of a, β, γ, θ, ψ, etc.) between the postures that become keys using the key frame method, a smooth animation can be generated. The action display image generation section 395B sets the posture information (motion data) constituting the representative feature vector Vi corresponding to the recognition ID as control points, interpolates the posture information (motion data) between the control points generated in correspondence with the time-series change of the recognition ID to generate, thereby generating a motion image. The display control section 392 causes the animation 367 to be displayed on the display screen 165a of the display device 165 based on the motion image generated by the action display image generation section 395B.

[0166] Reference Signs List Figure 22, which illustrates an example of interpolation of motion data using a key frame method. Figure 22 is a graph illustrating motion data interpolated when the recognition ID changes in the order of "5" → "7" → "10". Furthermore, for simplicity of explanation, the tilt angle θ, and the swing angle ψ are assumed to have no time change at a predetermined angle (0°) and are explained.

[0167] Here, if images representing the posture 368a corresponding to the recognition ID "5", the posture 368c corresponding to the recognition ID "7", and the posture 368e corresponding to the recognition ID "10" are sequentially displayed on the display device 165, the movement becomes not smooth. In the present modification, as described above, the key frame interpolation method is used in which the posture information of the representative feature vector Vi corresponding to the recognition IDs "5", "7", and "10" is set as a key frame, and a motion image between the key frames is generated. The action display image generation section 395B sets each feature amount (the boom angle α, the arm angle β, and the bucket angle γ) associated with the recognition ID as a control point (becomes a key motion data) and interpolates the motion data (the boom angle α, the arm angle β, and the bucket angle γ) at the time between the key frames using an interpolation function such as a spline function. That is, the action display image generation section 395B generates motion data (α, β, γ) representing the posture 368b between the posture 368a in the recognition ID "5" and the posture 368c in the recognition ID "7", and motion data (α, β, γ) representing the posture 368d between the posture 368c in the recognition ID "7" and the posture 368e in the recognition ID "10" based on the motion data (α, β, γ) of the three points of the recognition IDs "5", "7", and "10".

[0168] The action display image generation section 395B generates a motion image of the posture 368b based on the motion data (motion data obtained by the interpolation operation) at the time between the recognition ID "5" and the recognition ID "7". Similarly, the action display image generation section 395B generates a motion image of the posture 368d based on the motion data (motion data obtained by the interpolation operation) at the time between the recognition ID "7" and the recognition ID "10". Furthermore, in Figure 22In the present modification, a case in which the posture between the key frames is one is shown, but actually, motion data for representing a plurality of postures is generated by interpolation. An animation is generally composed of 30 images per 1 second. Therefore, in a case in which the time at which the identification ID changes in the order of "5" → "7" → "10" is 10 seconds, 300 images (10 seconds x 3 images / second) are required. That is, the dynamic image display image generation section 395B generates, in a case in which an animation of 10 seconds or so is generated, 300 or so of the boom angle a, the arm angle β, and the bucket angle γ (motion data) that change in time series, respectively. Next, the motion display image generation section 395B generates a motion image at each time of interpolation by the 3DCG technique (that is, performs shape projection conversion). Thereby, 300 or so of the motion images are generated. The display control section 392 sequentially switches the 300 or so of the images at a speed of 30 images / second or so and displays them on the display screen 165a of the display device 165. Thereby, smooth motion expression based on an animation can be realized. Further, a method in which a spline function is used as an interpolation function to interpolate and generate motion data is explained, but it is not limited thereto, and various interpolation methods can be used to generate motion data.

[0169] As such, in the present modification, the management server 150, in the information output process, sets the feature amount (posture information) that constitutes the representative feature vector Vi corresponding to the identification ID as a key frame, interpolates between the key frames generated in correspondence with the time series change of the identification ID to generate motion data, and causes the display device 165 to display an animation 367 of the hydraulic excavator 101A based on the motion data. According to such a present modification, as with the above-described third embodiment, a smooth behavior can be expressed by the animation 367, and therefore, a viewer of the display device 165 can easily grasp the motion of the hydraulic excavator 101A.

[0170] In addition, in a case in which the animation 367 is generated by the key frame method, the key frames are generally created by hand, and the creation of motion data takes effort. In contrast, in the present modification, motion data is automatically generated based on the representative feature vector Vi generated by clustering. Therefore, the effort of hand creation can be omitted, and an animation display of the motion of the hydraulic excavator 101A that is visually easy to understand can be realized.

[0171] Further, the management server 150 can not generate motion data interpolating between key frames using the key frame method, but can cause the graphic images generated based on the representative feature vectors Vi corresponding to the identification IDs (for example, still images of the postures 368a, 368c, 368e) to be displayed in order on the display screen 165a of the display device 165. Although this results in a less smooth motion representation, it is easier to grasp the motion compared to a case where the graphic images are not displayed on the display device 165. Further, since the management server 150 can employ a less expensive CPU 151 compared to a case where the animation 367 is generated using the key frame method, the management server 150 can be made less expensive.

[0172] <Variant 2 of the 3rd Embodiment>

[0173] In the 3rd Embodiment, an example in which the output control section 390 causes the similarity image 166, the motion name image 366f, the additional information image 366g, the time image 366h, and the animation 367 to be displayed on the display screen 165a of the display device 165 was described, but the present application is not limited to this. As shown in FIG. 36, the output control section 390 can cause an image (cumulative occurrence number image) 366j indicating a temporal change in the cumulative occurrence number of a prescribed motion, an image (failure risk image) 366i indicating a failure risk calculated based on the cumulative occurrence number, and a still image 366k of a characteristic posture of the prescribed motion to be displayed on the display screen 165a of the display device 165. Figure 23

[0174] Figure 19 As with the 2nd Embodiment, the motion recognition section 293, as shown in FIG. 33, judges whether or not a prescribed motion has been performed based on the similarity (cost), and creates a cumulative occurrence number table based on the result of the judgment. There are cases where the more the cumulative occurrence number of a prescribed motion, the higher the possibility of failure of the prescribed component. In this variant, the output control section 390 sets an image indicating the degree of possibility of failure (a level of failure risk) corresponding to the cumulative occurrence number of a prescribed motion as the failure risk image 366i and displays it on the display screen 165a of the display device 165 (see FIG. 36). Figure 23

[0175] The motion recognition section 293 sets the level of failure risk to 1 when the cumulative occurrence number is less than a 1st threshold value, sets the level of failure risk to 2 when the cumulative occurrence number is equal to or greater than the 1st threshold value and less than a 2nd threshold value, and sets the level of failure risk to 3 when the cumulative occurrence number is equal to or greater than a 3rd threshold value. The display control section 392 sets an image indicating the level of failure risk set by the motion recognition section 293 as the failure risk image 366i and displays it on the display screen 165a of the display device 165 (see FIG. 36). Figure 23 ​​). In addition, the display control section 392 causes a still image 366k of a characteristic posture of the prescribed motion to be displayed on the display screen 165a of the display device 165. The still image 366k is stored as additional information in correspondence with the prescribed motion. Further, the still image 366k can be created based on posture information at the time of the prescribed motion included in the multi-dimensional collection data table 117, or can be created based on a recognition ID constituting an entry ID pattern corresponding to the prescribed motion.

[0176] In the second embodiment, since only an image indicating the time variation of the cumulative occurrence number is displayed on the display device 165, the viewer needs to judge the degree of possibility of failure of the prescribed component. In contrast, in the second modification of the third embodiment, the management server 150 calculates the degree of possibility of failure of the prescribed component (the level of failure risk) and causes an image indicating the degree of possibility of failure (failure risk image) 366i to be displayed on the display device 165 based on the calculation result. Therefore, the viewer can easily judge the degree of possibility of failure. In addition, the occurrence frequency of a motion for which the degree of possibility of failure is high can be grasped from the cumulative occurrence number image 366j indicating the time variation of the cumulative occurrence number in relation to the time variation in the future.

[0177] Further, the method of setting the level of failure risk is not limited to the case where it is set based on the cumulative occurrence number of one motion, but can be set based on the cumulative occurrence numbers of a plurality of motions. In this case, a weight corresponding to the motion can be given. In addition, the level of failure risk can be set based on the cumulative fatigue damage degree or the life consumption amount instead of the cumulative occurrence number. Further, the level of failure risk is, for example, at least two or more levels such as 1 to 3 in order from low to high.

[0178] Figure 23 is a diagram showing an example of an image displayed on the display screen 165a of the display device 165 of the motion grasping system of the second modification of the third embodiment of the present application. In Figure 23In the example shown, the cumulative occurrence number image 366j is displayed so as to increase as time elapses. On the display screen 165a shown in the upper drawing, the cumulative occurrence number image 366j indicating the cumulative value of the occurrence number of the operation until the prescribed time tl is displayed. The cumulative occurrence number image 366j can be an image in which the cumulative occurrence number and the level of the failure risk corresponding to the number are associated. In the example shown in the drawing, the cumulative occurrence number image 366j includes a reference line 366ma indicating the first threshold value for setting the level of the failure risk and a reference line 366mb indicating the second threshold value. In addition, on the display screen 165a, the failure risk image 366i indicating that the level of the failure risk at the time tl is 1 is displayed. If the cumulative occurrence number increases as time elapses, the cumulative occurrence number image 366j indicating the cumulative value of the occurrence number of the operation until the prescribed time t2 is displayed on the display screen 165a as shown in the lower drawing. In addition, on the display screen 165a, the failure risk image 366i indicating that the level of the failure risk at the time t2 is 3 is displayed. Further, as illustrated, in the case where the level of the failure risk is 3, a message image 366n prompting a contact with the agent store or the like, such as "Please contact the nearby agent store", can be displayed on the display screen 165a.

[0179] Further, in Figure 23 the example shown, the level of the failure risk is displayed, but a message indicating a change in the failure risk (for example, the level of the failure risk of the component A is raised from 1 to 2) can be displayed on the display screen 165a of the display device 165. The form of the notification is not limited to display, and for example, can be a mail to an external terminal. Thus, the viewer can be notified that the failure risk of the prescribed component is high, and attention can be drawn to reduce the number of the prescribed operation. In addition, the background color of the failure risk image 366i can be made different according to the level of the failure risk. For example, the background color of the failure risk image 366i can be blue in the case where the level of the failure risk is 1, yellow in the case where the level of the failure risk is 2, and red in the case where the level of the failure risk is 3.

[0180] Further, an image in which a part of the work machine 101 and the cumulative degree of damage (or the failure risk) of the part are associated with each other can be displayed on the display screen 165a of the display device 165. For example, a frame image in which the corresponding part is surrounded by a circle or the like can be displayed in the still image 366k to specify the corresponding part. At this time, only the part with a high failure risk (level 3) can be surrounded by a red circle to be explicitly indicated, or a plurality of regions according to the levels can be displayed in combination with different colors. The combination can be arbitrarily set. Further, in a case where the number of display targets is large and the display area of the still image 366k is insufficient, the moving image 366f, the additional information image 366g, the failure risk image 366i, and the message image 366n can not be displayed, and the still image 366k can be displayed in a magnified manner in an area including an area in which these images were displayed. Thus, the viewer can intuitively and visually grasp which part has a high degree of damage from the entire image (still image 366k) of the work machine 101.

[0181] <4th Embodiment>

[0182] Reference Figure 24 An operation grasping system according to the 4th embodiment will be described. In the drawings, the same reference numerals are attached to the same or equivalent parts as those of the 1st and 2nd embodiments, and only the difference will be mainly described. In the operation grasping system according to the 4th embodiment, measurement of the duration of a prescribed operation is performed. The duration corresponds to, for example, the operation time from the start to the end of a prescribed operation such as a "digging operation", a "loading operation", a "digging and loading operation", and the like.

[0183] Figure 24 is a view showing an output control section 490 of the operation grasping system according to the 4th embodiment and data used by the output control section 490. As shown in Figure 24 , the output control section 490 has a duration measuring section 498. The duration measuring section 498 searches the start point (the start time of the prescribed operation) and the end point (the end time of the prescribed operation) of the entry ID pattern from the identification ID column of the identification ID table 185 generated by the identification ID column generating section 181, and calculates the time from the start time to the end time of the prescribed operation as the duration of the prescribed operation. The operation recognizing section 293 determines whether the prescribed operation is performed or not on the basis of the degree of similarity in the same manner as described above.

[0184] The duration measuring section 498 refers to the recognition ID table 185 to determine whether the recognition ID at the time when the prescribed action was judged to have been performed is continuously set also after that time. If the recognition ID is determined to have been continuously set, the duration measuring section 498 sets the last time of the same recognition ID for which the recognition ID was continuously set as the end time of the prescribed action. If the recognition ID is determined not to have been continuously set, the duration measuring section 498 sets the time when the prescribed action was judged to have been performed as the end time of the prescribed action.

[0185] The duration measuring section 498 refers to the recognition ID table 185 to set, among the times before the time when the prescribed action was judged to have been performed, the time when the first recognition ID of the entry ID pattern corresponding to the prescribed action appeared as the start time of the prescribed action.

[0186] Referring to Figure 14 , a specific example of the method of measuring the duration will be described. In the example shown in Figure 14 , the action recognition section 293 judges that the action A was performed at time "13:00:07". The duration measuring section 498 determines whether the recognition ID 203 at time "13:00:07" was set also at time 13:00:08. Since the recognition ID 203 was set also at time "13:00:08", the duration measuring section 498 sets the last time of the same recognition ID for which the recognition ID was continuously set, "13:00:08", as the end time of the action A. The duration measuring section 498 sets, among the times before the time "13:00:07" when the action A was judged to have been performed, the time "13:00:02" when the first recognition ID of the entry ID pattern "203"→"70"→"203" corresponding to the action A appeared as the start time of the action A. The duration measuring section 498 calculates the duration "6 sec" by subtracting the start time "13:00:02" from the end time "13:00:08" of the action A.

[0187] Figure 24 The display control section 492 shown in FIG. 12 causes a display image showing the average, the cumulative value, etc. of the duration of each action within a prescribed period to be displayed on the display screen 165a of the display device 165. Further, the duration of the action, the start time and the end time of the action can be transmitted to a device (for example, a mobile terminal held by a maintenance worker) on which an application program that performs analysis of the actions of the hydraulic excavator 101A is installed.

[0188] The management server 150 of this embodiment, in the information output process, calculates the duration of the prescribed action on the basis of the time-series change of the identification ID and the ID pattern model 184, generates output information indicating the duration on the basis of the calculation result, and causes the output information to be output using the display device (output device) 165. In this embodiment, moreover, the duration, the start time, and the end time of the action of the hydraulic excavator 101A are measured, and these pieces of information are caused to be displayed on the display device 165. Therefore, a viewer who views the display device 165 can use the duration, the start time, and the end time of the action displayed on the display device 165 for the work analysis of the hydraulic excavator 101A.

[0189] Modifications such as the following are also within the scope of the present application, and the structures shown in the modifications can be combined with the structures explained in the above-described embodiments, the structures explained in the different embodiments can be combined with each other, and the structures explained in the different modifications can be combined with each other.

[0190] <Modification 1>

[0191] In the above-described embodiments, an example in which the management device 105 functions as the action grasping system 10 of the hydraulic excavator 101A has been explained, but the present application is not limited to this. The action grasping system 10 can be constituted only by the hydraulic excavator 101A, and the action grasping systems 10B, 10C can also be constituted by the hydraulic excavator 101A and the management device 105 (refer to Figure 25 , Figure 26 ).

[0192] <Modification 1-1>

[0193] Figure 25 is the same as Figure 2 , is a functional block diagram of the action grasping system 10B of Modification 1-1. This modification differs from the first embodiment in that the body controller 510 of the hydraulic excavator 101A has the functions of creating the classification model 175 and creating the identification ID table 185, and the management server 550 does not have these functions.

[0194] As shown in Figure 25 , the body controller 510 of the hydraulic excavator 101A functions as the feature vector generation section 172, the classification section 173, the identification ID assignment section 174, the identification ID column generation section 181, and the data transmission section 129 by executing the program stored in the ROM 112. The identification ID column generation section 181 generates the identification ID table 185 on the basis of the feature amount data table 176 and the classification model 175, and stores it in the HDD 116. The data transmission section 129 transmits the identification ID table 185 stored in the HDD 116 to the management device 105 via the communication line 109 through the communication device 128.

[0195] The management server 550 of the management device 105 functions as the data acquisition section 171, the ID pattern extraction section 182, the similarity calculation section 191, and the display control section 192 by executing a program stored in the ROM 152.

[0196] The data acquisition section 171 acquires the identification ID table 185 and the operation data table 118 from the hydraulic excavator 101A via the communication line 109 through the communication device 162 and stores them in the HDD 156. Further, the ID pattern extraction section 182, the similarity calculation section 191, and the display control section 192 have the same functions as those of the first embodiment, and thus detailed description thereof is omitted.

[0197] As such, the operation grasping system 10B of the present modification example has the body controller (first control device) 510 possessed by the hydraulic excavator 101A and the management server (second control device) 550 possessed by the management device (external device) 105 disposed outside the hydraulic excavator 101A. The body controller 510 is capable of performing the following first information output process, that is, time-divisionally calculating a feature vector V that takes information of the hydraulic excavator 101A changing in time series as a feature amount, assigning an identification ID to a plurality of feature vectors V calculated by the time-divisional calculation on the basis of the classification model 175, and transmitting information of an identification ID column indicating a time-series change in the identification ID to the management device 105. In addition, the management server 550 is capable of performing the following second information output process, that is, generating output information for grasping a prescribed operation on the basis of a time-series change in the identification ID and the ID pattern model 184 and causing the output information to be output by the display device (output device) 165.

[0198] In the first embodiment, the multi-dimensional collection data table 117 composed of a plurality of kinds of collection information (physical quantities) acquired at each sampling period is transmitted from the hydraulic excavator 101A to the management device 105 via the communication line 109. In contrast, in the present modification example, the body controller 510 of the hydraulic excavator 101A generates the identification ID table 185 that is a table of data changing in time series as the identification ID and transmits the generated identification ID table 185 to the management device 105 via the communication line 109. The identification ID table 185 has a smaller data amount than the multi-dimensional collection data table 117. Therefore, according to the present modification example, reduction in the amount of communication can be achieved as compared with the first embodiment.

[0199] Further, in a case where the cost is calculated by the cost calculation method (calculation method B) based on the distance between the feature amount positions of the recognition IDs, it is only necessary to previously send the classification model 175 from the body controller 510 of the hydraulic excavator 101A to the management server 550 of the management device 105 via the communication line 109 and store it to the HDD 156. In addition, it is also possible to send the operation result in the similarity degree operation section 191 to the mobile terminal held by the maintenance operator via the communication line 109 and cause the similarity degree image 166 to be displayed on the display screen of the mobile terminal using an application program of the mobile terminal.

[0200] <Variant Example 1-2>

[0201] Figure 26 is the same as Figure 2 The same figure is a functional block diagram of the operation grasping system 10C of the variant example 1-2. This variant example differs from the first embodiment in that the body controller 610 of the hydraulic excavator 101A has a function of creating the classification model 175, a function of creating the recognition ID table 185, and a function of creating the ID style model 184, and the management server 650 does not have these functions.

[0202] As shown in Figure 26 , the body controller 610 of the hydraulic excavator 101A functions as the feature vector generation section 172, the classification section 173, the recognition ID assignment section 174, the recognition ID column generation section 181, the ID style extraction section 182, the similarity degree operation section 191, and the data transmission section 129 by executing the program stored in the ROM 112. The recognition ID column generation section 181 generates the recognition ID table 185 based on the feature amount data table 176 and the classification model 175 (see FIG. 6) and stores it to the HDD 116. In addition, the ID style extraction section 182 generates the ID style model 184 based on the recognition ID table 185 and the operation data table 118 and stores it to the HDD 116. The data transmission section 129 transmits the operation result in the similarity degree operation section 191 to the management device 105 via the communication line 109 through the communication device 128. Figure 8

[0203] The management server 650 of the management device 105 functions as the data acquisition section 171, the analysis section 693, and the data transmission section 692 by executing the program stored in the ROM 152.

[0204] The data acquisition section 171 acquires the operation result (data) in the similarity degree operation section 191 from the hydraulic excavator 101A via the communication line 109 through the communication device 162 and stores it to the HDD 156.

[0205] ​The analysis section 693 is, for example, the action recognition section 293 explained in the second embodiment and the modification example 1 of the second embodiment, and generates analysis data for analyzing the action of the hydraulic excavator 101A, such as the cumulative occurrence number of the action and the occurrence frequency of the action. The data transmission section 692 transmits the analysis data to the communication terminal 91, 92, 93 held by the user who analyzes the action of the hydraulic excavator 101A, such as the maintenance operator, via the communication line 109 by the communication device 162. The communication terminal 91, 92, 93 displays a display image (for example, a cumulative occurrence number image, an occurrence frequency image) related to the analysis data on the display screen of the communication terminal 91, 92, 93 by executing a prescribed application program.

[0206] As such, the action grasping system 10C of the present modification example has the vehicle body controller (first control device) 610 possessed by the hydraulic excavator 101A, and the management server (second control device) 650 possessed by the management device (external device) 105 arranged outside the hydraulic excavator 101A. The vehicle body controller 610 is capable of performing the following first information output process, that is, time-divisionally operating a feature vector V that takes information of the hydraulic excavator 101A changing in time series as a feature amount, giving a recognition ID to a plurality of feature vectors V operated time-divisionally based on the classification model 175, generating output information for grasping a prescribed action based on the time-series change of the recognition ID and the ID pattern model 184, and transmitting the output information to the management device 105. In addition, the management server 650 is capable of performing the following second information output process, that is, outputting the output information to the communication terminal 91, 92, 93 by the communication device (output device) 162. According to such a modification example, compared with the first embodiment, reduction of the communication amount can be achieved. Furthermore, the management server 650 can also output the output information by the display device (output device) 165 in the second information output process.

[0207] <Modification Example 1-3>

[0208] Although not illustrated, the vehicle body controller 110 of the hydraulic excavator 101A can also have the function of the output control section 390B explained in the modification example 1 of the third embodiment. In this case, the output control section 390B transmits the motion data of the hydraulic excavator 101A to the management device 105 via the communication line 109. The management device 105 causes the animation 367 of the hydraulic excavator 101A to be displayed on the display screen 165a of the display device 165 based on the acquired motion data. Here, it is preferable that the motion data transmitted from the hydraulic excavator 101A be used as the feature amount of the classification model 175. Figure 22corresponding to the postures 368a, 368c, 368e, and the motion data corresponding to the postures 368b, 368d are interpolated and displayed on the display screen 165a of the display device 165 by the key frame method on the management device 105 side. Thus, reduction of the amount of data transmitted from the hydraulic excavator 101A to the management device 105 can be achieved.

[0209] <Modified Example 2>

[0210] In the above-described embodiment, an example in which the information of the work machine 101 is transmitted to the management device 105 via the communication line 109 is described, but the present application is not limited thereto. The information of the work machine 101 can be stored in a removable storage medium such as a memory card, and the information of the storage medium can be read into the management device 105.

[0211] <Modified Example 3>

[0212] In the first embodiment, an example in which the management device 105 generates information of a display image for grasping the prescribed operation of the hydraulic excavator 101A as output information, and causes the display image to be displayed on the display screen 165a of the display device 165 is described, but the present application is not limited thereto. The management server 150 can also generate information of a print image for grasping the prescribed operation of the hydraulic excavator 101A as output information, and cause the output information to be output by a printing device as an output device, that is, cause the print image to be printed on a paper medium. In addition, the management server 150 can also transmit (output) the output information for grasping the prescribed operation of the hydraulic excavator 101A to a mobile terminal such as a smartphone, a tablet PC, a mobile phone, and the like via the communication device 162 as an output device. Thus, the owner of the mobile terminal can cause the information for grasping the prescribed operation of the hydraulic excavator 101A to be displayed on a display screen by operating the mobile terminal.

[0213] <Modified Example 4>

[0214] In the modified example 1 of the third embodiment, an example in which the operation display image generation section 395B generates a motion image corresponding to all of the recognition IDs constituting the recognition ID table 185, and thereby generates the animation 367 is described, but the present application is not limited thereto. It is also possible to generate a motion image only for the operation recognized by the operation recognition section 293. Further, it is also possible that, in a case where it is determined by the operation recognition section 293 that the recognition ID series "5"→"7"→"8"→"10" generated by the recognition ID series generation section 181 is an operation of the entry ID pattern "5"→"7"→"10", the operation display image generation section 395B generates a motion image based on the entry ID pattern "5"→"7"→"10". That is, it is also possible to generate an animation of a motion image in which the recognition ID "8" is omitted.

[0215] <Modification example 5>

[0216] In the above-described embodiment, an example in which the multi-dimensional collection data table 117a, 117b for learning is created based on actual machine data of the hydraulic excavator 101A is described, but the present application is not limited thereto. A simulator that simulates the operation of the hydraulic excavator 101A can be used to perform a simulation operation, and the multi-dimensional collection data table 117a, 117b for learning can be created using the simulation data obtained at that time.

[0217] <Modification example 6>

[0218] In the above-described embodiment, an example in which the work machine 101 is the hydraulic excavator 101A, the dump truck 101B, and the work robot 101C is described, but the present application is not limited thereto. The present application can be applied to a wheel loader, a crawler crane, or the like. In addition, the present application is not limited to the work machine 101, and can be applied to a machine whose operation changes in time series.

[0219] The above describes an embodiment of the present application, but the above-described embodiment merely shows a part of the application examples of the present application, and the gist is not to limit the technical scope of the present application to the specific configuration of the above-described embodiment.

[0220] Explanation of reference signs

[0221] 4: work implement, 10, 10B, 10C: action grasping system, 101: work machine, 101A: hydraulic shovel (machine), 101B: dump truck (machine), 101C: work robot (machine), 105: management device (external device), 109: communication line, 110: vehicle body controller (1st control device, control device), 117: multi-dimensional collection data table, 118: action data table, 121: cylinder bottom pressure sensor (sensor), 122: cylinder rod pressure sensor (sensor), 123: boom operation sensor (sensor), 124: boom angle sensor (attitude sensor, sensor), 125: temperature sensor (sensor), 128: communication device (output device), 150: management server (2nd control device, control device), 162: communication device (output device), 165: display device (output device), 166: similarity image, 170: classification model creating section, 172: feature vector generating section, 173: classification section, 174: recognition ID assigning section, 175: classification model, 176: feature amount data table, 180: ID pattern model creating section, 181: recognition ID column generating section, 182: ID pattern extracting section, 184: ID pattern model, 185: recognition ID table, 190: output control section, 191: similarity calculation section, 192: display control section (output control section), 290: output control section, 293: action recognition section, 366g: additional information image, 366i: failure risk image, 366j: cumulative occurrence number image, 366k: still image, 367: animation (moving image), 390, 390B: output control section, 392: display control section, 394: additional information selecting section, 395, 395B: action display image generating section, 396: additional information data table, 397: action display database, 490: output control section, 492: display control section, 498: duration measuring section, 510: vehicle body controller (1st control device), 550: management server (2nd control device), 610: vehicle body controller (1st control device), 650: management server (2nd control device), 693: analysis section, C: cost (similarity).

Claims

1. An action grasping system, comprising: a display device that outputs information; and a control device that acquires information of a machine that changes over time in association with an operation, generates output information for grasping an action of the machine on the basis of the information of the machine, and causes the output information to be output by the display device, characterized in that the control device is capable of performing the following processing: a classification model creation process of time-divisionally operating a feature vector that sets information of the machine that changes over time as a feature quantity, classifying a plurality of feature vectors time-divisionally operated into a plurality of clusters, and assigning an identification ID to the plurality of clusters, thereby creating a classification model; an ID pattern model creation process of time-divisionally operating a feature vector that sets information of the machine that changes over time as a feature quantity, assigning the identification ID to a plurality of feature vectors time-divisionally operated on the basis of the classification model, and storing a pattern of the identification ID constituted by a plurality of identification ID columns in which the identification IDs are arranged in time series in correspondence with a prescribed action of the machine and identification information of the prescribed action in correspondence with the pattern of the identification ID, thereby creating an ID pattern model; and an information output process of time-divisionally operating a feature vector that sets information of the machine that changes over time as a feature quantity, assigning the identification ID to a plurality of feature vectors time-divisionally operated on the basis of the classification model, generating output information for grasping the prescribed action on the basis of the identification ID columns in which a plurality of the identification IDs are arranged in time series and the ID pattern model created in advance, and causing the output information to be output by the display device.

2. The action grasping system according to claim 1, characterized in that the control device generates, as the output information, a time series change of the degree of similarity between the identification ID columns in which a plurality of the identification IDs are arranged in time series and the ID pattern model for each of the identification information of the prescribed action in the information output process.

3. The action grasping system according to claim 1, characterized in that a storage device that stores additional information in correspondence with the identification information is provided, and the control device generates the output information on the basis of the identification information and the additional information and causes the output information to be output by the display device in the information output process.

4. The action grasping system according to claim 1, characterized in that the control device calculates the number of times the prescribed action is performed on the basis of the time series change of the identification ID and the ID pattern model in the information output process, generates the output information on the basis of the calculation result, and causes the output information to be output by the display device.

5. The action grasping system according to claim 1, characterized in that the control device calculates at least one of the frequency of occurrence on a time axis and the frequency of occurrence in a prescribed region of the prescribed action on the basis of the time series change of the identification ID and the ID pattern model in the information output process, generates the output information on the basis of the calculation result, and causes the output information to be output by the display device. ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ 6. The action grasping system according to claim 1, characterized in that, the control device generates, in the information output process, motion data of the machine based on the posture information of the machine and the shape data of the machine used in the operation of the feature vector, and causes an animation of the machine to be displayed on the display device based on the motion data.

7. The action grasping system according to claim 1, characterized in that, the control device, in the information output process, sets a feature vector corresponding to the recognition ID as a key frame, generates motion data by interpolating between the key frames generated in correspondence with the temporal change of the recognition ID, and causes an animation of the machine to be displayed on the display device based on the motion data.

8. The action grasping system according to claim 1, characterized in that, the control device, in the information output process, causes a still image generated based on a feature vector corresponding to the recognition ID to be displayed on the display device.

9. The action grasping system according to claim 1, characterized in that, the control device, in the information output process, operates a duration of the prescribed action based on the temporal change of the recognition ID and the ID pattern model, generates the output information based on the operation result, and causes the output information to be output by the display device.

10. The action grasping system according to claim 1, characterized in that, the machine is a work machine having a work device, and has a plurality of sensors, the control device sets a physical quantity calculated from a plurality of different kinds of physical quantities detected by the sensors as a feature quantity.

11. The action grasping system according to claim 1, characterized in that, the machine is a work machine having a work device, the control device has: a first control device possessed by the work machine; and a second control device possessed by an external device disposed outside the work machine, the first control device is capable of performing a first information output process of operating, in time division, feature vectors that set information of the machine changing in time series as feature quantities, assigning recognition IDs to a plurality of feature vectors operated in time division based on the classification model, and transmitting information of a recognition ID column indicating the temporal change of the recognition ID to the external device, the second control device is capable of performing a second information output process of generating output information for grasping the prescribed action based on the temporal change of the recognition ID and the ID pattern model, and causing the output information to be output by the display device.

12. The action grasping system according to claim 1, characterized in that, the machine is a work machine having a work device, the control device has: a first control device possessed by the work machine; and a second control device possessed by an external device disposed outside the work machine, The first control device can execute the first information output process of time-divisionally calculating a feature vector that features information of the machine that changes over time, assigning the recognition ID to a plurality of feature vectors calculated by the time-divisional calculation on the basis of the classification model, generating output information for grasping the prescribed action on the basis of a time-series change in the recognition ID and the ID pattern model, and transmitting the output information to the external device, The second control device can execute the second information output process of causing the output information to be output by the display device.

Images