Timing prediction method, timing prediction device, timing prediction system, program, and construction machinery system

Abstract

The present invention acquires movement data for a plurality of points on an observed object that performs a prescribed movement operation, performs a frequency analysis on the movement data to generate feature value data, segments the feature value data to extract a transition array for a basic operation, and, by analyzing the frequency of occurrence of the basic operation during the prescribed movement operation on the basis of the transition array and an operation model obtained by learning a transition array for a basic movement obtained by segmenting the movement of a training object that performs a movement operation, extracts a pattern for the basic operation and thereby predicts the end timing of the prescribed movement operation.

Description

[Technical Field]

[0001] The present invention relates to a timing prediction method, a timing prediction device, a timing prediction system, a program, and a construction machinery system. This application claims priority based on Japanese Patent Application No. 2020-090968, filed in Japan on May 25, 2020, and the contents of that application are incorporated herein by reference. [Background technology]

[0002] In the field of civil engineering, research is being conducted to automate the process of loading soil and sand scooped up by a backhoe into a dump truck. This type of research allows for the analysis of the movements of the backhoe and dump truck, and the prediction of the relationship between their stopping positions when loading soil and sand (see, for example, Non-Patent Documents 1 and 2). [Prior art documents] [Non-patent literature]

[0003] [Non-Patent Document 1] Keiji Nagatani, Yota Konno, Kazunori Ohno, Takahiro Suzuki, Taro Suzuki, Yukinori Shibata, Kimitaka Asano, Tomohiro Komatsu, and Yuji Oguri, "Research and Development on Automated Driving of Six-Wheel Dump Trucks Applicable to Small and Medium-Sized Construction Companies," 18th System Integration Conference (SI2017), 2017, pp. 1013-1016. [Non-Patent Document 2] Kazunori Ohno, Ryunosuke Hamada, Naoki Mizuno, Shunpei Yamaguchi, Tatsuya Hoshi, Taro Suzuki, Yukinori Shibata, Kimitaka Asano, Takahiro Suzuki, and Satoshi Tadokoro, "Measurement of Cooperative Operation of a Six-Wheel Dump Truck and a Backhoe," Robotics & Mechatronics Conference 2018, June 2018, pp. 2A2-B06(1)~2A2-B06(3) [Overview of the Initiative] [Problems that the invention aims to solve]

[0004] To automate the loading process from a backhoe to a dump truck, it is necessary not only to predict the dump truck's stopping position relative to the backhoe, but also to predict when the backhoe will have completed its loading preparations. In other words, the development of technology to predict the completion time of a predetermined motion of a moving object is desirable. [Means for solving the problem]

[0005] According to a first aspect of the present invention, a timing prediction method is provided. The timing prediction method may acquire motion data for multiple locations on an observed object performing a predetermined motion. The timing prediction method may generate feature data by performing frequency analysis on the acquired motion data. The timing prediction method may extract a sequence of primitive motion transitions by segmenting the generated feature data. Based on the extracted sequence of primitive motions and a motion model obtained by learning the sequence of primitive motion transitions obtained by segmenting the motion state of a training object performing a motion, the timing prediction method may predict the timing of the end of the predetermined motion by analyzing the frequency of occurrence of primitive motions in the predetermined motion and extracting a pattern of primitive motions.

[0006] The above timing prediction method divides the predetermined motion performed by the observed object into multiple motions, analyzes the frequency of primitive motions in each motion based on the extracted transition sequence and motion model to extract a pattern of primitive motions, evaluates the extracted pattern step by step to calculate a confidence level, and predicts the timing of the end of the predetermined motion according to the calculated confidence level.

[0007] The above timing prediction method may involve performing frequency analysis on the acquired motion data to select a frequency band containing frequency components resulting from the predetermined motion, and then generating feature data in the selected frequency band.

[0008] The above timing prediction method applies a hidden Markov model to the above operation model, and the feature data is segmented by computational processing using the above hidden Markov model to extract the sequence of primitive operation transitions.

[0009] The timing prediction method described above may include angular velocity data and acceleration data in the motion data.

[0010] A second aspect of the present invention provides a timing prediction device. The timing prediction device may include a motion model storage unit that stores a motion model obtained by learning a sequence of primitive motion transitions obtained by segmenting the motion state of a training object performing a motion. The timing prediction device may include a motion data acquisition unit that acquires motion data for multiple locations on an observed object performing a predetermined motion. The timing prediction device may include a feature generation unit that generates feature data by performing frequency analysis on the motion data acquired by the motion data acquisition unit. The timing prediction device may include a motion analysis unit that extracts a sequence of primitive motion transitions by segmenting the feature data generated by the feature generation unit. The timing prediction device may include a prediction unit that predicts the end timing of a predetermined motion by analyzing the frequency of occurrence of primitive motion in the predetermined motion and extracting a pattern of primitive motion based on the sequence of transitions extracted by the motion analysis unit and the motion model stored in the motion model storage unit.

[0011] A third aspect of the present invention provides a timing prediction system. The timing prediction system may include a motion measurement device that measures motion data for multiple locations on an observed object performing a predetermined motion, and a timing prediction device. The timing prediction device may include a motion model storage unit that stores a motion model obtained by learning a sequence of primitive motion transitions obtained by segmenting the motion state of a training object performing a motion. The timing prediction device may include a motion data acquisition unit that acquires the motion data from the motion measurement device. The timing prediction device may include a feature generation unit that generates feature data by performing frequency analysis on the motion data acquired by the motion data acquisition unit. The timing prediction device may include a motion analysis unit that extracts a sequence of primitive motion transitions by segmenting the feature data generated by the feature generation unit. The timing prediction device may include a prediction unit that predicts the timing of the end of the predetermined motion by analyzing the frequency of appearance of primitive motion in the predetermined motion and extracting a pattern of primitive motion based on the sequence of transitions extracted by the motion analysis unit and the motion model stored in the motion model storage unit.

[0012] According to a fourth aspect of the present invention, a program is provided. The program may be processed by a computer equipped with a motion model storage unit that stores a motion model obtained by learning a sequence of primitive motion transitions obtained by segmenting the motion state of a training object performing a motion. The computer may perform a process to acquire motion data for multiple locations on an observed object performing a predetermined motion. The computer may perform a process to generate feature data by performing frequency analysis on the acquired motion data. The computer may perform a process to extract a sequence of primitive motion transitions by segmenting the generated feature data. Based on the extracted sequence of primitive motions and the motion model stored in the motion model storage unit, the computer may perform a process to predict the timing of the end of the predetermined motion by analyzing the frequency of occurrence of primitive motions in the predetermined motion and extracting a pattern of primitive motions.

[0013] A fifth aspect of the present invention provides a construction machinery system. The construction machinery system may include a motion measurement device for measuring motion data at multiple locations on a construction machine performing a predetermined motion, and a timing prediction device. The timing prediction device may include a motion model storage unit for storing a motion model obtained by learning a sequence of primitive motions obtained by segmenting the motion state of a training construction machine performing a motion. The timing prediction device may include a motion data acquisition unit for acquiring the motion data from the motion measurement device. The timing prediction device may include a feature generation unit for generating feature data by performing frequency analysis on the motion data acquired by the motion data acquisition unit. The timing prediction device may include a motion analysis unit for extracting a sequence of primitive motions by segmenting the feature data generated by the feature generation unit. The timing prediction device may include a prediction unit for predicting the end timing of a predetermined motion by analyzing the frequency of occurrence of primitive motions in the predetermined motion and extracting a pattern of primitive motions based on the sequence of primitive motions extracted by the motion analysis unit and the motion model stored in the motion model storage unit.

[0014] It should be noted that the above summary of the invention does not list all the features necessary for the present invention. Furthermore, subcombinations of these features may also constitute an invention. [Effects of the Invention]

[0015] According to the above-described aspect of the present invention, it is possible to predict the timing of the end of a predetermined motion of an object performing a motion. [Brief explanation of the drawing]

[0016] [Figure 1] This diagram schematically shows an example of the timing prediction system 100. [Figure 2] This figure shows an example of the general functional configuration of the timing prediction system 100. [Figure 3]This flowchart shows an example of the general procedure for the learning process of the behavioral model performed by the timing prediction system 100. [Figure 4A] This is the first graph showing an example of motion data recorded by the recording unit 143 of the motion measurement device 140. The graph shows the x-axis acceleration data against elapsed time for each of the motion measurement devices 140A, 140B, and 140C. [Figure 4B] This is a second graph showing an example of motion data recorded by the recording unit 143 of the motion measurement device 140. The graph shows the angular velocity data around the yaw angle as a function of elapsed time for motion measurement devices 140A, 140B, and 140C, respectively. [Figure 5] This graph shows an example of frequency intensity obtained by the feature generation unit 114 after performing a Fast Fourier Transform. [Figure 6] This graph shows an example of the transition of primitive movements in a motion model stored in the motion model storage unit 116. [Figure 7] This flowchart shows an example of the general procedure for the timing prediction process performed by the timing prediction system 100 during actual operation. [Figure 8] This graph shows an example of the transitions between primitive movements, as demonstrated by a motion model trained on learning momentum data. [Figure 9] This graph shows an example of the relationship between the time before the completion of a preparatory work action and the degree of confidence. [Figure 10] This figure shows an example of a schematic hardware configuration of a computer 1000 that functions as a timing prediction device 110. [Figure 11] A diagram showing an example of the configuration of the control unit 1001 in a modified example. [Figure 12] A figure showing an example of the experimental results of the first evaluation experiment in a modified example. [Figure 13] The first explanatory diagram illustrating the second evaluation experiment in the modified example. [Figure 14] A second explanatory diagram illustrating the second evaluation experiment in a modified example. [Figure 15] A third explanatory diagram illustrating the second evaluation experiment in the modified example. [Figure 16] A figure showing an example of the experimental results of the second evaluation experiment in a modified example. [Figure 17] A flowchart illustrating an example of the processing flow performed in the pattern acquisition process, confidence level acquisition process, and loading timing estimation process in the modified example. [Figure 18] A figure showing an example of the experimental results of the third evaluation experiment in a modified example. [Modes for carrying out the invention]

[0017] The present invention will be described below through embodiments of the invention, but these embodiments are not intended to limit the invention as defined in the claims. Furthermore, not all combinations of features described in the embodiments are necessarily essential to the solution of the invention.

[0018] In this embodiment, the backhoe, operated by a passenger, is a construction machine that performs a series of tasks, from leveling the ground beneath the machine and tidying up soil, to scooping up soil and loading the scooped soil onto a dump truck that approaches the machine. This embodiment describes a timing prediction method, timing prediction device, timing prediction system, program, and construction machinery system for predicting the completion timing of the backhoe's motion from scooping up soil after leveling work to the completion of preparation for loading the scooped soil onto a dump truck (hereinafter referred to as "preparation work operation").

[0019] Figure 1 is a schematic diagram illustrating an example of the timing prediction system 100. The timing prediction system 100 is a system that predicts the completion timing of preparatory work operations in a backhoe 300 that performs various motion operations.

[0020] As shown in Figure 1, the backhoe 300 is equipped with a backhoe attachment 320 on the main body 310. The backhoe 300 scoops up soil and other materials by pulling the bucket 324 attached to the tip of the backhoe attachment 320 towards the main body 310, and loads the scooped materials onto a dump truck (not shown).

[0021] The main body 310 consists of a lower traveling body 311 with an upper rotating body 312 attached. The lower traveling body 311 is the lower mechanism of the backhoe 300, equipped with a traveling function for moving the machine and a function for supporting the upper rotating body 312. The upper rotating body 312 is part of the backhoe 300, consisting of various mechanisms including a rotating frame 312A and a cab 312B mounted on the rotating frame 312A. The upper rotating body 312 is rotated by a rotating mechanism. The rotating frame 312A is the frame on which the various devices constituting the upper rotating body 312 are mounted. The cab 312B is the cockpit installed on the upper rotating body 312.

[0022] The backhoe attachment 320 comprises a boom 321, an arm 322, a bucket link 323, and a bucket 324, and is mounted on the main body 310. The boom 321 is a support column attached by pins to the front of the slewing frame 312A, supporting the arm 322, bucket 324, etc. The arm 322 is an arm that connects the bucket 324 to the tip of the boom 321. The bucket link 323 is a link mechanism for operating the bucket 324 with the bucket cylinder 325. The bucket cylinder 325 is a hydraulic cylinder for operating the bucket 324. The bucket 324 is a container equipped with cutting blades, etc., for directly excavating the work target or scooping up materials.

[0023] The timing prediction system 100 includes a timing prediction device 110, motion measurement devices 140A, 140B, 140C, and a foot switch 150.

[0024] Motion measurement devices 140A, 140B, and 140C (hereinafter referred to as motion measurement device 140 unless otherwise specified) are devices for measuring momentum and time. Motion measurement device 140 detects, for example, the rotational motion and translational motion of each of the three orthogonal axes, and measures angular velocity from the rotational motion and acceleration from the translational motion. For example, motion measurement device 140 measures inertial momentum with an angular velocity measurement range of ±250 [deg / s] and an acceleration measurement range of ±16 [G], with a measurement bandwidth of 200 [Hz]. The time measured by motion measurement device 140 may be the elapsed time from the start of measurement or the time of measurement. Motion measurement device 140 outputs motion data relating angular velocity data and acceleration data for each of the three axes, as well as time information.

[0025] For example, the motion measurement device 140 includes a gyroscope (gyro sensor) for measuring angular velocity and an acceleration sensor for measuring acceleration as sensors. An example of the motion measurement device 140 is an inertial measurement unit (IMU). As will be described in detail later, the motion measurement device 140 can record motion data obtained through measurement on its own device.

[0026] The type of momentum measured by the motion measurement device 140, the number of dimensions, the measurement bandwidth, or the measurement range are not limited to the examples above. These specifications may be determined as appropriate depending on the motion and cost of the teaching object, observation object, teaching construction machine, or construction machine. The following describes an example in which the motion measurement device 140 measures the angular velocity and acceleration of each of the three orthogonal axes as momentum.

[0027] The motion measurement device 140A is mounted on the top surface of the cab 312B and is connected to the timing prediction device 110 via communication. When the motion measurement device 140A detects momentum, such as when the upper rotating body 312 rotates, it records the detected motion data in its own device. Alternatively, the motion measurement device 140A transmits the detected motion data to the timing prediction device 110.

[0028] The motion measurement device 140B is mounted on the side of the boom 321 and is connected to the timing prediction device 110 via communication. When the motion measurement device 140B detects motion, such as when the boom 321 rises or falls, it records the detected motion data in its own device. Alternatively, the motion measurement device 140B transmits the detected motion data to the timing prediction device 110.

[0029] The motion measurement device 140C is mounted on the side of the arm 322 and is connected to the timing prediction device 110 via communication. When the motion measurement device 140C detects motion, such as when the arm 322 is pushed out or pulled back, it records the detected motion data in its own device. Alternatively, the motion measurement device 140C transmits the detected motion data to the timing prediction device 110.

[0030] The foot switch 150 is mounted on the floor of the cab 312B and is connected to the timing prediction device 110. The foot switch 150 outputs a switch signal (e.g., a pulse signal) indicating the timing of the press when pressed by the foot of the operator of the backhoe 300.

[0031] Note that the foot switch 150 is just one example of a switch; instead of the foot switch 150, a button switch that outputs a switch signal when operated by the passenger's hand or fingers may be provided near the control stick of the cab 312B.

[0032] The posture of the backhoe 300 changes as the upper slewing body 312, boom 321, arm 322, etc., move when excavating or loading soil. The timing prediction device 110 predicts the timing of the completion of the backhoe 300's preparation work by analyzing the motion data obtained from the motion measurement device 140 in a time series in accordance with the changes in the backhoe 300's posture.

[0033] The timing prediction device 110 may be installed at a location separated from the backhoe 300, or it may be mounted on the backhoe 300, within a range where it can communicate with the motion measurement device 140 and the foot switch 150.

[0034] In the following description, the backhoe 300 is an example of a teaching object, observation object, teaching construction machine, and construction machine. Another specific example of a teaching object, observation object, teaching construction machine, and construction machine is a loading shovel. The backhoe 300 is a type of shovel that performs scooping work by moving the bucket 324 in a direction that pulls it towards the main body 310, while a loading shovel is a type of shovel that performs scooping work by holding the bucket facing forward and moving it in a pushing direction. Thus, it may be a type of shovel in which the movement of the bucket is different from that of the backhoe 300. In this way, any object or construction machine (including excavating machines) whose own posture changes can be used as a teaching object, observation object, construction machine, and teaching construction machine.

[0035] Figure 2 shows an example of the schematic functional configuration of the timing prediction system 100. The timing prediction system 100 comprises a timing prediction device 110, a motion measurement device 140, and a foot switch 150.

[0036] The timing prediction system 100 can operate by switching between a learning mode and an actual operation mode. The learning mode is an operation mode in which the timing prediction device 110 is taught an operation model, which will be described later, using a teaching object or a teaching construction machine. In the learning mode, the timing prediction device 110 executes the operation model learning process. The actual operation mode is an operation mode in which the timing of the completion of preparatory work operations is predicted using an observed object or construction machinery. In the actual operation mode, the timing prediction device 110 performs timing prediction processing for the actual operation.

[0037] As shown in Figure 2, the timing prediction device 110 includes a switch signal receiving unit 111, a timing specification unit 112, a motion data acquisition unit 113, a feature generation unit 114, a motion analysis unit 115, a motion model storage unit 116, a prediction unit 117, and an output unit 118.

[0038] Although not shown in Figure 2, the timing prediction device 110 may include a main control unit for selecting whether to operate the timing prediction system 100 in learning mode or actual operation mode, and for controlling the entire system according to the selected operation mode. For example, the main control unit may have an interface for switching between learning mode and actual operation mode via operation by an external operator or remote operation by an information processing device.

[0039] When the switch signal receiving unit 111 receives a switch signal transmitted from the foot switch 150, it outputs switch information indicating that the switch signal has been received to the timing specification unit 112.

[0040] When the timing specification unit 112 receives switch information from the switch signal receiving unit 111, it outputs a timing specification signal to the motion measurement device 140. In addition, depending on whether the main control unit is set to any of the operating modes, the timing specification unit 112 outputs a measurement start instruction signal to the motion measurement device 140 indicating an instruction to start measurement.

[0041] The motion data acquisition unit 113 acquires motion data from the motion measurement device 140 and outputs the acquired motion data to the feature generation unit 114. In this embodiment, the motion data acquisition unit 113 acquires motion data for the cab 312B, boom 321, and arm 322 of the backhoe 300 that performs motion operations, and outputs the acquired motion data to the feature generation unit 114.

[0042] The feature generation unit 114 generates time-series feature data by performing frequency analysis on the motion data input from the motion data acquisition unit 113, and outputs the generated feature data to the motion analysis unit 115.

[0043] The motion analysis unit 115 segments the time-series feature data input from the feature generation unit 114 and extracts a sequence of primitive motion transitions (also called a state transition sequence). Primitive motion is an action as an element. The motion analysis unit 115 segments the feature data based on a Hidden Markov Model (HMM) and extracts a sequence of primitive motion transitions. A Hidden Markov Model is one method of segmenting states, but in a basic Hidden Markov Model, it is necessary to determine the number of states in advance. However, in this embodiment, the motion of the backhoe 300 may differ depending on the operator of the backhoe 300, for example, how to level the ground with one's feet or how to scoop up soil. In other words, the number of segments in the preparation work motion may differ for each operator. Therefore, as a Hidden Markov Model to be applied to the motion analysis unit 115, there is an NPHMM (Non-Parametric Bayes Hidden Markov Model) which can update the number of state segments from an initial value.

[0044] Furthermore, as a method for segmenting time-series feature data and extracting the sequence of primitive movements, for example, a BP-HMM (Beta-Process Hidden Markov Model), which can estimate primitive movements in a data-driven manner for each time series, may be applied to the motion analysis unit 115. Alternatively, a Hierarchical Dirichlet Process Hidden Markov Model (HMM), which does not require a predetermined number of states, may be applied to the motion analysis unit 115. Alternatively, a convolutional neural network (CNN) may be applied to the motion analysis unit 115 to segment the time-series feature data and extract the sequence of primitive motion transitions.

[0045] In learning mode, the motion analysis unit 115 inputs a sequence of primitive motion transitions into the motion model stored in the motion model storage unit 116, thereby training the motion model. In actual operation mode, the motion analysis unit 115 outputs a sequence of primitive motion transitions to the prediction unit 117. The motion analysis unit 115 stores the learning results, merges the feature data input from the feature generation unit 114 with the stored learning results, performs analysis, and extracts a sequence of primitive motion transitions.

[0046] The motion model storage unit 116 stores motion models obtained by learning a sequence of primitive motion transitions obtained by segmenting the motion state of a backhoe 300 (an example of a teaching object or teaching construction machine) that performs motion movements.

[0047] In actual operation mode, the prediction unit 117 analyzes the frequency of primitive movements in the preparation work of the backhoe 300 based on the transition sequence input from the motion analysis unit 115 and the motion model stored in the motion model storage unit 116, and extracts primitive movement patterns. Then, the prediction unit 117 predicts the end timing of the preparation work based on the extracted primitive movement patterns. The prediction unit 117 outputs timing information indicating the predicted end timing to the output unit 118. The timing information includes, for example, information on the time remaining before the predicted end timing and the degree of confidence.

[0048] In actual operation mode, the output unit 118 outputs timing information input from the prediction unit 117 to the outside.

[0049] The motion measurement device 140 comprises a measurement unit 141, a control unit 142, and a recording unit 143. The measurement unit 141 measures the momentum related to the device itself. The control unit 142 controls the entire motion measurement device 140. The control unit 142 also has a timing function and acquires time information by measuring the elapsed time since receiving the measurement start instruction signal from the timing specification unit 112 of the timing prediction device 110. Alternatively, the control unit 142 may have a clock function and acquire time information by measuring the time.

[0050] In learning mode, the motion measurement device 140, under the control of the control unit 142, records motion data in the recording unit 143 that correlates the amount of momentum measured by the measurement unit 141 with time information obtained by timing. Furthermore, when the motion measurement device 140 receives a timing specification signal from the timing specification unit 112 of the timing prediction device 110, the control unit 142 records the timing at which the timing specification signal was received in the recording unit 143. The recording unit 143 is composed of, for example, a semiconductor memory device. The semiconductor memory device is, for example, a non-volatile memory that allows data rewriting.

[0051] In actual operation mode, the motion measurement device 140, under the control of the control unit 142, outputs motion data to the timing prediction device 110, relating the amount of momentum measured by the measurement unit 141 with time information obtained by timing.

[0052] Next, the operation of the timing prediction system 100 according to this embodiment will be explained by dividing it into the learning process of the operation model in the learning mode and the timing prediction process in the actual operation mode.

[0053] [Motion model learning process] This section describes the learning process for the motion model. In this explanation of the learning process, the backhoe 300 is an example of a teaching object or teaching construction machine. The operator of the backhoe 300 performs teaching operations (operation for teaching) to teach the motion to be the same as that of actual work.

[0054] When the power to the timing prediction device 110 and the motion measurement device 140 is turned on, the main control unit of the timing prediction device 110 initializes the timing prediction device 110, and the control unit 142 of the motion measurement device 140 initializes the motion measurement device 140. When the main control unit is set to learning mode by an external operation, the main control unit works in cooperation with the control unit 142 to perform the motion model learning process.

[0055] Figure 3 is a flowchart illustrating an example of the general procedure for the learning process of the behavioral model performed by the timing prediction system 100. The operator of the backhoe 300 performs the following tasks as part of the instruction work: leveling the ground under the lower vehicle 311, preparing to scoop up soil with the bucket 324 and begin loading it onto the dump truck, and signaling with the horn when the preparation work is complete.

[0056] In this instructional procedure, the operator presses the foot switch 150 when the backhoe 300 begins scooping up soil (indicated as "S"), when the backhoe 300 begins loading soil onto the dump truck (indicated as "L"), and when the horn is honked (indicated as "H").

[0057] When the teaching process begins, the motion measurement device 140 measures the amount of motion related to itself and also measures the time (for example, the elapsed time since the measurement start instruction signal was received), and records motion data relating the amount of motion and time information in itself. At the same time, if the motion measurement device 140 receives a timing specification signal output from the timing prediction device 110 when the foot switch 150 is pressed, it also records the timing at which the timing specification signal was received in itself (step S101).

[0058] Specifically, during the teaching process, the control unit 142 of the motion measurement device 140 records motion data in the recording unit 143, relating the amount of motion measured by the measurement unit 141 with the time information that has been timed.

[0059] During the teaching process, when the passenger presses the foot switch 150 at timings S, L, and H, the foot switch 150 outputs a switch signal. When the switch signal receiving unit 111 of the timing prediction device 110 receives the switch signal from the foot switch 150, it outputs switch information indicating that the switch signal has been received to the timing specification unit 112. When the timing specification unit 112 receives the switch information from the switch signal receiving unit 111, it outputs a timing specification signal to the motion measurement device 140. When the motion measurement device 140 receives a timing specification signal from the timing specification unit 112 of the timing prediction device 110, the control unit 142 records the timing at which the timing specification signal was received in the recording unit 143.

[0060] The motion data recorded in the recording unit 143 can be maintained after the teaching operation. Specifically, for example, the entire operation of the backhoe 300 during the teaching operation can be filmed (including recording) with a video camera, and the timings of the scenes corresponding to S, L, and H in the filmed video and recorded audio can be checked and corrected to match the timings of S, L, and H specified by the foot switch 150, and the motion data can be labeled to indicate S, L, and H. This maintenance can be performed manually by the operator, or it can be performed automatically by having a computer perform image recognition processing and voice recognition processing on the filmed video and recorded audio.

[0061] In addition, during the teaching process, the timing recording process to the recording unit 143 by pressing the foot switch 150 may be omitted, and S, L, and H labels may be assigned based on the captured video and recorded audio obtained from the video camera.

[0062] Figures 4A and 4B are graphs showing examples of motion data recorded by the recording unit 143 of the motion measurement device 140. Figure 4A shows the x-axis acceleration data [m / s²] against elapsed time [s] for motion measurement devices 140A, 140B, and 140C, respectively. 2 This is a graph of ]. Figure 4B is a graph of angular velocity data [deg / s] around the yaw angle against elapsed time [s] for motion measurement devices 140A, 140B, and 140C, respectively. In both Figure 4A and Figure 4B, the elapsed time represents the time elapsed since the start of the teaching process. In Figures 4A and 4B, the top row "A" corresponds to the motion data of motion measurement device 140A, the middle row "B" corresponds to the motion data of motion measurement device 140B, and the bottom row "C" corresponds to the motion data of motion measurement device 140C. In the graphs of Figures 4A and 4B, the straight lines (dotted lines) perpendicular to the time axis (horizontal axis) indicate the timing when the motion measurement device 140 receives the timing specification signal, i.e., when the foot switch 150 is pressed. The letters L, S, and H written near each line represent labels added during maintenance.

[0063] Figures 4A and 4B show that distinctive features are evident in the movements of the cab 312B, boom 321, and arm 322 of the backhoe 300 during the actions of scooping up soil, loading the scooped soil, and giving signals and waiting.

[0064] Let's return to the explanation of the flowchart in Figure 3. After recording the instructional motion data in the recording unit 143 and completing maintenance as necessary, the control unit 142 of the motion measurement device 140 controls the recording unit 143 to output the motion data recorded in it. When the motion data acquisition unit 113 of the timing prediction device 110 acquires motion data from the motion measurement device 140, it outputs the acquired motion data to the feature generation unit 114. The feature generation unit 114 performs frequency analysis on the motion data input from the motion data acquisition unit 113 to generate time-series feature data, and outputs the generated feature data to the motion analysis unit 115 (step S102).

[0065] Specifically, the feature generation unit 114 divides the acquired momentum data by type and axis into time windows with a window interval of 64 (0.32 seconds) and an operating window length of N = 128 (0.64 seconds), and then applies the window function Hann window of equation (1) to perform the Fast Fourier Transform process.

[0066]

number

[0067] Figure 5 is a graph showing an example of frequency intensity obtained by the feature generation unit 114 after performing a Fast Fourier Transform. The graph in Figure 5 shows the frequency intensities of three frequency components obtained by the feature generation unit 114 performing a Fast Fourier Transform on the x-axis acceleration data measured by the motion measurement device 140B installed on the boom 321 of the backhoe 300. The horizontal axis represents the elapsed time [s] after the start of the teaching operation. In the graph in Figure 5, the dotted line represents the frequency intensity when the frequency component is 0 [Hz]. The dashed line represents the frequency intensity when the frequency component is 1.5625 [Hz]. The solid line represents the frequency intensity when the frequency component is 3.1250 [Hz].

[0068] In the graph in Figure 5, during the period of approximately 160[s] to 175[s] of elapsed time, the motion of the cab 312B, boom 321, and arm 322 of the backhoe 300 is stopped. During this period of motion stoppage, a gravity component appears in the frequency intensity of the 0[Hz] and 1.5625[Hz] frequency components. This gravity component frequency intensity changes depending on the orientation of the boom 321 and arm 322. In other words, the feature generation unit 114 detects different acceleration features depending on the stationary posture of the backhoe 300. This is undesirable because, in the post-processing motion analysis unit 115's segmentation processing of feature data, the stationary state may be segmented into multiple primitive motions.

[0069] Furthermore, depending on the frequency components, superimposed components may appear in the frequency intensity due to the relationship between the vibrations generated by the engine mounted on the Backhoe 300 and the Backhoe 300's inherent resonant frequency.

[0070] Therefore, in order to prevent segmentation in the stationary state, the feature generation unit 114 extracts the frequency intensity of the frequency component with the lowest frequency intensity (3.1250 [Hz] in the example in Figure 5) when the backhoe 300 is stationary as acceleration feature data. Furthermore, the feature generation unit 114 compares the frequency components of the backhoe 300 when it is stationary and when it is in operation, selects a frequency band that contains frequency components caused by the preparation work operations of the backhoe 300, and extracts the frequency intensity in the selected frequency band.

[0071] In other words, the feature generation unit 114 selects a frequency band that contains frequency components in which features based on the motion of the cab 312B, boom 321, and arm 322 are more numerous than features based on vibration components unique to the backhoe 300, and extracts frequency intensity in the selected frequency band.

[0072] Furthermore, since angular velocity is not affected by gravity as described above, the frequency intensity of the 0 Hz frequency component, which largely reflects the motion of the backhoe 300, is extracted as the angular velocity feature data.

[0073] Let's return to the explanation of the flowchart in Figure 3. When the feature generation unit 114 generates time-series feature data, the motion analysis unit 115 segments the feature data to extract a sequence of primitive motion transitions, and uses the extracted sequence of transitions to train the motion model stored in the motion model storage unit 116 (step S103).

[0074] Specifically, the motion analysis unit 115 segments time-series feature data based on, for example, an NPHMM to extract a sequence of primitive motion transitions, and then uses the extracted sequence of transitions to train a motion model.

[0075] Here, between the scooping action of the backhoe 300 and the loading of the scooped soil onto the dump truck, there is a rotational movement of the backhoe 300, and this rotational movement can serve as a guide for switching between tasks. Therefore, in order to use the change in rotational speed for timing prediction, the transition sequence may be rearranged and used with feature data based on angular velocity around the yaw angle measured by the motion measurement device 140A installed on the top surface of the cab 312B as a key.

[0076] Figure 6 is a graph showing an example of the transition of primitive movements in a motion model stored in the motion model storage unit 116. The graph in Figure 6 shows the state transitions of the backhoe 300, which are segmented into 18 states. The horizontal axis represents the elapsed time [s] after the start of the teaching task, and the vertical axis represents the ID (identification number) indicating the primitive motion. In the graph of Figure 6, the straight line (dotted line) perpendicular to the time axis (horizontal axis) indicates the timing when the motion measurement device 140 receives the timing specification signal, that is, when the foot switch 150 is pressed. The letters L, S, and H written near each line represent labels added during maintenance.

[0077] [Timing prediction processing in actual operation] This section describes the timing prediction process in actual operation of the timing prediction system 100, which uses the timing prediction device 110 that has completed learning the motion model. In this description of the timing prediction process, the backhoe 300 is an example of an object to be observed or a construction machine. The operator of the backhoe 300 operates the machine for the actual work related to the preparation work.

[0078] When the power to the timing prediction device 110 and the motion measurement device 140 is turned on, the main control unit of the timing prediction device 110 initializes the timing prediction device 110, and the control unit 142 of the motion measurement device 140 initializes the motion measurement device 140. When the main control unit is set to the actual operation mode by an external operation, the main control unit works in cooperation with the control unit 142 to perform the timing prediction processing in the actual operation.

[0079] Figure 7 is a flowchart illustrating an example of the general procedure for the timing prediction process performed by the timing prediction system 100 during actual operation.

[0080] When the actual work related to the preparatory work begins, the motion measurement device 140 measures the amount of motion related to itself and also measures the time (for example, the elapsed time since the measurement start instruction signal was received), and outputs motion data relating the amount of motion and time information to the timing prediction device 110 (step S111).

[0081] When the motion data acquisition unit 113 of the timing prediction device 110 acquires motion data from the motion measurement device 140, it outputs the acquired motion data to the feature generation unit 114. The feature generation unit 114 performs frequency analysis on the motion data input from the motion data acquisition unit 113 to generate time-series feature data, and outputs the generated feature data to the motion analysis unit 115 (step S112).

[0082] The motion analysis unit 115 segments the feature data output by the feature generation unit 114 to extract the sequence of primitive motion transitions, and outputs the extracted sequence of transitions to the prediction unit 117 (step S113).

[0083] The prediction unit 117 analyzes the frequency of primitive movements in the preparation work of the backhoe 300 based on the transition sequence input from the motion analysis unit 115 and the motion model stored in the motion model storage unit 116, and extracts primitive movement patterns. Then, the prediction unit 117 predicts the completion timing of the preparation work based on the extracted primitive movement patterns and outputs timing information indicating the predicted completion timing to the output unit 118 (step S114).

[0084] The process by which the prediction unit 117 predicts the completion timing of the preparation work will be explained in detail. Primitive actions included in the transition sequence may appear in common throughout a series of work processes. For example, the work of scooping up soil and the work of leveling the soil at one's feet both involve the action of scooping up soil, so the primitive actions included in these common actions appear in both work processes. Therefore, the prediction unit 117 divides the preparatory work actions into, for example, the following three stages, and predicts the timing at which the preparatory work actions will be completed by sequentially capturing the primitive actions that characterize the actions of each stage.

[0085] First, labels are attached during each of the operations before the loading of earth and sand, during the loading of earth and sand, and other operations. Note that the label attachment may be performed based on a switch signal input from the foot switch 150 pressed by the passenger of the backhoe 300 at the timing of starting the loading.

[0086] The preparatory operation of the backhoe 300 is divided into three stages of operations f1 to f3 based on the attached label. Operation f1: Scoop up earth and sand. Operation f2: While lifting the scooped earth and sand, turn toward the planned loading position. Operation f3: Gradually slow down the turning speed toward the planned loading position.

[0087] The prediction unit 117 extracts the primitive operation q k characterizing each operation f k from the transition sequence related to the preparatory operation. To extract the prediction pattern common to all preparatory operations, the frequency of occurrence p appe (f k , i) = 1 is satisfied for i, and i is extracted as the primitive operation q k . However, let the number of operations f k in which i appears be n(f k , i), and the total number of preparatory operations be N A .

[0088]

Number

[0089] Next, the prediction unit 117 uses the primitive operation q k characterizing each operation f k at each stage of the preparatory operation to extract the pattern S (boldface) key for specifying the end timing of the preparatory operation. Note that “(boldface)” means that the character “S” immediately preceding it is represented in boldface. Pattern S (boldface) keyThe condition for connecting the primitive actions within is O(q k ,q k+1 The primitive motion r expressed by equation (3) may be obtained by state analysis of the transition sequence output by the motion analysis unit 115, or by observation. ku Using this, pattern S (bold font) key This can be expressed as equation (4).

[0090]

number

[0091]

number

[0092] Pattern S (bold font) key Regarding this, in order to evaluate the accuracy of predicting the completion timing of the preparation work, the prediction unit 117 calculates the confidence level p for each stage g(k). conf Determine (k).

[0093] Specifically, the prediction unit 117 uses a positive integer i to determine the pattern S (bold font). key The stages for capturing the primitive movements within are defined as follows: Stage g(1): Detect q1. Step g(2i): After step g(2i-1) O(q i ,q i+1 It continuously detects primitive actions that satisfy the following conditions. Stage g(2i+1): After stage g(2i), q i+1 Detect primitive actions that satisfy the following conditions.

[0094] n is the number of steps in leveling the soil that has progressed to stage g(k). B (k) Then, the confidence level p is the confidence level to capture the end of the preparatory work action at each stage. conf (k) is expressed as equation (5).

[0095]

number

[0096] The output unit 118 outputs the timing information input from the prediction unit 117 to the outside (step S115). For example, an information processing device may be connected to the output unit 118, and the timing information output by the output unit 118 may be acquired by the information processing device. The information processing device may be, for example, a computer or a portable information terminal. Based on the acquired timing information, the information processing device may calculate the predicted time when the preparation work operation will be completed or the time remaining until completion, according to the confidence level, and present this information to the user of the information processing device.

[0097] Next, a motion model is created and trained using 12 sets of training momentum data related to preparatory work movements, and then pattern S (bold) is created using 4 other sets of evaluation momentum data related to preparatory work movements. key We will now explain the results of the experiment in which the extraction was performed. Note that each of the 16 preparation work operations included 4 to 5 soil loading operations per set.

[0098] By training a motion model based on training momentum data, 14 primitive movements were segmented from the time-series feature data related to preparatory work movements, as shown in Figure 8. Figure 8 is a graph showing an example of primitive motion transitions by a motion model trained based on learning momentum data. This graph represents the state transitions of the motion related to the preparation work of the backhoe 300, which is segmented into 14 states. The horizontal axis is the elapsed time [s] after the start of the teaching work, and the vertical axis is the ID (identification number) representing the primitive motion. As shown in the graph of Figure 8, in the 12 sets of main work, each motion f k The occurrence rate p appe (f k The IDs of primitive actions q1, q2, and q3 that satisfy i)=1 were detected to be 6,2, and 11 (F(1), F(2), and F(3) in Figure 8).

[0099] primitive behavior q k The condition between O(q k ,q k+1 The following observations were made: Condition O(q1,q2): primitive behavior r 1u The ID will not be smaller than q1. Condition O(q1,q2): q2 is detected multiple times. If the ID of the primitive action is 2 or more less than q2, the primitive action r 2u The ID continues to decrease down to q3.

[0100] Figure 9 is a graph showing an example of the relationship between the time before the completion of a preparatory work action and the confidence level. In the graph in Figure 9, the horizontal axis represents the time before the completion (end time) set to 0 (zero), and the vertical axis represents the confidence level. Condition O(q) based on training momentum data k ,q k+1 Pattern S (bold font) reflecting ) key The relationship between the time before the end of the preparation work and the confidence level was as shown by the dashed line in Figure 9. According to the dashed line graph in Figure 9, the timing prediction device 110 predicted the end time with a confidence level of approximately 0.6 (probability of approximately 60%) from approximately 4.5 [s] before the end time to just before the end time, based on the learning momentum data.

[0101] Furthermore, pattern S (bold font) was extracted based on the momentum data used for evaluation. key The results of predicting the end of four preparatory work operations using this method are shown by the solid line in Figure 9. According to the solid line graph in Figure 9, the timing prediction device 110 predicted the end of the operation with a confidence level of approximately 0.67 (probability of approximately 67%) from approximately 6 seconds before the end to approximately 0.8 seconds before the end, and with a confidence level of 1 (probability of 100%) from approximately 0.8 seconds before the end to just before the end, based on the evaluation momentum data.

[0102] As described above, in this embodiment, motion data is acquired from three locations on the backhoe 300 (observed object) performing the preparatory work operation. Feature data is generated by frequency analysis of the motion data. A sequence of primitive motion transitions is extracted by segmenting the feature data. Based on the sequence of transitions and a motion model obtained by learning the sequence of primitive motion transitions obtained by segmenting the motion state of the backhoe 300 (training object) performing the motion operation, the frequency of primitive motion occurrences in the preparatory work operation is analyzed to extract a pattern of primitive motion, thereby predicting the timing of the completion of the preparatory work operation.

[0103] With this configuration, it is possible to predict the timing of the completion of the preparation work for the backhoe 300 (observed object) that performs the motion.

[0104] Furthermore, in this embodiment, the preparatory work operations performed by the backhoe (observation object) are divided into three operations. For each divided operation, the frequency of occurrence of primitive operations is analyzed based on the extracted transition sequence and operation model to extract primitive operation patterns. The patterns are evaluated step by step to calculate the confidence level, and the timing of the completion of the preparatory work operations is predicted according to the confidence level.

[0105] In this way, by focusing on multiple segmented actions and evaluating the patterns step by step, the accuracy of predicting the completion timing of preparatory work actions can be improved.

[0106] Furthermore, the number of divisions required for the analysis to predict the completion timing of the preparatory work is not limited to three. Divisions may not be necessary, or the number of divisions may be determined according to the characteristic movements included in the work or the structure related to the movement of the backhoe 300.

[0107] Furthermore, in this embodiment, by performing frequency analysis on the acquired motion data, a frequency band containing frequency components resulting from preparatory work actions is selected, and feature data is generated in the selected frequency band.

[0108] With this configuration, for example, the influence of gravity components and the natural vibrations of the backhoe 300 can be eliminated or suppressed, allowing for accurate and high-quality frequency analysis of the backhoe 300's momentum.

[0109] Furthermore, in this embodiment, a hidden Markov model is applied to the behavioral model, and the feature data is segmented and the sequence of primitive behavioral transitions is extracted by computational processing using the hidden Markov model.

[0110] This configuration allows for the extraction of detailed operational elements included in the preparation work of the backhoe 300, and enables a detailed capture of their transitions.

[0111] Furthermore, in this embodiment, the motion data includes angular velocity data and acceleration data.

[0112] With this configuration, inertial data of the backhoe 300's operation can be acquired, and the characteristics of its operation can be extracted efficiently and effectively.

[0113] [Example 1] While the backhoe 300 primarily scoops up soil and sand, the operation of the backhoe 300, particularly the operation of the bucket 324, can vary subtly depending on the type of soil or sand being handled. For example, there may be differences in the operation of the arm 322 and bucket 324 when handling materials with relatively high mass per unit volume and resistance during scooping, such as crushed stone, compared to materials with relatively low mass per unit volume and resistance during scooping, such as wood chips. In such cases, the operations characteristic of the material being scooped are segmented and extracted as primitive operations.

[0114] Therefore, the prediction unit 117 may extract the characteristics of the major (basic and fundamental) motion of scooping up the object to be scooped and rotating it, and perform analysis processing in a way that ignores the smaller (auxiliary) motions that fine-tune the motion according to the differences in the object to be scooped.

[0115] By performing this analysis process, the prediction unit 117 can predict the timing of the completion of the preparation work without being affected by differences in the material being scooped. For example, it can predict the timing of the completion of the preparation work with a stable degree of confidence, regardless of whether or not light soil is mixed in with gravel or other soil, and regardless of the proportion of light soil that is mixed in.

[0116] [Differentiation 2] The timing prediction device 110 may be mounted on a dump truck that receives soil from the backhoe 300 to constitute a construction machinery system. Specifically, a construction machinery system including motion measurement devices 140A, 140B, and 140C installed on the cab 312B, boom 321, and arm 322 of a backhoe 300, which is an example of construction machinery, and the timing prediction device 110 mounted on a dump truck may have the following configuration.

[0117] An information processing device is connected to the output unit 118 of the timing prediction device 110. An output device is connected to the information processing device. The information processing device may calculate the predicted time when the preparation work operation will be completed and the time remaining until completion based on the timing information acquired from the timing prediction device 110, according to the confidence level, and present the calculated time information to the dump truck driver using the output device.

[0118] In other words, this construction machinery system implements a reminder system (alarm system) that notifies the dump truck driver of predictive information such as "loading from the backhoe 300 will begin in X seconds."

[0119] As a concrete example, an audio output device may be connected to the information processing device, and by performing speech recognition processing on the calculated time information and announcement information indicating that loading preparations are complete, an audio notification such as "Loading preparations will be complete in ○ seconds" may be output from the audio output device.

[0120] Another specific example is to connect a display device to the information processing device and display information such as text and graphics on the display device, based on the calculated time information and announcement information indicating that loading preparations are complete.

[0121] Furthermore, in this construction machinery system, the timing prediction device 110 may be mounted on the backhoe 300, and the information processing device and output device may be mounted on the dump truck.

[0122] [Difference 3] The backhoe 300 performs a series of repetitive actions: leveling the soil, scooping it up, and rotating to load it. Therefore, by having the motion measurement device 140 acquire a series of momentum data for the information processing device, and the information processing device statistically analyzes and recognizes the posture of the boom 321 and arm 322 based on the series of momentum data, and then calculating the position of the bucket 324 based on the recognized posture, the loading position of the bucket 324 can be predicted.

[0123] This allows for the prediction of when the backhoe 300 is ready to load and the loading position of the bucket 324, thereby supporting the automated operation of dump trucks.

[0124] The motion measurement device 140 may also be equipped with a GPS, which can receive positioning data from satellites to measure the device's position and output the resulting position data. The position data may include, for example, latitude, longitude, and altitude information. The information processing device may acquire the position data and calculate the position of the bucket 324 based on the position data.

[0125] Furthermore, the dump truck may also be equipped with GPS. The motion measurement device 140 and the dump truck measure their respective positions and output the measured position data to the information processing device, which can then obtain the relative distance and direction between the bucket 324 and the dump truck. Based on the obtained relative distance and direction, the information processing device can then estimate the state of loading onto the dump truck by the backhoe 300. This allows for greater accuracy in determining the completion timing of preparatory work.

[0126] According to the above construction machinery system, the backhoe 300 can easily move and park the dump truck at the loading position while leveling, scooping, and turning to complete the loading preparation, thereby achieving efficient loading operations.

[0127] [Differentiation Example 4] Furthermore, in this embodiment, the motion measurement device 140 is installed in three locations: the cab 312B, the boom 321, and the arm 322, to measure the motion of the backhoe 300. However, the installation locations of the motion measurement device 140 are not limited to this example. For example, the motion measurement device 140 may be installed in a location that does not interfere with excavation or scooping, such as the outer side of the bucket 324. In addition, the motion measurement device 140 may be installed in a position between the bucket link 323 and the bucket 324, in addition to the three locations of the cab 312B, boom 321, and arm 322.

[0128] This allows for motion analysis that captures the subtle movements of bucket 324.

[0129] Figure 10 shows an example of a schematic hardware configuration of a computer 1000 that functions as a timing prediction device 110. The computer 1000 according to this embodiment comprises a CPU peripheral unit having a CPU (Central Processing Unit) 1200, RAM (Random Access Memory) 1300, and graphics controller 1400 which are interconnected by a host controller 1100, and an input / output unit having a ROM (Read Only Memory) 1600, communication I / F (interface) 1700, hard disk drive 1800, and input / output chip 1900 which are connected to the host controller 1100 by an input / output controller 1500.

[0130] The CPU 1200 operates based on programs stored in the ROM 1600 and RAM 1300, and controls each component. The graphics controller 1400 acquires image data generated by the CPU 1200 and other components on a frame buffer provided in the RAM 1300 and displays it on the display. Alternatively, the graphics controller 1400 may include an internal frame buffer for storing the image data generated by the CPU 1200 and other components.

[0131] The communication interface 1700 communicates with other devices via a network, either wired or wirelessly. The communication interface 1700 also functions as hardware for communication. The hard disk drive 1800 stores programs and data used by the CPU 1200.

[0132] ROM 1600 stores the boot program that computer 1000 executes when it starts up, as well as programs that depend on the computer 1000's hardware. The input / output chip 1900 connects various input / output devices to the input / output controller 1500 via, for example, a parallel port, serial port, keyboard port, mouse port, etc.

[0133] The program provided to the hard disk drive 1800 via RAM 1300 is stored on a recording medium such as an IC (Integrated Circuit) card and provided by the user. The program is read from the recording medium, installed on the hard disk drive 1800 via RAM 1300, and executed by the CPU 1200.

[0134] A program installed on the computer 1000, which causes the computer 1000 to function as a timing prediction device 110, may act on the CPU 1200 and other components to cause the computer 1000 to function as each part of the timing prediction device 110. The information processing code described in these programs is read into the computer 1000 and functions as a switch signal receiving unit 111, a timing specification unit 112, a motion data acquisition unit 113, a feature quantity generation unit 114, a motion analysis unit 115, a motion model storage unit 116, a prediction unit 117, and an output unit 118, which are specific means of cooperation between the software and the various hardware resources described above. Then, by realizing the calculation or processing of information according to the purpose of use of the computer 1000 in this embodiment, a timing prediction device 110 specific to the purpose of use is constructed.

[0135] In addition, in computer 1000, the host controller 1100, CPU 1200, RAM 1300, graphics controller 1400, input / output controller 1500, and ROM 1600 may be implemented as a control unit 1001.

[0136] Figure 11 shows an example of the configuration of the control unit 1001 in a modified example. The control unit 1001 includes a host controller 1100, a CPU 1200, RAM 1300, a graphics controller 1400, an input / output controller 1500, and a ROM 1600.

[0137] The CPU 1200, graphics controller 1400, and input / output controller 1500 read programs stored in RAM 1300, ROM 1600, and hard disk drive 1800, and execute the read programs, thereby allowing the computer 1000 (i.e., the timing prediction device 110) to function as a device comprising a control unit 1001, communication interface 1700, hard disk drive 1800, and input / output chip 1900.

[0138] Furthermore, the CPU 1200, graphics controller 1400, and input / output controller 1500 read programs stored in RAM 1300, ROM 1600, and hard disk drive 1800, and execute the read programs, thereby allowing the control unit 1001 to function as a device comprising a timing specification unit 112, motion data acquisition unit 113, feature generation unit 114, motion analysis unit 115, and prediction unit 117. In other words, the control unit 1001 comprises a timing specification unit 112, a motion data acquisition unit 113, a feature generation unit 114, a motion analysis unit 115, and a prediction unit 117. Note that the input / output chip 1900 and communication I / F 1700 are examples of the switch signal receiving unit 111.

[0139] The hard disk drive 1800 is an example of the operating model storage unit 116. The hard disk drive 1800 is a computer-readable storage medium device such as a magnetic hard disk drive or a semiconductor storage device.

[0140] The motion model is obtained by learning the sequence of primitive motion transitions, as described above. Furthermore, as described above, a hidden Markov model is applied to the motion model, and the motion model extracts the sequence of primitive motion transitions by segmenting the feature data through computational processing using the hidden Markov model. Also as described above, in the learning mode, the motion analysis unit 115 inputs the sequence of primitive motion transitions into the motion model and trains the motion model. In the actual motion mode, the motion analysis unit 115 extracts the sequence of primitive motion transitions by segmenting the time-series feature data.

[0141] As mentioned above, the Hidden Markov Model is one example of a method used to segment time-series feature data and extract a sequence of primitive actions. Therefore, the action model is a mathematical model that shows the relationship between time-series feature data and the sequence of primitive actions, and in learning mode, it is a mathematical model (i.e., a learning model) that is updated by machine learning methods. More specifically, one example of a method for extracting a sequence of primitive actions is to segment time-series feature data using the Markov chain Monte Carlo (MCMC) algorithm, which applies the Hidden Markov Model. The Hidden Markov Model models the transition probabilities between primitive actions.

[0142] Therefore, more specifically, the behavior model is a learning model that estimates the sequence of primitive behaviors based on time-series feature data. The machine learning method can be unsupervised or supervised. For example, the Hidden Markov Model described above can be an unsupervised Hidden Markov Model. In the case of supervised learning, the dataset used for learning consists of time-series feature data for the training data (i.e., the data on the explanatory variables) and a sequence of primitive behaviors for the ground truth data (i.e., the data on the target variable).

[0143] The motion analysis unit 115, in the actual operation mode, uses a trained motion model to estimate the sequence of primitive motions based on the input time-series feature data. Therefore, the sequence of primitive motions estimated by the motion analysis unit 115 is the extracted sequence of primitive motions described above. A trained model means that a predetermined termination condition has been met for the learning process. The predetermined termination condition may be, for example, that a predetermined number of learning sessions have been performed. Alternatively, the predetermined termination condition may be, for example, that the change in the trained model due to learning is less than a predetermined change.

[0144] Updating a learning model means adjusting the parameter values ​​in the learning model to a suitable level. This adjustment process might involve, for example, reducing a predetermined loss. The loss, for instance, represents the difference between the estimation result using the learning model and the ground truth data.

[0145] Furthermore, updating a learning model also means that the circuits that represent the learning model—such as electronic circuits, electrical circuits, optical circuits, and integrated circuits—are updated through learning. When a circuit is updated through learning, it means that the values ​​of the circuit's parameters are updated. By updating the values ​​of the parameters of the circuit that represents the learning model, the values ​​of the parameters of the learning model represented by the circuit are updated.

[0146] Furthermore, the timing prediction device 110 does not necessarily have to perform the operation in learning mode. The timing prediction device 110 may not perform the operation in learning mode, but instead use a learned operation model generated by an external device to perform the operation in actual operation mode. Also, the timing prediction device 110 does not necessarily have to perform the operation in actual operation mode. The timing prediction device 110 may not perform the operation in actual operation mode, but instead perform the operation in learning mode.

[0147] As described above, the motion measurement device 140 outputs motion data that associates angular velocity data and acceleration data for each of the three axes with time information. Therefore, the feature data generated by the feature generation unit 114 is, for example, 18 types of feature data obtained from each motion data of the 18-dimensional motion data. The 18-dimensional motion data consists of a total of 18 types of motion data showing the angular velocity and acceleration in the three orthogonal axes for each of the arm 322, boom 321, and cab 312B. Hereinafter, the set of 18 types of feature data obtained from each motion data of the 18-dimensional motion data will be referred to as the 18-dimensional feature data set.

[0148] Let's explain the contents of each element in the 18-dimensional feature data set in more detail. Six of the 18 types are feature data obtained from the six types of motion data, namely the angular velocity and acceleration for each of the three orthogonal axes of arm 322. Hereafter, the six types of feature data obtained from the six types of motion data, namely the angular velocity and acceleration for each of the three orthogonal axes of arm 322, will be referred to as the arm 6-dimensional feature data.

[0149] Six of the 18 types are feature data obtained from the six types of motion data, specifically the angular velocity and acceleration for each of the three orthogonal axes of the boom 321. Hereinafter, the six types of feature data obtained from the six types of motion data, specifically the angular velocity and acceleration for each of the three orthogonal axes of the boom 321, will be referred to as boom 6-dimensional feature data.

[0150] Six of the 18 types are feature data obtained from the six types of motion data, namely angular velocity and acceleration for each of the three orthogonal axes of the CAB 312B. Hereinafter, the six types of feature data obtained from the six types of motion data, namely angular velocity and acceleration for each of the three orthogonal axes of the CAB 312B, will be referred to as CAB 6-dimensional feature data.

[0151] Thus, the 18-dimensional feature data set is a collection of 6-dimensional feature data for 6 types of arms, 6-dimensional feature data for 6 types of booms, and 6-dimensional feature data for 6 types of cabs.

[0152] 18-dimensional feature data can be represented, for example, by a tensor having a total of 18 elements, each containing the value of one of the 18 different feature data points. A tensor with a total of 18 elements is, for example, an 18-dimensional vector.

[0153] Note that the feature data generated by the feature generation unit 114 does not necessarily have to be the 18 types of feature data from the 18-dimensional feature data set. The feature data generated by the feature generation unit 114 may be the three types of feature data from the basic feature data set. The basic feature data set is a set of three types of feature data: arm 1D feature data, boom 1D feature data, and cab 1D feature data.

[0154] The arm one-dimensional feature data is obtained from motion data showing the angular velocity of arm 322 in the Z-axis direction. The boom one-dimensional feature data is obtained from motion data showing the angular velocity of boom 321 in the Z-axis direction. The cab one-dimensional feature data is obtained from motion data showing the angular velocity of cab 312B in the Z-axis direction.

[0155] The Z-axis direction of arm 322 is parallel to the rotation axis of the joint between arm 322 and boom 321. The central axis of arm 322 is the axis connecting the joints at both ends of arm 322. Thus, the angular velocity in the Z-axis direction of arm 322 is the angular velocity corresponding to the joint angle of arm 322.

[0156] The Z-axis direction of the boom 321 is parallel to the rotation axis of the joint between the boom 321 and the upper slewing body 312. The central axis of the boom 321 is the axis connecting the joints at both ends of the boom 321. Thus, the angular velocity in the Z-axis direction of the boom 321 is the angular velocity corresponding to the joint angle of the boom 321.

[0157] The Z-axis direction of the cab 312B is parallel to the rotation axis of the upper rotating body 312. The central axis of the cab 312B is perpendicular to the ground surface on which the backhoe 300 is located. Thus, the angular velocity in the Z-axis direction of the cab 312B is the angular velocity corresponding to the joint angle of the cab 312B.

[0158] Therefore, the three angular velocities in the Z-axis direction of the arm 322, boom 321, and cab 312B are all three-dimensional angular velocities coaxial with the joint. Thus, the angular velocities in the Z-axis direction of the arm 322, boom 321, and cab 312B are the angular velocities of the rotational movement around the joint axis corresponding to the motion of the backhoe 300, which is the movement caused by the motion of the backhoe 300.

[0159] The corresponding joint axis for arm 322 is the Z-axis direction of arm 322. The Z-axis direction of arm 322 is parallel to the rotation axis of the joint between arm 322 and boom 321. The corresponding joint axis for boom 321 is the Z-axis direction of boom 321. The Z-axis direction of boom 321 is parallel to the rotation axis of the joint between boom 321 and cab 312B. The corresponding joint axis for cab 312B is the Z-axis direction of cab 312B. The Z-axis direction of cab 312B is perpendicular to the ground surface on which the backhoe 300 is located, or parallel to the rotation axis of the joint between cab 312B and lower traveling body 311.

[0160] In other words, the angular velocity of arm 322 in the Z-axis direction is the angular velocity of the rotational movement of arm 322 around its joint axis caused by the motion of the backhoe 300. The angular velocity of boom 321 in the Z-axis direction is the angular velocity of the rotational movement of boom 321 around its joint axis caused by the motion of the backhoe 300. The angular velocity of cab 312B in the Z-axis direction is the angular velocity of the rotational movement of cab 312B around its joint axis caused by the motion of the backhoe 300.

[0161] Note that the preparation work is just one example of a motion. Therefore, the three types of feature data in the basic feature data set generated by the feature generation unit 114 may be feature data obtained from motion data showing the angular velocity of movement that occurs as a result of a motion and has a component of movement around the rotation axis of the corresponding joint. In other words, the three types of feature data in the basic feature data set generated by the feature generation unit 114 may be the angular velocity of rotational movement around the joint axes of the arm 322, boom 321, and cab 312B that occurs as a result of a motion.

[0162] Since the angular velocities in the Z-axis direction of the arm 322, boom 321, and cab 312B are angular velocities of motion with a component in the vertical direction, the angular velocities in the Z-axis direction of the arm 322, boom 321, and cab 312B are more frequently affected by steering motion than angular velocities in other directions. Features that strongly reflect the influence of steering motion strengthen the dependence of the motion model on steering motion and have the effect of preventing gravity, swaying of the machine, etc., from segmenting the motion.

[0163] Since the backhoe 300 is a mining machine, it moves in the Z-axis direction more frequently than in other directions. Therefore, the timing prediction device 110 can sometimes predict the completion timing of the preparation work with higher accuracy by using the three types of feature data from the basic feature data set rather than using the 18 types of feature data from the 18-dimensional feature data set. The reason for this is as follows.

[0164] When using 18 types of feature data from an 18-dimensional feature data set as feature data, the timing prediction device 110 also uses information where the influence of gravity is relatively weak. Using 18 types of feature data from an 18-dimensional feature data set provides more information than using 3 types of feature data from a basic feature data set. However, information is mixed with noise. Therefore, as the amount of information increases, there is a possibility that the amount of noise information will exceed the amount of information about the influence of the piloting actions.

[0165] Since noise information may outweigh information about the effects of piloting actions, a larger number of feature data types is not necessarily preferable for the timing prediction device 110 to use for prediction. For this reason, it may be preferable for the feature data used for prediction by the timing prediction device 110 to consist of fewer types of feature data than the 18 types of feature data, such as the three types of feature data in the basic feature data set.

[0166] Furthermore, the feature data obtained from acceleration, one of the 18 types of feature data in the 18-dimensional feature data set, is more strongly influenced by the posture of the backhoe 300 and the posture of the motion measurement device 140 than by angular velocity. Therefore, the feature generation unit 114 does not necessarily need to generate feature data obtained from acceleration.

[0167] Furthermore, if the three types of feature data from the basic feature data set are used instead of the 18 types of feature data from the 18-dimensional feature data set, the feature generation unit 114 may be input either all 18 types of motion data or only the three types of motion data. The feature generation unit 114 only needs to be input motion data representing the sum of the three types of angular velocities in the Z-axis direction of the arm 322, boom 321, and cab 312B. Even if all 18 types of motion data are input, the feature generation unit 114 only needs to use the motion data representing the sum of the three types of angular velocities in the Z-axis direction of the arm 322, boom 321, and cab 312B to generate the three types of feature data from the basic feature data set.

[0168] An example of the experimental results from the first evaluation experiment conducted using the timing prediction device 110 is shown. The first evaluation experiment was an experiment to compare the accuracy of prediction by the timing prediction device 110 when 18 types of feature data from an 18-dimensional feature data set were used as feature data, and when 3 types of feature data from basic feature data were used.

[0169] Figure 12 shows an example of the experimental results of the first evaluation experiment in a modified example. Figure 12 shows three experimental results: Result D101, Result D102, and Result D103. Result D101 shows the confidence level of the timing prediction device 110's prediction of the completion timing of the preparation work when an experienced operator A is operating a small machine α. Result D102 shows the confidence level of the timing prediction device 110's prediction of the completion timing of the preparation work when an experienced operator A is operating a large machine β. Result D103 shows the confidence level of the timing prediction device 110's prediction of the completion timing of the preparation work when an inexperienced operator B is operating a large machine β. Both the small machine α and the large machine β are examples of backhoe 300.

[0170] Experienced pilot A had a total flight time of 90 hours and an average pre-loading operation time of (19.5 ± 3.3) seconds. Less experienced pilot B had a total flight time of 35 hours and an average pre-loading operation time of (31.35 ± 13.0) seconds.

[0171] The smaller machine α has an arm length of 10.1 meters and a bucket volume of 1.4 cubic meters, and its swaying during operation is less than that of the larger machine β. The larger machine β has an arm length of 12.6 meters and a bucket volume of 2.8 cubic meters, and its swaying during operation is greater than that of the smaller machine α.

[0172] D3 indicates that the feature data used by the timing prediction device 110 consisted of three types of feature data from the basic feature data set. 18 This indicates that the feature data used by the timing prediction device 110 consisted of 18 types of feature data from an 18-dimensional feature data set.

[0173] This section describes four datasets: Dataset A, Dataset B, Dataset C, and Dataset D. Each of the four datasets represents a portion of data from 16 loading operations performed with the same operator and the same backhoe. In the experiment, the 16 data points were divided into four sets of four data points each: Set 1, Set 2, Set 3, and Set 4. That is, Sets 1, 2, 3, and 4 each contained four data points, and no two sets contained the same data.

[0174] In the experiment, three sets of data from sets 1 through 4 were used for training, and the remaining set was used for evaluation. The difference between the four datasets lies in the combination of the three sets used for training and the one set used for evaluation. Specifically, one of the four datasets used sets 1, 2, and 3 for training and set 4 for evaluation. Another dataset used sets 2, 3, and 4 for training and set 1 for evaluation. Yet another dataset used sets 3, 4, and 1 for training and set 2 for evaluation. The final dataset used sets 4, 1, and 2 for training and set 3 for evaluation.

[0175] The confidence level has a maximum value of 1, and the closer the value is to 1, the higher the degree of confidence. In other words, a confidence level closer to 1 indicates a higher probability that the prediction is correct.

[0176] Result D102 indicates that when the feature data consists of 18 types of feature data from an 18-dimensional feature data set, the timing prediction device 110 was unable to make a prediction with a confidence level of 1.0. On the other hand, result D102 indicates that when the feature data consists of 3 types of feature data from a basic feature data set, the timing prediction device 110 was able to make a prediction with a confidence level of 1.0.

[0177] Results D101 to D103 indicate that when the feature data used by the timing prediction device 110 consists of the three types of basic feature data, a prediction with a confidence level of 1.0 was made in all of the results D101 to D103.

[0178] Thus, Figure 12 shows that a larger number of species in the feature dataset is not necessarily always better.

[0179] An example of the experimental results from the second evaluation experiment conducted using the timing prediction device 110 is shown. In the second evaluation experiment, three types of feature data from the basic feature data set were used as feature data. The second evaluation experiment was an experiment to evaluate the accuracy of prediction by the timing prediction device 110 in the case where the position and orientation of the inertial measurement device in the actual operation mode differed from the position and orientation of the inertial measurement device at the time of acquisition of the data used for learning in the learning mode. Specifically, the inertial measurement device is the motion measurement device 140. Hereafter, the inertial measurement device refers to the motion measurement device 140. Figures 13 to 15 below are explanatory diagrams illustrating the second evaluation experiment.

[0180] Figure 13 is the first explanatory diagram illustrating the second evaluation experiment in the modified example. Figure 13 shows inertial measurement devices D1 and D2. The arrangement of inertial measurement device D1 is an example of the arrangement of inertial measurement devices on the cab when acquiring motion data used for learning in learning mode. The arrangement of inertial measurement device D2 is an example of the arrangement of inertial measurement devices on the cab when acquiring motion data used in actual operation mode. The orientations of inertial measurement devices D1 and D2 differ by 135°.

[0181] Figure 14 is a second explanatory diagram illustrating the second evaluation experiment in the modified example. Figure 14 shows inertial measurement devices D3 and D4. The arrangement of inertial measurement device D3 is an example of the arrangement of inertial measurement devices on the side of the boom when acquiring motion data used for learning in learning mode. The arrangement of inertial measurement device D4 is an example of the arrangement of inertial measurement devices on the side of the boom when acquiring motion data used in actual operation mode. The orientations of inertial measurement devices D3 and D4 differ by 135°.

[0182] Figure 15 is a third explanatory diagram illustrating the second evaluation experiment in the modified example. Figure 15 shows inertial measurement devices D5 and D6. The arrangement of inertial measurement device D5 is an example of the arrangement of inertial measurement devices on the side of the arm when acquiring motion data used for learning in learning mode. The arrangement of inertial measurement device D6 is an example of the arrangement of inertial measurement devices on the side of the arm when acquiring motion data used in actual operation mode. Inertial measurement devices D5 and D6 are positioned differently. Specifically, inertial measurement device D6 was positioned 1 meter away from inertial measurement device D5.

[0183] Figure 16 shows an example of the experimental results of the second evaluation experiment in a modified example. More specifically, Figure 16 shows an example of the results when the arrangement of the inertial measurement device during motion data acquisition differs between the learning mode and the actual operation mode. In the second evaluation experiment, there were three loading times. The loading time is the time when the backhoe 300 is ready to load soil onto the dump truck. Therefore, the loading time is the timing to move the dump truck towards the backhoe 300, and is the timing to be predicted. Thus, three loading times mean three timings to be predicted. The results in Figure 16 show the performance of the timing prediction device 110 when predicting the loading time, expressed as a confidence level. Figure 16 shows that the confidence level was 1.0 for all datasets A to D.

[0184] Figure 16 shows that when basic feature data is used as feature data, the timing prediction device 110 can predict with high confidence even if there is a difference in the arrangement of the inertial measurement devices between the learning mode and the actual operation mode. This is because, as mentioned above, the feature data reflecting attitude has been reduced. Note that reduction means not using it. Therefore, Figure 16 shows that by reducing the feature data reflecting attitude, the robustness of the timing prediction device 110 to the difference in the arrangement of the inertial measurement devices between the learning mode and the actual operation mode has been increased.

[0185] The prediction unit 117 determines the primitive action q that characterizes the action. k (hereinafter referred to as "key action q") k The motion analysis unit 115 may perform the following key motion extraction process as a process to extract q from the sequence of primitive motions extracted by the motion analysis unit 115. In the key motion extraction process, q defined by the following equation (6) k The key action is q k It is extracted as follows.

[0186]

number

[0187] N L n represents the total number of operations in the training data before loading. L (q k ) is a key operation q before loading. k This represents the total number of occurrences. (i) t qk This is the i-th operation among the operations before loading, which is key operation q. k This indicates the time of appearance.

[0188] In the key action extraction process, the prediction unit 117 determines q that satisfies equation (6) based on the transition sequence of primitive actions. k The prediction unit 117 estimates q in this way. k This is obtained as a key action.

[0189] The prediction unit 117 executes a pattern acquisition process after the key action extraction process. The pattern acquisition process is a process that acquires the time average S (bold) of the probability of occurrence of transitions between key actions in order to identify transitions that satisfy the following constraints as the end timing of the preparation work action.

[0190] For a more detailed explanation of the pattern acquisition process, the maximum time length l between key operations is specified. qk qk+1 and the time average of the probability of occurrence of transitions between key actions (i) S qk qk+1 (t) Minimum value th qk qk+1This explains the following.

[0191] Minimum value th qk qk+1 and time average (i) S qk qk+1 (t) is defined by the following equations (7) to (10).

[0192]

number

[0193]

number

[0194]

number

[0195]

number

[0196] (i) F qk qk+1 (j, k) are times (i) t qk From time (i) t qk+1 This matrix shows whether or not a transition from an action with action identifier j to an action with action identifier k occurred during the period up to that point. Action identifiers are identifiers that distinguish key actions from each other. t0 The definition is the primitive action detected at time t0. t0+1 The definition is the primitive action detected at time (t0+1).

[0197] (i) n qk qk+1 (j, k) are times (i) t qk From time (i) t qk+1It is the total number of occurrences of the transition from the operation of operation identifier j to the operation of operation identifier k during the period up to

[0198] In the pattern acquisition process, the prediction unit 117, for the key operation q k from the key operation q k+1 even when there is an original operation not included in the model sandwiched between them, in order to handle it, l qk+1 qk is provided. To provide means to set the maximum value of the time obtained when transitioning to the key operation q k based on the key operation q k+1 as a threshold after obtaining the time of the transition. To set information means that information is recorded in a storage device such as the RAN1300, ROM1600, hard disk drive 1800, etc. The recording is performed, for example, by the CPU1200.

[0199] To correspond means that the pre-loading prediction functions. Note that to function means that the process is executed by the prediction unit 117. Next, in the pattern acquisition process, when the prediction unit 117 discovers a key operation in the motif within the same time as in the model, a process of determining that the same pre-loading operation is being performed is executed. Hereinafter, that section is defined as the period from time (i) t qk to time (i) t qk+1 up to. That section is the period during which the key operation in the motif discovered by the prediction unit 117 occurs.

[0200] Note that the definition of l qk+1 qk is the maximum time from detecting the key operation q k to detecting the key operation q('k + 1). The definition of the key operation q k+1 is, in the time series, the definition of the key operation that appears next to the key operation q k . Next, in the pattern acquisition process, the prediction unit 117, the threshold l of the time (hereinafter referred to as the "search time") for searching for these corresponding key operations qk+1 qkSet it. As described above, setting information means that the information is recorded in a storage device such as the RAN 1300, the ROM 1600, or the hard disk drive 1800. The recording is performed, for example, by the CPU 1200. Next, in the pattern acquisition process, the prediction unit 117 qk+1 qk searches for the corresponding key operation within that time (i.e., the search time) based on the threshold l

[0201] Next, the prediction unit 117 predicts the pre-loading operation in the pattern acquisition process. k from the key operation q k+1 to the key operation q (i) S qk qk+1 The time average qk qk+1 of the occurrence probability of the primitive operation between is greater than the minimum value th

[0202] qk+1 qk In the pattern acquisition process, when there are many transitions of the primitive operation where transitions that do not match within the model are observed exceeding the maximum time length l between key operations, the prediction unit 117 determines that the observed operation is not in the pre-loading work process. Having many transitions means that transitions occur so as to satisfy the condition that the time average S is greater than the minimum value th.

[0203] Conversely, even when the key operation q qk +l qk+1 qk k+1 is not detected by the HMM model based on the learning data after time t k+1 , if the transition of the primitive operation is close to that between key operations, the prediction unit 117 determines in the pattern acquisition process that the constraint condition applies to the observed operation as the pre-loading work process. Note that the transition of the primitive operation being close to that between key operations means that the degree to which the time average S of the occurrence probability of the primitive operation is greater than th is small. By using the constraint condition in the pattern acquisition process in this way, the timing prediction device 110 can also handle a pre-loading work process longer than the model.

[0204] In the pattern acquisition process, the prediction unit 117 acquires, as the pattern S (bold body), the transition that satisfies the constraint conditions in this way.

[0205] After executing the key operation extraction process and the pattern acquisition process, the prediction unit 117 executes a confidence level acquisition process. The confidence level acquisition process is a process of acquiring a confidence level based on the pattern S (bold body) acquired in the pattern acquisition process. The confidence level acquired in the confidence level acquisition process may be represented by the above-described formula (5), or may be represented by, for example, the following formula (11).

[0206] [Number]

[0207] n p (P1) represents the number of cases where the time when q k is detected is the standby or collection time. n p (P2) represents the number of cases where the time when q k is detected is immediately before the loading period of the prediction target. N p (P2) represents the total number of loading periods of the prediction target.

[0208] When the confidence level is equal to or greater than a predetermined value, the prediction unit 117 estimates that the current timing is the loading period. "Current" refers to the time when the prediction unit 117 determines that the confidence level is equal to or greater than a predetermined value. The predetermined value is, for example, 1 when the confidence level is defined by formula (5) or formula (11). Note that the prediction unit 117 executes the key operation extraction process, the pattern acquisition process, the confidence level acquisition process, and the process of estimating the loading timing in the learning mode and the actual operation mode.

[0209] Figure 17 is a flowchart illustrating an example of the processing flow performed in the pattern acquisition process, the confidence level acquisition process, and the loading timing estimation process in the modified example. For simplicity of explanation, Figure 17 illustrates the processing flow performed in the pattern acquisition process, the confidence level acquisition process, and the loading timing estimation process using the case where the confidence level is defined by equation (11) as an example.

[0210] The prediction unit 117 updates the time to t+1 (step S201). Next, the prediction unit 117 calculates primitive z t Obtain (step S202). primitive z t The definition is the primitive action estimated at time t. Next, the prediction unit 117 determines that k is 1 or greater and l qk t ga l q qk+1 Determine whether it is greater than or less (step S203).

[0211] k is 1 or greater and l qk t ga l qk qk+1 If it is greater than (step S203: YES), the prediction unit 117 will qk qk+1 (t) is th qk qk+1 Determine whether or not the above is true (step S204). qk qk+1 (t) is th qk qk+1 If the above is true (step S204: YES), the prediction unit 117 will determine z t ga q k+1 Determine whether it is equal to (step S205). t ga q k+1If it is equal to (Step S205: YES), the prediction unit 117 updates the value of the key operation identifier k to k+1 (Step S206). Next, the prediction unit 117 determines whether the confidence level is equal to 1 (Step S207). If the confidence level is equal to 1 (Step S207: YES), the prediction unit 117 estimates that the timing when the confidence level is 1 is the loading time (Step S208). After Step S208, the process returns to Step S201.

[0212] On the other hand, if k is 1 or greater, l qk t ga l qk qk+1 If the condition "greater than" is not met (step S203: NO), proceed to step S205. qk qk+1 (t) is th qk qk+1 If the above condition is not met (step S204: NO), the prediction unit 117 updates the value of the key operation identifier k to 0 (step S209). Also, z t ga q k+1 If it is not equal to (Step S205: NO), return to step S201. Also, if the confidence level is not equal to 1 (Step S207: NO), return to step S201.

[0213] The process shown in Figure 17 terminates when a predetermined termination condition is met. This predetermined termination condition is, for example, when the power to the timing prediction system 100 shown in Figure 17 is turned off.

[0214] An example of the experimental results from the third evaluation experiment conducted using the timing prediction device 110 is shown. The third evaluation experiment was an experiment to evaluate the accuracy of the predictions of the timing prediction device 110. In the third evaluation experiment, the prediction unit 117 performed the key operation extraction process, pattern acquisition process, confidence level acquisition process, and loading timing estimation process in the modified example. In the third evaluation experiment, three types of feature data from the basic feature data set were used as feature data. In addition, four different datasets were used in the third evaluation experiment.

[0215] Figure 18 shows an example of the experimental results of the third evaluation experiment in the modified example. Figure 18 shows the prediction results by the timing prediction device 110 in the modified example and the prediction results by three comparison methods. The three comparison methods were one threshold method, one LSTM (pattern recognition), and one LSTM (regression prediction). Note that for information on the prediction by the timing prediction device 110 in the modified example, the row labeled "Method" corresponds to the column labeled "Timing Prediction Device".

[0216] Figure 18 shows the accuracy, recall, and confidence level for each of the prediction results using these four methods. The confidence level used in the experiment was defined by equation (11). Figure 18 shows that the timing prediction device 110 in the modified example can predict with more than twice the accuracy of the other methods. Figure 18 shows that the recall of the timing prediction device 110 in the modified example is about the same as the other methods. Figure 18 shows that the confidence level of the timing prediction device 110 in the modified example is more than twice as high as the other methods. Note that the values ​​in Figure 18 are the average values ​​obtained from the prediction results for the four datasets used in the third evaluation experiment.

[0217] Thus, the accuracy and confidence of the predictions made by the modified timing prediction device 110 are higher than those of other methods, and the reproducibility is comparable to that of other methods. Therefore, the modified timing prediction device 110 configured in this way can make predictions with high accuracy.

[0218] Although the present invention has been described above using embodiments, the technical scope of the present invention is not limited to the scope described in the above embodiments. It will be apparent to those skilled in the art that various modifications or improvements can be made to the above embodiments. It will be clear from the claims that such modified or improved forms may also be included in the technical scope of the present invention.

[0219] It should be noted that the execution order of operations, procedures, steps, and stages in the apparatus, system, program, and method described in the claims, specification, and drawings is not explicitly stated as "before" or "prior to," and that the processes can be implemented in any order unless a previous process is used in a later process. Even if the operation flow in the claims, specification, and drawings is described using phrases such as "first," "next," etc., for convenience, this does not mean that it is essential to perform the operations in that order. [Explanation of symbols]

[0220] 100 Timing prediction system, 110 Timing prediction device, 111 Switch signal receiving unit, 112 Timing specification unit, 113 Motion data acquisition unit, 114 Feature generation unit, 115 Motion analysis unit, 116 Motion model storage unit, 117 Prediction unit, 118 Output unit, 140 Motion measurement device, 140A Motion measurement device, 140B Motion measurement device, 140C Motion measurement device, 141 Measurement unit, 142 Control unit, 143 Recording unit, 150 Foot switch, 300 Backhoe, 310 Main body, 311 Lower traveling body, 312 Upper slewing body, 312A Slewing frame, 312B Cab, 320 Backhoe attachment, 321 Boom, 322 Arm, 323 Bucket link, 324 Bucket, 325 Bucket cylinder, 1001 Control unit

Claims

1. We acquire motion data for multiple points on an observed object performing a predetermined motion. Feature data is generated by performing frequency analysis on the acquired motion data. The generated feature data is segmented using a hidden Markov model to extract a sequence of primitive behavioral transitions, which are the extracted states. Based on the extracted transition sequence and a motion model obtained by learning a sequence of primitive motion transitions obtained by segmenting the motion state of a training object performing a motion, the frequency of occurrence of primitive motions in the predetermined motion is analyzed and a pattern of primitive motions is extracted to predict the timing of the end of the predetermined motion. The observation object and the teaching object are shovels. The generated feature data is obtained from motion data showing the angular velocity of the movement, which has a component in the rotational movement around the joint axes of the shovel's arm, boom, and cab, resulting from the motion. Timing prediction method.

2. The predetermined motion performed by the observed object is divided into multiple motions, and for each divided motion, the frequency of occurrence of primitive motions in the motion is analyzed based on the extracted transition sequence and the motion model to extract a pattern of primitive motions, the extracted pattern is evaluated step by step to calculate a degree of confidence, and the timing of the end of the predetermined motion is predicted according to the calculated degree of confidence. The timing prediction method according to claim 1.

3. By performing frequency analysis on the acquired motion data, a frequency band containing frequency components resulting from the predetermined motion is selected, and feature data is generated in the selected frequency band. The timing prediction method according to claim 1 or 2.

4. A hidden Markov model is applied to the aforementioned behavioral model, and the feature data is segmented and a sequence of primitive behavioral transitions is extracted by computational processing using the hidden Markov model. A timing prediction method according to any one of claims 1 to 3.

5. The motion data includes angular velocity data and acceleration data. A timing prediction method according to any one of claims 1 to 4.

6. A motion model storage unit stores a motion model obtained by learning a sequence of primitive motion transitions obtained by segmenting the motion state of a training object that performs motion movements, A motion data acquisition unit that acquires motion data for multiple locations on an observed object performing a predetermined motion, A feature generation unit generates feature data by performing frequency analysis on the motion data acquired by the motion data acquisition unit, A motion analysis unit extracts a sequence of primitive motion transitions, which are states extracted by segmenting the feature data generated by the feature generation unit using a hidden Markov model. A prediction unit predicts the timing of the end of the predetermined movement by analyzing the frequency of occurrence of primitive movements in the predetermined movement and extracting patterns of primitive movements based on the transition sequence extracted by the motion analysis unit and the motion model stored in the motion model storage unit, Equipped with, The observation object and the teaching object are shovels. The generated feature data is obtained from motion data showing the angular velocity of the movement, which has a component in the rotational movement around the joint axes of the shovel's arm, boom, and cab, resulting from the motion. Timing prediction device.

7. A timing prediction system comprising a motion measurement device that measures motion data at multiple locations on an observed object performing a predetermined motion, and a timing prediction device, The timing prediction device is, A motion model storage unit stores a motion model obtained by learning a sequence of primitive motion transitions obtained by segmenting the motion state of a training object that performs motion movements, A motion data acquisition unit that acquires motion data from the motion measurement device, A feature generation unit generates feature data by performing frequency analysis on the motion data acquired by the motion data acquisition unit, A motion analysis unit extracts a sequence of primitive motion transitions, which are states extracted by segmenting the feature data generated by the feature generation unit using a hidden Markov model. A prediction unit predicts the timing of the end of the predetermined movement by analyzing the frequency of occurrence of primitive movements in the predetermined movement and extracting patterns of primitive movements based on the transition sequence extracted by the motion analysis unit and the motion model stored in the motion model storage unit, Equipped with, The observation object and the teaching object are shovels. The generated feature data is obtained from motion data showing the angular velocity of the movement, which has a component in the rotational movement around the joint axes of the shovel's arm, boom, and cab, resulting from the motion. Timing prediction system.

8. A computer equipped with a motion model storage unit that stores a motion model obtained by segmenting the motion state of a training object performing a motion and learning the sequence of primitive motion transitions obtained therefrom, We acquire motion data for multiple points on an observed object performing a predetermined motion. Feature data is generated by performing frequency analysis on the acquired motion data. The generated feature data is segmented to extract the sequence of primitive operation transitions, Based on the extracted transition sequence and the motion model stored in the motion model storage unit, the frequency of occurrence of primitive movements in the predetermined motion is analyzed and the pattern of primitive movements is extracted to predict the termination timing of the predetermined motion. The aforementioned primitive operation is a state in which the feature data is segmented and extracted by a hidden Markov model, The observation object and the teaching object are shovels. The generated feature data is obtained from motion data showing the angular velocity of the movement, which has a component in the rotational movement around the joint axes of the shovel's arm, boom, and cab, resulting from the motion. A program for executing a process.

9. A construction machine system comprising a motion measurement device for measuring motion data at multiple locations on a construction machine performing a predetermined motion, and a timing prediction device, The timing prediction device is, A motion model storage unit stores a motion model obtained by learning a sequence of primitive motion transitions obtained by segmenting the motion state of a training construction machine that performs motion movements, A motion data acquisition unit that acquires motion data from the motion measurement device, A feature generation unit generates feature data by performing frequency analysis on the motion data acquired by the motion data acquisition unit, The motion analysis unit segments the feature data generated by the feature generation unit and extracts a sequence of transitions of primitive movements. A prediction unit predicts the timing of the end of the predetermined movement by analyzing the frequency of occurrence of primitive movements in the predetermined movement and extracting patterns of primitive movements based on the transition sequence extracted by the motion analysis unit and the motion model stored in the motion model storage unit, Equipped with, The aforementioned primitive operation is a state in which the feature data is segmented and extracted by a hidden Markov model, The aforementioned construction machine and the aforementioned teaching construction machine are shovels. The generated feature data is obtained from motion data showing the angular velocity of the movement, which has a component in the rotational movement around the joint axes of the shovel's arm, boom, and cab, resulting from the motion. Construction machinery systems.

Images