Control devices for industrial vehicles, control systems for industrial vehicles, and programs for industrial vehicle control devices.

Abstract

To provide an industrial vehicle control device, an industrial vehicle control system and a program for the industrial vehicle control device capable of estimating a work state of an industrial vehicle with improved appropriateness.SOLUTION: A control device 20 has a work state estimation section 30 which estimates a work state of a forklift 1. The work state estimation section 30 receives input of not only operation information related to an operation state of the forklift 1 but also photographic information obtained by photographing a cargo. The work state estimation section 30 is capable of determining not only whether operation of the forklift 1 is normal or abnormal but also whether a packing style of the cargo is normal or abnormal on the basis of the photographic information. Thus, the work state estimation section 30 determines whether or not the packing style is in an abnormal state and outputs a work state. Thereby, the work state estimation section 30 can estimate the work state including not only the state of the forklift 1 itself but also the abnormal state of the packing style.SELECTED DRAWING: Figure 4

Description

Technical Field

[0001] The present invention relates to a control device for an industrial vehicle, a control system for an industrial vehicle, and a program for a control device for an industrial vehicle.

Background Art

[0002] As a control device for an industrial vehicle, for example, one described in Patent Document 1 is known. The control device for an industrial vehicle described in Patent Document 1 estimates the working state of the industrial vehicle based on operation information for the industrial vehicle, and controls the industrial vehicle based on the estimation result of the working state.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of Invention

Problems to be Solved by the Invention

[0004] Here, the cargo that is the handling target of the industrial vehicle may be in an abnormal state where the cargo form is not suitable for cargo handling. For example, if the cargo is not normally loaded on a pallet, the cargo may not be handled appropriately. However, the above-mentioned control device estimates the working state of the industrial vehicle based on operation information, and thus cannot detect up to the cargo form of the cargo. Accordingly, the above-mentioned control device has a problem that it cannot estimate the working state including detection of an abnormality in the cargo form. Therefore, it has been required to estimate the working state more appropriately, including up to the abnormal state of the cargo form.

[0005] An object of the present invention is to provide a control device for an industrial vehicle, a control system for an industrial vehicle, and a program for a control device for an industrial vehicle, which are capable of more appropriately estimating the working state of the industrial vehicle.

Means for Solving the Problems

[0006] A control device for an industrial vehicle according to one aspect of the present invention is a control device for estimating the working state of an industrial vehicle, and includes a working state estimation unit for estimating the working state of the industrial vehicle. The working state estimation unit receives input of operation information relating to the operating state of the industrial vehicle and photographic information of the cargo, determines whether the cargo is in an abnormal state or not, and outputs the working state.

[0007] The control device for industrial vehicles includes a work state estimation unit that estimates the working state of the industrial vehicle. The work state estimation unit receives not only operational information regarding the operating state of the industrial vehicle, but also photographic information of the cargo. Based on the imaging information, the work state estimation unit can determine whether the cargo is in a normal or abnormal state, in addition to the operation of the industrial vehicle. Therefore, the work state estimation unit determines whether the cargo is in an abnormal state and outputs the working state. This allows the work state estimation unit to estimate the working state including not only the state of the industrial vehicle itself, but also any abnormalities in the cargo. As a result, the working state of the industrial vehicle can be estimated more appropriately.

[0008] The work state estimation unit can estimate the work state based on a work state estimation model set up by machine learning, and may output the work state based on operation information and photographic information. Since the work state estimation model is set up by machine learning based on actual past data, it is possible to estimate the work state with high accuracy, including abnormal conditions of the packaging.

[0009] The operation information may include at least one of the following: accelerator input, tire angle, lift input, reach input, and tilt input. These parameters reflect the operator's intentions in industrial vehicles. The work state estimation unit can use these parameters as operation information to estimate the appropriate work state.

[0010] The control device for industrial vehicles further includes a feature vector acquisition unit that acquires feature vectors from images of cargo using machine learning. The feature vector acquisition unit may use a CNN as its machine learning model, and the work state estimation unit may estimate the work state based on the feature vectors acquired by the feature vector acquisition unit. By representing the captured information as feature vectors, the feature vector acquisition unit can reduce the amount of information output to the work state estimation unit. Furthermore, by using a CNN as its machine learning model, the feature vector acquisition unit can acquire feature vectors that suitably reflect abnormal cargo conditions based on past performance. [Effects of the Invention]

[0011] According to the present invention, it is possible to provide an industrial vehicle control device, an industrial vehicle control system, and a program for an industrial vehicle control device that can more appropriately estimate the working state of an industrial vehicle. [Brief explanation of the drawing]

[0012] [Figure 1] This is a block diagram showing a driver assistance system according to an embodiment of the present invention. [Figure 2] (a) is a perspective view showing how the camera unit is attached to the forklift, and (b) is a schematic diagram illustrating the angle of the camera unit. [Figure 3] This is a diagram illustrating the working state of a forklift. [Figure 4] This is a block diagram of the work status acquisition unit. [Figure 5] This diagram shows what happens when a package is in an abnormal condition. [Figure 6] This figure shows the machine learning process for the work state estimation unit and the feature vector acquisition unit. [Figure 7] This is a flowchart showing the processing steps of the driver assistance system. [Modes for carrying out the invention]

[0013] Preferred embodiments of the present invention will be described in detail below with reference to the drawings. In the drawings, the same or equivalent elements are denoted by the same reference numerals, and redundant descriptions are omitted.

[0014] Figure 1 is a block diagram showing a driver assistance system 100 (industrial vehicle control system) including a control device 20 according to an embodiment of the present invention. The driver assistance system 100 is a system for remotely controlling an industrial vehicle. As shown in Figure 1, the driver assistance system 100 comprises a forklift 1 (industrial vehicle) and a remote control device 2.

[0015] The forklift 1 comprises a driving control unit 11 and a plurality of imaging units 12. The driving control unit 11 receives command signals from the remote control device 2 and performs driving control and steering control based on the command signals. The plurality of imaging units 12 are installed in various parts of the forklift 1 and capture images of the surrounding environment of the forklift 1. The imaging units 12 acquire the captured images as support information used for work assistance and transmit them to the display control unit 22, which will be described later. An example of the mounting positions of the plurality of imaging units 12 is shown in Figure 2(a). The imaging units 12 are installed at the front end of the forklift 1's body, the ends in the width direction, the ceiling, etc. Note that when XYZ coordinates are set as shown in Figure 2(b), the imaging units 12 at each location may be installed in a tilted state around the X, Y, and Z axes. In Figure 2(a), eight imaging units 12 are installed, each acquiring images of a different location.

[0016] As shown in Figure 1, the remote control device 2 comprises a storage unit 15, an operation unit 16, a display unit 17, an operation information acquisition unit 18, and a control device 20. The storage unit 15 is a device for storing various types of information. The operation unit 16 is a user interface for an operator to input operations to remotely control the forklift 1. The display unit 17 is a user interface for outputting video. The display unit 17 has a first area D1, a second area D2, and a third area D3 that can output different video from each other.

[0017] The operation information acquisition unit 18 acquires operation information when the operator is operating the operation target (here, the operation unit 16). The operation information acquisition unit 18 is configured by, for example, a sensor provided on the operation lever of the operation unit 16, or a means for detecting the operation content based on a signal indicating the operation content of the operation unit 16, and the like.

[0018] The control device 20 is a control unit that controls the entire remote operation device 2. The control device 20 includes an ECU [Electronic Control Unit] that comprehensively manages the remote operation device 2. The ECU is an electronic control unit having a CPU [Central Processing Unit], a ROM [Read Only Memory], a RAM [Random Access Memory], a CAN [Controller Area Network], a communication circuit, and the like. In the ECU, for example, various functions are realized by loading a program stored in the ROM into the RAM, and executing the program loaded into the RAM by the CPU. The control device 20 includes a travel command unit 21, a display control unit 22, a work state acquisition unit 23, and a viewpoint information acquisition unit 26.

[0019] The travel command unit 21 is a unit that generates a command signal based on the operation input by the operation unit 16 and transmits the command signal to the travel control unit 11.

[0020] The display control unit 22 is a unit that controls the display content of the display unit 17. The display control unit 22 causes the display unit 17 to display information for supporting the operator's remote operation work. The display control unit 22 controls the display content of the first area D1, the second area D2, and the third area D3 of the display unit 17 using the information in the database of the storage unit 15. In addition, the display control unit 22 transmits the video acquired from the imaging unit 12 to the work state acquisition unit 23.

[0021] The display control unit 22 selects the image to be displayed on the display unit 17 based on the viewpoint information acquired by the viewpoint information acquisition unit 26. The first area D1 and the second area D2 are large screen areas that display a specific image in a larger size, and the image from the shooting unit 12 selected by the display control unit 22 from among the images from multiple shooting units 12 is displayed. The third area D3 corresponds to a small screen area that displays multiple images in a smaller size, and displays images from all shooting units 12 on the forklift 1 as environmental information for checking the surroundings (see also Figure 4).

[0022] For example, Figure 3(a) is a table defining the working states of forklift 1. Here, as an example of working states, seven states related to the movement of forklift 1 and the loading / unloading of cargo—"Stop," "Move fwd," "Approach," "Adjust heading," "Load / Unload," "Retreat," and "Move reverse"—are divided into states with and without cargo, resulting in a total of 14 working states. The movement of forklift 1 corresponds to "Stop," "Move fwd," "Approach," "Retreat," and "Move reverse," while the loading / unloading of cargo corresponds to "Adjust heading" and "Load / Unload." The definitions and number of each working state are not limited to those shown above and can be changed as appropriate. Furthermore, the work state may be defined solely in terms of the movement of the forklift 1 (industrial vehicle). In addition, a work state called "Anomaly" is set to indicate that the condition of the cargo is abnormal, regardless of whether there is cargo or not. As a result, a total of 15 work states are defined in Figure 3(a). Figure 3(b) is a schematic diagram showing each work state when the forklift 1 takes cargo out of and unloads cargo from the shelf SF. Note that the work state 15, which is an abnormal condition of the cargo, can be reached from any of the work states 1 to 14 in the figure. In the database of the memory unit 15, each work state is linked to the camera unit 12, which can display images of areas that should be checked in that work state. Therefore, the display control unit 22 can display images of areas that should be checked in particular on a large screen in the first area D1 and the second area D2, according to the work state of the forklift 1 being operated by the operator. Note that the images displayed on the display unit 17 by the control of the display control unit 22 for driving assistance may be referred to as "driving assistance images".

[0023] There are individual differences among operators regarding the appropriate timing for switching the driver assistance video. Also, when the work state performed by forklift 1 changes, the video to be displayed on the display unit 17 changes. In light of these circumstances, the work state acquisition unit 23 and the viewpoint information acquisition unit 26 adapt the driver assistance video to the individual operator and perform processing corresponding to the state transition of the work state.

[0024] The work status acquisition unit 23 acquires the work status being performed by the forklift 1. The work status acquisition unit 23 acquires the work status based on the operation information acquired by the operation information acquisition unit 18. The work status acquisition unit 23 acquires the work status using a work status estimation model in which estimation indicators such as transition conditions and transition thresholds for transitioning to each work status are set. The work status acquisition unit 23 may acquire the work status estimation model stored in the storage unit 15. Details of the work status acquisition unit 23 will be described later. The work status acquisition unit 23 transmits the acquired work status to the viewpoint information acquisition unit 26.

[0025] The viewpoint information acquisition unit 26 acquires the operator's viewpoint based on the work status acquired by the work status acquisition unit 23. Here, the storage unit 15 has a database in which the work status of the forklift 1 and viewpoint information based on the operator's gaze are linked. Therefore, the viewpoint information acquisition unit 26 acquires viewpoint information corresponding to the work status by comparing the work status of the forklift 1 with the database. The viewpoint information acquisition unit 26 transmits the acquired viewpoint information to the display control unit 22. As a result, the display control unit 22 controls the image displayed on the display unit 17 based on the viewpoint information, and can display the optimal image for the work status of the forklift 1 and the individual operator.

[0026] Next, with reference to Figure 4, the detailed configuration of the work state acquisition unit 23 will be described. Figure 4 is a block diagram showing the configuration of the work state acquisition unit 23. As shown in Figure 4, the work state acquisition unit 23 comprises a work state estimation unit 30 and a feature vector acquisition unit 31.

[0027] The work state estimation unit 30 estimates the work state of the forklift 1 using machine learning. The work state estimation unit 30 receives current operation information u(t) and current feature vector v(t), which is current image information, and, based on the learning results, determines whether the load is in an abnormal state or not, and outputs the current work state x(t). The work state estimation unit 30 has a machine learning model M1.

[0028] The classifications "1" to "15" shown in Figure 3(a) are used to indicate the working state. Operation information includes at least one of the following: accelerator operation amount, tire angle, lift operation amount, reach operation amount, and tilt operation amount. Accelerator operation amount is a parameter that indicates the amount of accelerator operation of forklift 1, and a larger accelerator operation amount indicates that forklift 1 is moving at a higher speed. Tire angle is a parameter that indicates the turning angle of the tires of forklift 1, and a larger tire angle indicates that forklift 1 turns more sharply. Lift operation amount is a parameter that indicates the amount of operation to lift the forks of forklift 1, and a larger lift operation amount indicates that the forks are lifted more sharply. Reach operation amount is a parameter that indicates the amount of operation to extend the forks of forklift 1 forward, and a larger reach operation amount means that the forks are moved further forward. Tilt operation amount is a parameter that indicates the amount of operation to tilt the forks of forklift 1, and a larger tilt operation amount means that the forks are tilted more sharply.

[0029] The work state estimation unit 30 estimates and outputs the current work state x(t) by comparing it with the input current operation information u(t) and an operation state estimation model that has been set with estimation indicators such as transition conditions and transition thresholds for transitioning to each work state. The work state estimation model has been set which parameters of the above operation information need to be increased (decreased) and by how much for a transition to one work state to occur. For example, as shown in Figure 3(b), in "Work State 4" "Adjust heading", the forklift 1 is in a state where it is making a large turn, and in "Work State 5" "Load", the forklift 1 is moving slowly. Therefore, the work state estimation unit 30 can estimate that a transition from "Work State 4" to "Work State 5" has occurred when the accelerator operation amount is large and the tire angle is large, and each parameter falls below a predetermined threshold. In such a work state estimation model, machine learning is used to set the optimal estimation indicators for the operator based on the operator's past driving performance. Furthermore, an individual work state estimation model may be created for each operator, or a work state estimation model that is not limited to a specific operator may be created. The work state estimation models are stored in the memory unit 15, and the work state estimation unit 30 retrieves the appropriate work state estimation model corresponding to the operator from the memory unit 15 at the necessary timing. In addition, the work state estimation unit 30 can estimate and output the current work state x(t), including the abnormal state of the package's packaging (work state 15), by considering the feature vector v(t) input from the feature vector acquisition unit 31. The abnormal state of the package's packaging will be described later.

[0030] The feature vector acquisition unit 31 acquires image feature vectors from images of the cargo using machine learning. The feature vector acquisition unit 31 acquires the current image I(t) captured by the forklift 1's camera unit 12 via the display control unit 22 (see Figure 1). From among the multiple images from the camera unit 12, the feature vector acquisition unit 31 acquires the image I(t) that shows the cargo, such as the front of the forklift 1. The feature vector acquisition unit 31 receives the current image I(t) as input and, based on the learning results, outputs the current image feature vector v(t) to the work state estimation unit 30. The feature vector acquisition unit 31 has a machine learning model M2.

[0031] The feature vector acquisition unit 31 performs image analysis by comparing an extraction model that can extract feature vectors (e.g., 1024 dimensions) effective for estimating the working state from an image (e.g., 1920 × 1080 dimensions) with the input current image I(t), thereby acquiring and outputting the feature vector v(t) of the current image I(t). Such an extraction model is configured using machine learning, with the optimal extraction method set based on past operating performance. The extraction model is stored in the memory unit 15, and the feature vector acquisition unit 31 retrieves the extraction model from the memory unit 15 at the necessary timing.

[0032] Referring to Figure 5, we will describe feature vectors suitable for estimating that the packaging of the cargo is in an abnormal state. As shown in Figure 5, cargo 45F is arranged so as to span across the top surface of cargo 45A on pallet 40A and the top surface of cargo 45B on pallet 40B. In this type of packaging, when forklift 1 lifts pallet 40A or pallet 40B, cargo 45F may collapse. Therefore, as shown in region E1, it is preferable to extract feature vectors for the areas showing the degree of overlap of multiple cargoes 45. Also, cargo 45D on pallet 40D is touching pallet 40C. In this type of packaging, when forklift 1 lifts pallet 40C, cargo 45D may collapse. Alternatively, when pallet 40D is lifted, cargo 45D may fall. Therefore, as shown in region E2, it is preferable to extract feature vectors for the areas showing the degree of overlap between cargo 45 on one pallet and another pallet 40. Also, cargo 45E on pallet 40E is protruding from pallet 40E. In this type of packaging, there is a possibility that the cargo 45E may fall when the forklift 1 lifts the pallet 40E. Also, if another pallet 40 is to be lowered next to pallet 40E, that pallet 40 may collide with the cargo 45E. Therefore, it is preferable to extract feature vectors of the portion of the cargo 45 that protrudes from the pallet 40, as shown in region E3.

[0033] Note that Figure 5 merely shows one example of an abnormal packaging condition, and feature vectors can be extracted to detect other abnormal conditions. For example, if the cargo being transported by forklift 1 is positioned significantly misaligned with the pallet or protruding from the pallet, a feature vector can be extracted as an abnormal packaging condition. Furthermore, the object on which the cargo is placed is not limited to pallets as shown in Figure 5. For example, abnormal packaging conditions can be extracted when using a cage pallet, such as a box-shaped cage with four legs. For example, the normal state is when the four legs of the upper cage pallet are fitted into the four fittings on the upper end of the lower cage pallet. In contrast, if any of the legs (for example, one of the four) are not fitted into the fitting (fallen state), or if the fitting is insufficient due to distortion of the cage pallet, a feature vector can be extracted as an abnormal packaging condition.

[0034] The work state estimation unit 30 can estimate the work state based on the feature vector v(t) acquired by the feature vector acquisition unit 31. Regardless of which of the "work state 1" to "work state 14" (see Figure 3(a)) the forklift 1 is currently in, the work state estimation unit 30 may determine the abnormal state of the load based solely on the result of the input feature vector v(t). For example, if the input feature vector v(t) contains regions like E1, E2, and E3 in Figure 5, the work state estimation unit 30 may estimate that the load is in an abnormal state of "work state 15," regardless of which of the "work state 1" to "work state 14" the forklift 1 is currently in. In this case, the feature vector acquisition unit 31 may acquire the region like E1, E2, and E3 from the image as an abnormal state of the load and output a determination result indicating that "the load is in an abnormal state" to the work state estimation unit 30 as imaging information. The work state estimation unit 30 may determine that the packaging is in an abnormal state based on the imaging information showing the determination result from the feature vector acquisition unit 31.

[0035] Furthermore, the work state estimation unit 30 may determine if the load configuration is abnormal by considering both which of the "work state 1" to "work state 14" (see Figure 3(a)) the forklift 1 is currently in and the input feature vector v(t). For example, even if the input feature vector v(t) contains regions like E1, E2, and E3 in Figure 5, the work state estimation unit 30 may not determine that the load configuration is abnormal, depending on which of the "work state 1" to "work state 14" the forklift 1 is in. For example, even if regions like E1, E2, and E3 exist in the image, if the forklift 1 is attempting to lift a pallet 40 different from pallets 40A to 40E, it may not be determined that the load configuration is abnormal. Also, regarding pallet 40E, it may be possible to allow the forklift 1 to transport the load 45E even if it is slightly misaligned, but not to allow a collision between the load 45E and other pallets 40. In this case, if the work state estimation unit 30 is in "work state 5" where the forklift 1 is lifting pallet 40E, even if region E3 is extracted, it does not determine that the cargo configuration is abnormal and estimates that it is in "work state 5". On the other hand, if the work state estimation unit 30 is in "work state 12" where the forklift 1 is stacking another pallet next to pallet 40E, and region E3 is extracted, it determines that the cargo configuration is abnormal and estimates that it is in "work state 15".

[0036] Figure 6(a) is a block diagram showing the operation state estimation unit 30 during learning. As shown in Figure 6(a), the operation state estimation unit 30 performs machine learning by inputting operation information related to the operation state of the forklift 1, image feature vectors, and ground truth data of the operation state. The machine learning process shown in Figure 6(a) may be performed in advance, before the actual remote operation of the forklift 1 takes place. Pre-prepared data is used as operation information, feature vectors, and ground truth data. For example, the forklift 1 is actually operated in a laboratory, and data is acquired by linking operation information and image feature vectors at a certain time with the actual operation state at that time. The actual operation state at this time becomes the ground truth data.

[0037] The work state estimation unit 30 may use LSTM (Long Short Term Memory) as the machine learning model M1. In this case, the work state estimation unit 30 may perform machine learning using operation information acquired time-series by time and feature vectors of images acquired time-series by time as training data. The operation information and feature vectors acquired time-series by time are, for example, composed of a data set of "operation information u(t), feature vector v(t), and correct work state y(t)" acquired at predetermined intervals from "time 0" to "time T". The work state estimation unit 30 may also use RNN (Recurrent Neural Network) as the machine learning model M1, which can handle operation information acquired time-series by time. In addition, the machine learning model M1 is not limited to CNN as long as the work state estimation unit 30 can acquire it.

[0038] Figure 6(b) is a block diagram showing the state of the feature vector acquisition unit 31 during training. As shown in Figure 6(b), the feature vector acquisition unit 31 outputs a feature vector to the work state estimation unit 30 based on the input of an image of the cargo to be handled by the forklift 1 and the input of the correct work state corresponding to the image. The work state estimation unit 30 then outputs an estimated work state based on the feature vector. The feature vector acquisition unit 31 constructs machine learning models M2 and M3 so as to minimize the error between the correct work state (y(t)) and the estimated work state (y'(t)). Note that the machine learning model M3 of the work state estimation unit 30 is a model used only during training. Once training is complete, the feature vector acquisition unit 31 retrieves the machine learning model M2 and uses it in the work state acquisition unit 23. The machine learning process shown in Figure 6(b) may be performed in advance before the actual remote operation of the forklift 1 takes place. Pre-prepared data is used as the image and correct data. For example, prepare an image of a package that can be identified as being in an abnormal state, and extract feature vectors from areas within that image that clearly show the characteristics of the abnormal state (for example, regions E1, E2, E3 in Figure 5). These extracted feature vectors are then used as the ground truth feature vectors.

[0039] The feature vector acquisition unit 31 may use a CNN (Convolutional Neural Network) as the machine learning model M2. Here, the feature vector acquisition unit 31 may perform machine learning using images acquired in time series as training data, or it may perform machine learning using images that are not in time series. For example, training may be performed by preparing a large number of images that clearly show the characteristics of abnormal and normal packaging. For example, the training data consists of a data set of multiple "image I(i) and ground truth feature vector z(i)" from "pattern 0" to "pattern N". In addition, the machine learning model M2 is not limited to a CNN as long as the feature vector acquisition unit 31 can acquire it.

[0040] Next, with reference to Figure 7, an example of the processing content showing the driving support method by the control device 20 will be described. The processing shown in Figure 7 is performed when a driving support video is displayed on the display unit 17 and the operator is remotely operating the forklift 1. As shown in Figure 7, the work state acquisition unit 23 acquires operation information from the operator via the operation information acquisition unit 18 and acquires an image of the load via the display control unit 22 (step S10). Next, the feature vector acquisition unit acquires a feature vector based on the image acquired in step S10 (step S20). Next, the work state estimation unit 30 estimates the work state based on the operation information acquired in step S10 and the feature vector acquired in step S20 (step S30). Next, the viewpoint information acquisition unit 26 acquires viewpoint information based on the work state estimated in step S30 (step S40). Next, the display control unit 22 selects the driving support video to be displayed on the display unit 17 based on the viewpoint information acquired in step S40 (step S50).

[0041] Next, the operation and effects of the control device 20 according to this embodiment will be described.

[0042] The control device 20 has a work state estimation unit 30 that estimates the working state of the forklift 1. Here, the work state estimation unit 30 receives not only operational information regarding the operating state of the forklift 1, but also photographic information of the cargo. The work state estimation unit 30 can determine whether the cargo is in a normal or abnormal state based not only on the operation of the forklift 1 but also on the image information. Accordingly, the work state estimation unit 30 determines whether the cargo is in an abnormal state and outputs the work state. As a result, the work state estimation unit 30 can estimate the work state including not only the state of the forklift 1 itself but also the abnormal state of the cargo.

[0043] The work state estimation unit 30 can estimate the work state based on a work state estimation model set by machine learning, and may output the work state based on operation information and photographic information. Since the work state estimation model is set by machine learning based on actual past data, it is possible to estimate the work state with high accuracy, including abnormal conditions of the packaging.

[0044] The operation information may include at least one of the following: accelerator operation amount, tire angle, lift operation amount, reach operation amount, and tilt operation amount. These parameters reflect the operator's intentions in the forklift 1. The work state estimation unit 30 can estimate the appropriate work state by using these parameters as operation information.

[0045] The control device 20 further includes a feature vector acquisition unit 31 that acquires feature vectors from images of the cargo using machine learning. The feature vector acquisition unit 31 uses a CNN as its machine learning model, and the work state estimation unit 30 may estimate the work state based on the feature vectors acquired by the feature vector acquisition unit 31. By representing the captured information as feature vectors, the feature vector acquisition unit 31 can reduce the amount of information output to the work state estimation unit 30. Furthermore, by using a CNN as its machine learning model, the feature vector acquisition unit 31 can acquire feature vectors that suitably reflect abnormal conditions of the cargo based on past performance.

[0046] Although several preferred embodiments of the present invention have been described above, the present invention is not limited to the above embodiments.

[0047] In the above-described embodiment, the driving support system assisted the operator's work during remote operation, but it may also provide assistance when the industrial vehicle is operated by a human. Furthermore, it may provide assistance when the operator is performing a simulated operation of the industrial vehicle.

[0048] Industrial vehicles are not limited to forklifts; towing tractors, skid steer loaders, and other types of vehicles may also be used.

[0049] In the above-described embodiment, the work state estimation unit 30 and the feature vector acquisition unit 31 performed processing using machine learning, but it is not necessarily required to use machine learning.

[0050] [Form 1] A control device for estimating the working state of an industrial vehicle, It has a work state estimation unit that estimates the work state of the industrial vehicle, The aforementioned work state estimation unit is a control device for an industrial vehicle that receives operational information regarding the operating state of the industrial vehicle and photographic information of the cargo, determines whether the cargo is in an abnormal state, and outputs the work state. [Form 2] The control device for an industrial vehicle according to Embodiment 1, wherein the work state estimation unit is capable of estimating the work state based on a work state estimation model set by machine learning, and outputs the work state based on the operation information and the shooting information. [Form 3] The control device for an industrial vehicle according to Embodiment 1 or 2, wherein the operation information includes at least one of the accelerator operation amount, tire angle, lift operation amount, reach operation amount, and tilt operation amount. [Form 4] The system further includes a feature vector acquisition unit that uses machine learning to acquire feature vectors from images of the aforementioned luggage, The feature vector acquisition unit uses a CNN as the machine learning model, The control device for an industrial vehicle according to any one of embodiments 1 to 3, wherein the work state estimation unit estimates the work state based on the feature vector acquired by the feature vector acquisition unit. [Form 5] A control system for an industrial vehicle having a control device for an industrial vehicle as described in any one of the four forms. [Form 6] A program for an industrial vehicle control device used in any one of the industrial vehicle control devices described in any one of Forms 1 to 4. [Explanation of Symbols]

[0051] 1...Forklift, 20...Control device, 30...Working state estimation unit, 31...Feature vector acquisition unit, 100...Driving support system (industrial vehicle control system).

Claims

1. A control device for estimating the working state of an industrial vehicle, A work state estimation unit for estimating the work state of the industrial vehicle, It includes a feature vector acquisition unit that uses machine learning to acquire feature vectors from images of luggage, The feature vector acquisition unit uses a CNN as the machine learning model, The control device for an industrial vehicle includes an industrial vehicle control unit which receives operation information relating to the operating state of the industrial vehicle and the feature vector acquired by the feature vector acquisition unit as input, estimates the industrial state based on an industrial state estimation model set by machine learning and the operation information, and determines an abnormal state of the cargo's packaging by considering both the estimated industrial state and the feature vector, thereby differently determining the abnormal state of the cargo's packaging based on the feature vector according to the estimated industrial state, and outputs an industrial state including the abnormal state of the cargo's packaging.

2. The control device for an industrial vehicle according to claim 1, wherein the operation information includes at least one of the accelerator operation amount, tire angle, lift operation amount, reach operation amount, and tilt operation amount.

3. A control system for an industrial vehicle having the control device for an industrial vehicle as described in claim 1.

4. A program for an industrial vehicle control device used in the industrial vehicle control device described in claim 1.

Images