Anomaly detection device and vehicle

Abstract

To identify an abnormal state for a vehicle.SOLUTION: An abnormality detection device comprises: a control device; and an auxiliary storage device. The auxiliary storage device stores map data and association data. The control device acquires operation value data. The control device acquires image data from a camera. The control device acquires state ID by inputting the operation value data and the image data as input data to a map defined by the map data. The control device identifies an abnormal state for a vehicle from the association data in which the state ID is associated with the abnormal state for the vehicle.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] The present disclosure relates to an abnormality detection device and a vehicle. [Background technology]

[0002] Patent Document 1 discloses a forklift as an autonomously traveling vehicle. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent Publication No. 2021-62964 Summary of the Invention [Problem to be solved by the invention]

[0004] An abnormality may occur in the vehicle while it is in operation. When an abnormality occurs in the vehicle, it is required to detect the abnormality. [Means for solving the problem]

[0005] An abnormality detection device that solves the above problem is an abnormality detection device that detects abnormalities related to a vehicle, and the abnormality detection device includes a control device and a memory device. The memory device stores mapping data that defines a mapping that outputs a state ID corresponding to an abnormal state related to the vehicle as output data when operation value data and image data of the vehicle are input as input data, and correspondence data that associates the state ID with the abnormal state related to the vehicle. The control device acquires the operation value data, acquires the image data from a camera equipped on the vehicle, and acquires the state ID by inputting the operation value data and the image data into the mapping as the input data, and identifies the abnormal state related to the vehicle from the state ID and the correspondence data acquired from the mapping.

[0006] The control device can obtain the state ID by inputting the operation value data and image data as input data into the mapping. The control device can identify an abnormal state related to the vehicle from the state ID and the corresponding data.

[0007] In the above abnormality detection device, the vehicle may be a cargo handling vehicle, and the abnormal state relating to the vehicle may include an abnormal state relating to a cargo handling operation. In the anomaly detection device, the mapping data may be a machine-learned model, and the model may include a feature extraction unit that extracts a feature vector from the input data, and a discretization layer that converts the feature vector into a discrete value associated with the state ID.

[0008] In the above-mentioned anomaly detection device, the vehicle is a vehicle operated by an operator on board the vehicle, and the mapping outputs the state ID as the output data by inputting the operation value data, the image data, and attitude data related to the operator's attitude as the input data, and the control device may acquire the attitude data from an attitude sensor equipped in the vehicle, and use the operation value data, the image data, and the attitude data as the input data.

[0009] A vehicle that solves the above problem includes the above abnormality detection device. The control device can identify an abnormal state related to the vehicle from the state ID and corresponding data. [Effects of the Invention]

[0010] According to the present invention, an abnormal state relating to a vehicle can be identified. [Brief explanation of the drawings]

[0011] [Figure 1] FIG. [Figure 2] FIG. 1 is a schematic configuration diagram of a cargo handling vehicle. [Figure 3] 10 is a flowchart showing abnormality detection control. [Figure 4]FIG. 10 is a diagram for explaining preprocessing. [Figure 5] FIG. 10 is a diagram illustrating input data. [Figure 6] FIG. 1 is a schematic diagram of a model. [Figure 7] FIG. 1 is a schematic diagram illustrating the configuration of a learning model. [Figure 8] FIG. 10 is a schematic configuration diagram showing a modified example of the anomaly detection device. DETAILED DESCRIPTION OF THE INVENTION

[0012] An embodiment of an abnormality detection device and a vehicle will be described. As shown in Fig. 1, the cargo handling vehicle 10 is a reach-type forklift. The cargo handling vehicle 10 is a vehicle. The cargo handling vehicle 10 is configured to be switchable between manual operation by an operator on board the cargo handling vehicle 10 and automatic operation. In the following description, front, rear, left, right and front are determined based on the cargo handling vehicle 10.

[0013] The cargo handling vehicle 10 includes a vehicle body 11, a reach leg 12, front wheels 13, rear wheels 14, a cargo handling device 21, and an operating unit 40. Two reach legs 12 are provided spaced apart from each other in the left-right direction. The reach legs 12 extend forward from the vehicle body 11.

[0014] One front wheel 13 is provided on each reach leg 12. The rear wheels 14 are provided on the vehicle body 11. The rear wheels 14 are steering wheels. The rear wheels 14 are driving wheels. The loading device 21 is provided in front of the vehicle body 11. The loading device 21 includes a mast 22, a lift bracket 25, a fork 26, a lift cylinder 31, a tilt cylinder 32, and a reach cylinder 33.

[0015] The mast 22 is a multi-stage mast and includes an outer mast 23 and an inner mast 24. The inner mast 24 is provided so as to be able to move up and down relative to the outer mast 23.

[0016] The forks 26 are fixed to the lift bracket 25. Two forks 26 are provided spaced apart from each other in the left-right direction. The lift cylinder 31 is a hydraulic cylinder. The lift bracket 25 moves up and down by supplying and discharging hydraulic oil to the lift cylinder 31. The fork 26 moves up and down together with the lift bracket 25.

[0017] The tilt cylinder 32 is a hydraulic cylinder. The lift bracket 25 tilts in the front-rear direction by supplying and discharging hydraulic oil to the tilt cylinder 32. The fork 26 tilts together with the lift bracket 25.

[0018] The reach cylinder 33 is a hydraulic cylinder. The mast 22 moves in the front-rear direction by supplying and discharging hydraulic oil to the reach cylinder 33. The fork 26 moves in the front-rear direction together with the mast 22.

[0019] 2, the operation unit 40 includes a direction operation unit 41, a lift operation unit 42, a tilt operation unit 43, and a reach operation unit 44. The operation unit 40 is operated when the cargo handling vehicle 10 is operated manually.

[0020] The direction operation unit 41 is a lever. The direction operation unit 41 tilts forward or backward from a neutral position. The direction operation unit 41 is operated when the cargo handling vehicle 10 is traveling. The direction operation unit 41 can determine the direction of travel of the cargo handling vehicle 10. The direction operation unit 41 can adjust the speed of the cargo handling vehicle 10. The direction of travel of the cargo handling vehicle 10 is either forward or backward.

[0021] The lift operating unit 42 is a lever. The lift operating unit 42 tilts forward or backward from a neutral position. The lift operating unit 42 is operated when raising or lowering the forks 26. The lift operating unit 42 can determine the direction in which the forks 26 are raised or lowered. The lift operating unit 42 can also adjust the speed at which the forks 26 are raised or lowered.

[0022] The tilt operation unit 43 is a lever. The tilt operation unit 43 tilts forward or backward from a neutral position. The tilt operation unit 43 is operated when tilting the forks 26. The tilt operation unit 43 can determine the tilt direction of the forks 26. The tilt operation unit 43 can also adjust the tilt speed of the forks 26.

[0023] The reach operating unit 44 is a lever. The reach operating unit 44 tilts forward or backward from a neutral position. The reach operating unit 44 is operated when moving the forks 26 in the forward or backward direction. The direction of movement of the forks 26 can be determined by the reach operating unit 44. The movement speed of the forks 26 can be adjusted by the reach operating unit 44.

[0024] The cargo handling vehicle 10 is equipped with a sensor 50, an automatic driving sensor 56, a camera 57, a drive mechanism 61, a hydraulic mechanism 62, a vehicle control device 63, and an abnormality detection device 70. The sensors 50 include a direction sensor 51 , a lift sensor 52 , a tilt sensor 53 , a reach sensor 54 , and a tire angle sensor 55 .

[0025] The direction sensor 51 detects the operation value of the direction operation unit 41. The direction sensor 51 outputs an electric signal according to the operation value of the direction operation unit 41 to the vehicle control device 63.

[0026] The lift sensor 52 detects the operation value of the lift operating unit 42. The lift sensor 52 outputs an electric signal corresponding to the operation value of the lift operating unit 42 to the vehicle control device 63. The tilt sensor 53 detects the operation value of the tilt operation unit 43. The tilt sensor 53 outputs an electric signal according to the operation value of the tilt operation unit 43 to the vehicle control device 63.

[0027] The reach sensor 54 detects the operation value of the reach operation unit 44. The reach sensor 54 outputs an electric signal corresponding to the operation value of the reach operation unit 44 to the vehicle control device 63. The tire angle sensor 55 detects the steering angle of the steered wheels. In this embodiment, the tire angle sensor 55 detects the steering angle of the rear wheels 14. The tire angle sensor 55 outputs an electrical signal corresponding to the steering angle to the vehicle control device 63.

[0028] The automatic driving sensor 56 is a sensor used when automatically operating the cargo handling vehicle 10. The automatic driving sensor 56 includes, for example, an external sensor used to estimate the self-position of the cargo handling vehicle 10, and a distance meter used to align the fork 26 with the cargo handling object. The automatic driving sensor 56 outputs the detection result to the vehicle control device 63. The cargo handling object is, for example, a cargo to be picked up by the cargo handling device 21, and a location where the cargo is to be placed by the cargo handling device 21. Examples of locations where the cargo is to be placed include trucks and shelves. The cargo includes a pallet and cargo loaded on a pallet.

[0029] The camera 57 is a digital camera. The camera 57 includes an imaging element. Examples of the imaging element include a CCD image sensor (Charge Coupled Device image sensor) and a CMOS image sensor (Complementary Metal Oxide Semiconductor image sensor). The camera 57 is a monocular camera. Examples of the camera 57 include an RGB camera, an infrared camera, a grayscale camera, and a visible light camera. The camera 57 is positioned so as to capture an image of the range visible to the operator when the operator is operating the cargo handling vehicle 10. The camera 57 is positioned so as to capture an image of the range extending forward from the driver's seat, for example. The camera 57 outputs image data obtained by capturing the image to the anomaly detection device 70.

[0030] The drive mechanism 61 is a member for causing the cargo handling vehicle 10 to travel. The drive mechanism 61 includes a drive source for driving the rear wheels 14 and a steering mechanism for steering the rear wheels 14. If the cargo handling vehicle 10 uses a motor as the drive source, the drive mechanism 61 includes a motor driver. If the cargo handling vehicle 10 uses an engine as the drive source, the drive mechanism 61 includes a fuel injection device.

[0031] The hydraulic mechanism 62 is a member for controlling the supply and discharge of hydraulic oil to the hydraulic equipment. The hydraulic equipment includes the lift cylinder 31, the tilt cylinder 32, and the reach cylinder 33. The hydraulic mechanism 62 includes a pump that discharges hydraulic oil and a control valve that controls the supply and discharge of hydraulic oil to the hydraulic equipment.

[0032] The vehicle control device 63 controls driving operations and loading / unloading operations. The vehicle control device 63 includes a processor 64 and a storage unit 65. The processor 64 may be, for example, a central processing unit (CPU), a graphics processing unit (GPU), or a digital signal processor (DSP). The storage unit 65 includes a random access memory (RAM) and a read-only memory (ROM). The storage unit 65 stores a program for operating the loading / unloading vehicle 10. The storage unit 65 can be said to store program code or instructions configured to cause the processor 64 to execute processing. The storage unit 65, i.e., a computer-readable medium, includes any available medium accessible by a general-purpose or dedicated computer. The vehicle control device 63 may be configured by a hardware circuit such as an application-specific integrated circuit (ASIC) or a field-programmable gate array (FPGA). The vehicle control device 63, which is a processing circuit, may include one or more processors that operate according to a computer program, one or more hardware circuits such as ASICs or FPGAs, or a combination thereof.

[0033] <Control performed by the vehicle control device during manual operation> When the cargo handling vehicle 10 is operated manually, the vehicle control device 63 causes the cargo handling vehicle 10 to travel in accordance with the detection result of the direction sensor 51. The vehicle control device 63 recognizes the operation value of the direction operation unit 41 from the detection result of the direction sensor 51. The vehicle control device 63 controls the drive mechanism 61 so that the cargo handling vehicle 10 travels in a direction corresponding to the operation value of the direction operation unit 41 at a speed corresponding to the operation value of the direction operation unit 41.

[0034] When the loading vehicle 10 is operated manually, the vehicle control device 63 causes the loading vehicle 10 to perform a loading operation in accordance with the detection results of the lift sensor 52, the tilt sensor 53, and the reach sensor 54. The vehicle control device 63 recognizes the operation value of the lift operation unit 42 from the detection result of the lift sensor 52. The vehicle control device 63 controls the hydraulic mechanism 62 so that the forks 26 move up and down in a lifting direction corresponding to the operation value of the lift operation unit 42, at a speed corresponding to the operation value of the lift operation unit 42. The vehicle control device 63 recognizes the operation value of the tilt operation unit 43 from the detection result of the tilt sensor 53. The vehicle control device 63 controls the hydraulic mechanism 62 so that the forks 26 tilt in a tilting direction corresponding to the operation value of the tilt operation unit 43, at a speed corresponding to the operation value of the tilt operation unit 43. The vehicle control device 63 recognizes the operation value of the reach operation unit 44 from the detection result of the reach sensor 54. The vehicle control device 63 controls the hydraulic mechanism 62 so that the forks 26 move in a direction corresponding to the operation value of the reach operation unit 44 at a speed corresponding to the operation value of the reach operation unit 44 .

[0035] <Control performed by the vehicle control device during automatic operation> When the cargo handling vehicle 10 operates automatically, the vehicle control device 63 generates a route while estimating its own position using the automatic driving sensor 56. The vehicle control device 63 controls the drive mechanism 61 so that the cargo handling vehicle 10 follows the route. At this time, the vehicle control device 63 generates command values ​​related to the traveling operation and controls the drive mechanism 61 in accordance with the command values. The command values ​​related to the traveling operation are commands related to the traveling direction of the cargo handling vehicle 10 and the speed of the cargo handling vehicle 10. The command values ​​can be said to be virtual operation values ​​of the direction operation unit 41.

[0036] During automatic operation of the cargo handling vehicle 10, the vehicle control device 63 aligns the forks 26 while recognizing the positions of the cargo handling target and the forks 26 using the automatic driving sensors 56. At this time, the vehicle control device 63 generates command values ​​related to the cargo handling operation and controls the hydraulic mechanism 62 in accordance with the command values. The command values ​​related to the cargo handling operation include command values ​​related to the lifting direction and lifting speed of the forks 26, command values ​​related to the tilting direction and tilting speed of the forks 26, and command values ​​related to the movement direction and movement speed of the forks 26. The command values ​​related to the lifting direction and lifting speed of the forks 26 can be considered virtual operation values ​​of the lift operation unit 42. The command values ​​related to the tilting direction and tilting speed of the forks 26 can be considered virtual operation values ​​of the tilt operation unit 43. The command values ​​related to the movement direction and movement speed of the forks 26 can be considered virtual operation values ​​of the reach operation unit 44.

[0037] <Anomaly detection device> The abnormality detection device 70 includes a control device 71 and an auxiliary storage device 74 . The control device 71 has, for example, the same hardware configuration as the vehicle control device 63. The control device 71 has a processor 72 and a storage unit 73. Image data output by the camera 57 is input to the control device 71. The control device 71 and the vehicle control device 63 can obtain information from each other using a vehicle communication protocol.

[0038] The auxiliary storage device 74 stores information that can be read by the control device 71. Examples of the auxiliary storage device 74 include a hard disk drive and a solid state drive. The auxiliary storage device 74 stores a model 81, which is mapping data. The auxiliary storage device 74 stores correspondence data 88. The auxiliary storage device 74 is a storage device.

[0039] The model 81 defines a mapping that outputs a state ID corresponding to the state of the cargo handling vehicle 10 as output data when input data is input. The correspondence data 88 associates a state ID with a state related to the cargo handling vehicle 10. Examples of states related to the cargo handling vehicle 10 include an approach state, a tilt state, a pallet interference state, a long-distance state, a state in which a load is present in the depth direction, and a cargo interference state. The approach state indicates a state in which the cargo handling vehicle 10 is approaching the load to be picked up. The tilt state indicates a state in which the inclination angle of the location where the load is placed or the location to be placed is excessively large. The pallet interference state indicates a state in which adjacent pallets are in contact with each other. The long-distance state indicates a state in which the cargo handling vehicle 10 cannot approach a position where the load can be picked up. An example of a long-distance state is a situation in which a load loaded on the bed of a truck is being picked up. When picking up a load loaded on the bed of a truck, the cargo handling vehicle 10 approaches the truck. Because the cargo handling vehicle 10 needs to approach the truck without coming into contact with the truck, there are cases in which the cargo handling vehicle 10 cannot approach a position where the load placed on the bed can be picked up. The state in which a load is present in the depth direction includes a state in which a load is present in the depth direction of the load placement area, and there is a risk of the load to be placed coming into contact with the load present in the depth direction.The state in which a load is present in the depth direction includes a state in which a load is present in the depth direction of the load to be removed, and there is a risk of the load to be removed coming into contact with the load present in the depth direction.The state in which a load interferes includes a state in which a load is in contact with a pallet other than the pallet on which the load is placed.The state in which a load interferes includes a state in which a load is in contact with a load placed on a pallet other than the pallet on which the load is placed.

[0040] The approach state, tilt state, pallet interference state, long distance state, state where load is present in the depth direction, and cargo interference state are states related to loading and unloading operations. In the case of the tilt state, pallet interference state, long distance state, state where load is present in the depth direction, and cargo interference state, there is a risk that the loading and unloading operations of the loading and unloading vehicle 10 may be hindered. These states are abnormal states related to loading and unloading operations. It can be said that the correspondence data 88 associates state IDs with abnormal states related to the loading and unloading vehicle 10.

[0041] <Anomaly detection control> The control device 71 executes abnormality detection control. The abnormality detection control is executed repeatedly at a predetermined control period, for example, while the cargo handling vehicle 10 is in operation. The abnormality detection control executed when the cargo handling vehicle 10 is operating automatically will be described.

[0042] As shown in FIG. 3, in step S1, the control device 71 acquires operation value data of the cargo handling vehicle 10. The operation value data includes data indicating the operation value of the direction operation unit 41, data indicating the operation value of the lift operation unit 42, data indicating the operation value of the tilt operation unit 43, data indicating the operation value of the reach operation unit 44, and data indicating the steering angle. The data indicating the operation value of the direction operation unit 41, data indicating the operation value of the lift operation unit 42, data indicating the operation value of the tilt operation unit 43, and data indicating the operation value of the reach operation unit 44 are each command values ​​generated by the vehicle control device 63. The data indicating the steering angle is the detection result of the tire angle sensor 55. The control device 71 acquires command values ​​indicating each operation value and the detection result of the tire angle sensor 55 from the vehicle control device 63. The operation value data is data indicating operation values ​​that reflect the operation by the operator when the operator manually operates the cargo handling vehicle 10. In other words, it is data indicating values ​​that can be changed depending on the operator's intention. The data indicating the operation value of the operation unit 40 changes depending on the amount of operation of the operation unit 40 by the operator. The data indicating the steering angle changes depending on the amount of operation of the steering wheel by the operator. Therefore, these are operation value data.

[0043] Next, in step S2, the control device 71 acquires image data from the camera 57. Next, in step S3, the control device 71 performs preprocessing on the operation value data and image data, including synchronization, feature extraction, and combining of the operation value data and image data.

[0044] As shown in FIG. 4, the control device 71 synchronizes the operation value data D1 with the image data D2 by matching the sampling rates of the operation value data D1 and the image data D2.

[0045] The control device 71 extracts features from the image data D2. The extraction of features from the image data D2 is performed using a deep learning model. In this embodiment, the control device 71 extracts features from the image data D2 using a convolution neural network.

[0046] The control device 71 generates input data D3 by combining the features obtained from the operation value data D1 and the image data D2. As shown in Fig. 5, the input data D3 is generated so as to become time-series data for a certain period of time by deleting the oldest data and adding the latest data.

[0047] 3, next, in step S4, the control device 71 inputs the input data D3 to a mapping defined by a model 81. The model 81 is a trained model that has been trained by machine learning. The mapping has the operation value indicated by the operation value data D1 and the pixel value of the image data as input variables, and the state ID as an output variable.

[0048] As shown in FIG. 6, the model 81 includes an input unit 82, a feature extraction unit 83, and a state estimation unit 86. Input data D3 is input to the input unit 82. The input unit 82 divides the input data D3 to create tokens.

[0049] The feature extraction unit 83 includes a position embedding layer 84 and a self-attention layer 85. The position embedding layer 84 adds position information to the tokens. More specifically, the position embedding layer 84 adds an eigenvector corresponding to the position of the token to the token as position information.

[0050] The self-attention layer 85 calculates points of interest from the input data D3. The self-attention layer 85 calculates three vectors: Query, Key, and Value. The self-attention layer 85 calculates the degree of relevance between Query and Key by calculating the inner product of Query and Key and applying softmax. The self-attention layer 85 calculates a weighted sum of Value according to the degree of relevance between Query and Key. This allows the feature extraction unit 83 to extract a feature vector, which is the feature of each token.

[0051] The state estimation unit 86 estimates a state ID from the feature vector. The state estimation unit 86 includes a discretization layer 87. The discretization layer 87 calculates the distance between the feature vector and the state vector. The discretization layer 87 identifies a state vector for which the distance between the feature vector and the state vector is smallest. The state vector is a discrete value. The discretization layer 87 can convert the feature vector into a discrete value. The discretization layer 87 derives a state ID associated with the state vector from a state ID map. The state ID map is a map that associates state vectors with state IDs. The state estimation unit 86 identifies the state vector for which the distance from the feature vector is smallest, and outputs the state ID associated with the state vector as the current state ID.

[0052] As shown in FIG. 3, next, in step S5, the control device 71 identifies the state of the cargo handling vehicle 10 associated with the state ID obtained in step S4 from the correspondence data 88. This allows the control device 71 to identify the current state of the cargo handling vehicle 10. It is assumed that the state ID "0" in the correspondence data 88 is associated with the approach state. If the state ID obtained in step S4 is "0", the control device 71 identifies the current state of the cargo handling vehicle 10 as the approach state. It is assumed that the state ID "1" in the correspondence data 88 is associated with the tilt state. If the state ID obtained in step S4 is "1", the control device 71 identifies the current state of the cargo handling vehicle 10 as the tilt state.

[0053] Next, in step S6, the control device 71 determines whether the cargo handling vehicle 10 is in an abnormal state based on the current state of the cargo handling vehicle 10. By classifying the state of the cargo handling vehicle 10 into an abnormal state and a normal state, the control device 71 can determine whether the cargo handling vehicle 10 is in a normal state or an abnormal state. In the example described above, if the state of the cargo handling vehicle 10 is an approach state, the control device 71 can determine that the cargo handling vehicle 10 is in a normal state. If the state of the cargo handling vehicle 10 is a pallet interference state, a long distance state, a state where a load is present in the depth direction, or a cargo interference state, the control device 71 can determine that the cargo handling vehicle 10 is in an abnormal state. In this way, the abnormality detection device 70 detects an abnormality related to the cargo handling vehicle 10.

[0054] If the control device 71 determines that the cargo handling vehicle 10 is in an abnormal state, subsequent control is optional. If the vehicle control device 63 can cause the cargo handling vehicle 10 to perform an operation corresponding to the abnormal state, the vehicle control device 63 may perform control corresponding to the abnormal state. The vehicle control device 63 may also stop the cargo handling vehicle 10. The vehicle control device 63 or the control device 71 may notify a higher-level control device that the cargo handling vehicle 10 is in an abnormal state.

[0055] When the cargo handling vehicle 10 is operated manually, the operation value obtained from the sensor 50 can be used as the operation value data D1. In this way, the state of the cargo handling vehicle 10 can be identified whether the cargo handling vehicle 10 is operating automatically or manually.

[0056] When the cargo handling vehicle 10 is being operated manually, if the control device 71 determines that the cargo handling vehicle 10 is in an abnormal state, a notification may be given to the operator. The notification may be given, for example, by displaying a message on a display unit that is visible to the operator, or by turning on a lamp or activating a buzzer.

[0057] <Model generation method> A method for generating the model 81 will be described. The model 81 is generated by self-supervised learning. As an example, a case where a VQ-VAE (Vector Quantized-Variational Auto Encoder) is used as a learning model will be described.

[0058] 7, the learning model 90 includes an encoder 91 and a decoder 93. The learning model 90 extracts a state vector, which is a feature, from the learning data D4 and reconstructs the learning data D4 from the state vector, thereby constituting a state estimation unit 92. The state estimation unit 92 is a latent space to which the state vector belongs.

[0059] Learning data D4 is input to the encoder 91. The encoder 91 extracts latent variables, which are feature quantities, from the learning data D4. The state estimation unit 92 converts the latent variables into a finite number of discrete values ​​by mapping the latent variables to a state ID map. The discrete values ​​are state vectors. The state ID map associates state IDs with state vectors. The state estimation unit 92 replaces the latent variables with the state vector that is closest in distance.

[0060] The decoder 93 reconstructs the training data D4 from the state vector. The learning data D4 includes operation value data D5 when the operator operates the cargo handling vehicle 10, visual data D6 when the operator operates the cargo handling vehicle 10, and posture data D7 when the operator operates the cargo handling vehicle 10. The learning data D4 can be acquired using a cargo handling vehicle with the same specifications as the cargo handling vehicle 10. A cargo handling vehicle with the same specifications is a cargo handling vehicle that performs the same operation at the same speed when the same operation values ​​are input in the same surrounding environment. As an example, a case where the learning data D4 is acquired using the cargo handling vehicle 10 will be described.

[0061] Operation value data D5 when the operator operates the cargo handling vehicle 10 can be acquired from the direction sensor 51, lift sensor 52, tilt sensor 53, reach sensor 54, and tire angle sensor 55. Visual data D6 when the operator operates the cargo handling vehicle 10 is data indicating the location of the operator's viewpoint. The visual data D6 can be acquired from an eye tracker that measures the operator's line of sight, by attaching the eye tracker to the operator. Posture data D7 when the operator operates the cargo handling vehicle 10 is data related to the operator's posture. More specifically, the posture data D7 is data indicating the operator's posture angle and joint coordinates when the operator operates the cargo handling vehicle 10. The posture data D7 can be acquired using a posture sensor. The posture sensor is, for example, a camera that captures an image of the operator, or a depth sensor.

[0062] There is a correlation between the state of the cargo handling vehicle 10 and the operation value data D5. For example, if the state of the cargo handling vehicle 10 is the "approach state," it can be assumed that the operation value of the direction operation unit 41 will be a value that instructs the cargo handling vehicle 10 to move forward. If the state of the cargo handling vehicle 10 is the "tilted state," the operator of the cargo handling vehicle 10 may adjust the tilt angle of the forks 26, thereby changing the operation value of the tilt operation unit 43. If the state of the cargo handling vehicle 10 is the "pallet interference state," the steering angle may be changed to release the pallets from contact with each other. In this way, it is possible to understand what state the cargo handling vehicle 10 was in from the operation value data D5.

[0063] The location of the operator's viewpoint changes depending on the state of the cargo handling vehicle 10. For example, when the state of the cargo handling vehicle 10 is the "approach state," it can be assumed that the operator's viewpoint will be the pallet. When the state of the cargo handling vehicle 10 is the "inclined state," it can be assumed that the operator's viewpoint will be the inclined location, i.e., the location where the pallet is placed or the location where the cargo is to be placed. When the state of the cargo handling vehicle 10 is the "pallet interference state," it can be assumed that the operator's viewpoint will be the location where the pallets are in contact with each other. In this way, it is possible to determine the location where the operator was focusing from the visual data D6.

[0064] The operator's posture changes depending on the state of the cargo handling vehicle 10. For example, if the state of the cargo handling vehicle 10 is a "pallet interference state" or a "long distance state," it can be assumed that the operator will lean forward in front of the cargo handling vehicle 10 in order to grasp the state of the pallet. Also, if the operator's vision is blocked by the cargo handling device 21, it can be assumed that the operator will assume a posture such that they are peering at the pallet from the left or right of the cargo handling vehicle 10.

[0065] As described above, by using the operation value data D5 and the visual data D6 as the learning data D4, it is possible to reflect in the model 81 what location the operator of the cargo handling vehicle 10 focused on and what state the cargo handling vehicle 10 was in. In other words, it is possible to learn what portion of the image data D2 to focus on in order to extract a feature vector and what state the cargo handling vehicle 10 was in. Furthermore, by using the posture data D7 as the learning data D4, it is possible to reflect in the model 81 what state the cargo handling vehicle 10 was in and what state the cargo handling vehicle 10 was in. It can be said that the model 81 specifies a mapping that outputs a state ID corresponding to the state of the cargo handling vehicle 10 as output data by inputting the operation value data D1, the image data D2, and the posture data as input data.

[0066] After learning is performed using the learning data D4, the encoder 91 and the state estimation unit 92 of the learning model 90 are used as the model 81. <How to generate correspondence data> The corresponding data 88 is generated after the model 81 is generated. By generating the model 81 as described above, a state ID map can be obtained in which a state ID is associated with a state vector. This state ID is a unique ID corresponding to a state related to the cargo handling vehicle 10. The state to which the state ID corresponds among the states related to the cargo handling vehicle 10 can be ascertained, for example, by a person checking the learning data D4. Then, the corresponding data 88 can be generated by associating the state ID with the state related to the cargo handling vehicle 10 that corresponds to the state ID.

[0067] [Operation of this embodiment] While the vehicle is operating, the state of the vehicle changes depending on the vehicle itself and the environment around the vehicle. In particular, during the loading operation of the cargo handling vehicle 10, there is a risk of an abnormal state occurring depending on the object being loaded. The operator of the cargo handling vehicle 10 can notice these abnormal states. In this embodiment, when an abnormal state occurs, the abnormal state is identified by utilizing the location at which the operator of the cargo handling vehicle 10 focuses their attention. The operation value data D5 and the visual data D6 are correlated with the state of the cargo handling vehicle 10. The model 81 obtained using these as learning data D4 can be said to represent the location at which the operator focused their attention to grasp the state of the cargo handling vehicle 10.

[0068] By using the operation value data D1 and the image data D2 as input data D3, the control device 71 can acquire a state ID corresponding to the state of the cargo handling vehicle 10. In particular, by using the model 81 generated as described above, it is possible to extract a feature vector based on the degree of association between the operation value data D1 and the position of the image data D2. In other words, it is possible to extract a feature vector by focusing on a location in the image data D2 that has a high degree of association with the state of the cargo handling vehicle 10.

[0069] [Effects of this embodiment] (1) The control device 71 inputs the operation value data D1 and the image data D2 as input data D3 to the mapping. This allows the control device 71 to obtain, as output data, a status ID corresponding to the status of the cargo handling vehicle 10. The correspondence data 88 associates the status ID with the status of the cargo handling vehicle 10. The control device 71 can identify an abnormal status of the cargo handling vehicle 10 from the status ID and the correspondence data 88.

[0070] (2) The cargo handling vehicle 10 is used in a variety of locations, such as ports, commercial facilities, and factories. In addition, the type of pallet and the type of cargo may vary depending on the user. For this reason, unknown data is likely to be input as the input data D3. By outputting a state ID using a mapping defined by the machine-learned model 81, it is possible to identify an abnormal state related to the cargo handling vehicle 10 even when unknown data is input as the input data D3.

[0071] (3) The model 81 includes a discretization layer 87. The discretization layer 87 converts the feature vector into a state vector, which is a discrete value. The input data D3, which is time-series data, can be converted into discrete values. This allows the time-series data to be classified into state IDs corresponding to the states of the cargo handling vehicle 10.

[0072] (4) The model 81 is generated by self-supervised learning. When generating a model by supervised learning, it is necessary to label the training data D4. In contrast, with self-supervised learning, it is only necessary to associate the state ID with the state of the cargo handling vehicle 10, which reduces the effort required compared to labeling.

[0073] (5) Even when a manual operation is being performed by the operator, the control device 71 can identify an abnormal state regarding the cargo handling vehicle 10. When the control device 71 notifies the operator of an abnormal state, the operator can recognize the abnormal state even if the operator has overlooked the abnormal state regarding the cargo handling vehicle 10.

[0074] [Example of change] The embodiment can be modified as follows: The embodiment and the following modifications can be combined with each other to the extent that they are not technically inconsistent.

[0075] The vehicle may be different from the cargo handling vehicle 10. For example, the vehicle may be a passenger car, a towing vehicle, or a transport vehicle. The cargo handling vehicle 10 may be a counter-loaded forklift. In this way, the type of vehicle may be changed as appropriate. In this case, an individual model 81 is generated for each type of vehicle. For example, in the case of a counter-loaded forklift, a model may be generated using data indicating the operation value of an accelerator sensor instead of the direction sensor 51.

[0076] The cargo handling vehicle 10 may be equipped with an attachment. In this case, if the cargo handling vehicle 10 is equipped with an attachment operating unit that operates the attachment, it is preferable to include data indicating the operation value of the attachment operating unit in the learning data D4. Examples of attachments include a side shift device that moves the forks 26 in the left-right direction, and a clamping device that clamps cylindrical loads.

[0077] The cargo handling vehicle 10 equipped with the anomaly detection device 70 may be a cargo handling vehicle that can only operate automatically. In this case, the learning data D4 may be acquired using a cargo handling vehicle equipped with the operation unit 40 and the sensor 50, and the cargo handling vehicle 10 equipped with the anomaly detection device 70 does not need to be equipped with the operation unit 40 or the sensor 50.

[0078] The cargo handling vehicle 10 equipped with the abnormality detection device 70 may be a cargo handling vehicle that can only be operated manually. As shown in FIG. 8, the cargo handling vehicle 10 may be equipped with an attitude sensor 58. The attitude sensor 58 may be, for example, a camera that captures an image of the operator or a depth sensor. The attitude sensor 58 detects attitude data. The attitude data is the same data as the attitude data D7. When executing abnormality detection control, the control device 71 uses the operation value data D1, image data D2, and the attitude data acquired from the attitude sensor 58 as input data D3. In this case, the cargo handling vehicle 10 is operated by an operator aboard the cargo handling vehicle 10. That is, the cargo handling vehicle 10 is the cargo handling vehicle 10 described in the embodiment, or a cargo handling vehicle that can only be operated manually. Using the attitude data as input data D3 can improve the accuracy with which the abnormality detection device 70 identifies an abnormal state.

[0079] A model 81 that has undergone additional learning depending on the user of the cargo handling vehicle 10 may also be used. The cargo handling vehicle 10 is used in a variety of locations, such as ports, commercial facilities, or factories. The types of pallets and cargo may also differ depending on the user. For this reason, the cargo handling vehicle 10 may experience abnormal conditions unique to each user. A model 81 that is tailored to the user can be obtained by performing additional learning using the operation value data D5, visual data D6, and posture data D7 obtained when the user uses the cargo handling vehicle 10 as learning data D4.

[0080] The state related to the cargo handling vehicle 10 may include a state related to the traveling operation. The state related to the traveling operation may include, for example, an abnormal state related to the traveling operation, such as when the cargo handling vehicle 10 is unable to travel due to an obstacle. If the cargo handling vehicle 10 imposes a speed limit on the vehicle when a specific condition is met, the state related to the traveling operation may include a speed limit state. Examples of the specific condition include when the cargo handling vehicle 10 is traveling in a specific location or when a specific obstacle is detected. Examples of the specific obstacle include a person.

[0081] The learning data D4 may consist only of the operation value data D5 and the visual data D6. In this case, the model 81 receives the operation value data D1 and the image data D2 as input data, and defines a mapping that outputs a state ID corresponding to the state of the cargo handling vehicle 10 as output data.

[0082] The model 81 may be generated by unsupervised learning. For example, the learning data D4 may be classified into groups using an algorithm that extracts a group structure from the learning data D4. These groups may be used as state IDs, and the state IDs may be associated with the states of the cargo handling vehicle 10 using the correspondence data 88.

[0083] The camera 57 may also be used for purposes other than anomaly detection control. For example, when the vehicle control device 63 automatically operates the cargo handling vehicle 10, the vehicle control device 63 may use the image data D2 to control the operation. For example, the vehicle control device 63 may use the image data D2 to align the pallet and the forks 26.

[0084] The cargo handling vehicle 10 may be remotely operated. In this case, the operator operates the cargo handling vehicle 10 from a remote location away from the cargo handling vehicle 10. The operator operates the cargo handling vehicle 10 while visually checking images captured by a camera installed on the cargo handling vehicle 10. The camera 57 may be used as the camera in this case.

[0085] The correspondence data 88 may simply associate a state ID with an abnormal state related to the cargo handling vehicle 10. Therefore, the state associated with the state ID by the correspondence data 88 does not have to include a normal state.

[0086] The model 81 and the correspondence data 88 may be stored in the memory unit 73 instead of the auxiliary storage device 74. In this case, the memory unit 73 is the memory device. The control device 71 is the anomaly detection device 70. One of the model 81 and the correspondence data 88 may be stored in the memory unit 73, and the other may be stored in the auxiliary storage device 74. In this case, the memory unit 73 and the auxiliary storage device 74 are the memory devices.

[0087] The abnormality detection device does not have to be provided in the cargo handling vehicle 10. For example, the abnormality detection device may be provided at the location where the cargo handling vehicle 10 is operated, or at a remote location away from the location where the cargo handling vehicle 10 is operated. In this case, the cargo handling vehicle 10 and the abnormality detection device each include a communication device. The communication device is a communication device capable of communicating using any wireless communication method, such as wireless LAN, Zigbee (registered trademark), LPWA (Low Power Wide Area), or a mobile communication system. In this case, the vehicle control device 63 transmits operation value data and image data to the abnormality detection device via the communication device. The abnormality detection device acquires the operation value data and image data by receiving the operation value data and image data via the communication device. The abnormality detection device then performs the processes of steps S3 to S6 using the operation value data and image data acquired from the cargo handling vehicle 10. In this case, the abnormality detection device may transmit the detection result to the cargo handling vehicle 10.

[0088] The anomaly detection device may include a first device provided in the cargo handling vehicle 10, and a second device provided at the location where the cargo handling vehicle 10 is operated or at a remote location away from the location where the cargo handling vehicle 10 is operated. Each of the first device and the second device may have, for example, the same hardware configuration as the anomaly detection device of the embodiment. Each of the first device and the second device may include a communication device. The first device performs some of the processes of steps S3 to S5. The second device performs processes subsequent to the process performed by the first device. For example, the first device generates input data D3 by performing the processes of steps S1 to S3. The first device transmits the input data D3 to the second device via the communication device. The second device receives the input data D3 via the communication device. The second device performs the processes of steps S4 to S6 using the input data D3. In this case, the second device may transmit the detection result to the first device. [Explanation of symbols]

[0089] D1...operation value data, D2...image data, D3...input data, 10...loading vehicle as vehicle, 57...camera, 58...posture sensor, 70...anomaly detection device, 71...control device, 74...auxiliary memory device as memory device, 81...model, 83...feature extraction unit, 87...discretization layer, 88...corresponding data.

Claims

1. An abnormality detection device that detects an abnormality related to a vehicle, the vehicle is operated by an operator on board the vehicle, The anomaly detection device a control device; a storage device, The storage device mapping data that defines a mapping that outputs, as output data, a state ID corresponding to an abnormal state related to the vehicle, by receiving, as input data, operation value data of the vehicle, image data, and posture data related to the posture of the operator; and storing correspondence data in which the state ID is associated with an abnormal state related to the vehicle, The control device Acquire the operation value data; acquiring the image data from a camera equipped in the vehicle; acquiring the attitude data from an attitude sensor provided in the vehicle; the state ID is acquired by inputting the operation value data, the image data, and the attitude data as the input data into the mapping; An abnormality detection device that identifies an abnormal state related to the vehicle from the state ID and the corresponding data acquired from the mapping.

2. the vehicle is a cargo handling vehicle, The abnormality detection device according to claim 1 , wherein the abnormal condition relating to the vehicle includes an abnormal condition relating to a loading and unloading operation.

3. The mapping data is a machine-learned model, The model is a feature extraction unit that extracts a feature vector from the input data; 3. The anomaly detection device according to claim 1, further comprising: a discretization layer that converts the feature vector into a discrete value associated with the state ID.

4. A vehicle equipped with the abnormality detection device according to claim 1.

Images