Determination device, forklift, and learning method

Abstract

To provide a determination device, a forklift, and a learning method capable of identifying a position of a pallet regardless of the position at which the pallet is placed or its peripheral environment.SOLUTION: A determination device 10 includes: an irradiation control unit 101 for operating a headlight 28 in a first mode and a second mode; an imaging control unit 102 for controlling the operation of an imaging unit 29 so as to image a pallet in a state of operating the headlight in the first mode and generate a first captured image; an identification unit 103 for identifying a position of the pallet based on the first captured image; a determination unit 104 for determining whether or not the position of the pallet is identified; and an output unit 105 for outputting information indicating the position of the pallet when it is determined that the position of the pallet is identified. When it is determined that the position of the pallet cannot be identified, the identification unit 103 identifies the position of the pallet based on a second captured image and the determination unit 104 further determines whether or not the position of the pallet can be identified.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present invention relates to a determination device, a forklift, and a learning method. [Background technology]

[0002] A forklift transports pallets on which objects to be loaded are placed. A technique is known for a forklift to identify the position of a pallet by performing image processing on an image of the pallet (for example, Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent Publication No. 2021-068138 Summary of the Invention [Problem to be solved by the invention]

[0004] Here, depending on the location of the pallet and the surrounding environment, even if an image of the pallet is captured, the characteristics of the pallet may not be properly shown in the image. In such a case, with the technology disclosed in Patent Document 1, even if image processing is performed on the image of the pallet, it may be difficult to identify the position of the pallet. [Means for solving the problem]

[0005] The determination device for achieving the above object includes an illumination control unit that controls an illumination unit that illuminates a pallet on which an object to be handled by a forklift is placed, the illumination control unit causing the illumination unit to operate in at least one of a first mode and a second mode different from the first mode; an imaging control unit that captures an image of the pallet and a pallet hole in the pallet in a state in which the illumination control unit has operated the illumination unit in the first mode, and controls the operation of the imaging unit to generate a first captured image; an identification unit that identifies the position of the pallet based on the first captured image; and a determination unit that determines whether the position of the pallet has been identified based on the first captured image. and an output unit that outputs information indicating the position of the pallet if the determination unit determines that the position of the pallet has been identified, and if the determination unit determines that the position of the pallet cannot be identified based on the first captured image, the imaging control unit images the pallet and a pallet hole in the pallet in a state in which the irradiation control unit operates the irradiation unit in the second mode, and controls the operation of the imaging unit to generate a second captured image, the identification unit identifies the position of the pallet based on the second captured image, and the determination unit further determines whether or not the position of the pallet has been identified based on the second captured image.

[0006] With this configuration, the position of the pallet can be identified regardless of the location where the pallet is placed or the surrounding environment. In the above-mentioned determination device, the irradiation unit may have a plurality of light sources that irradiate light in different wavelength ranges, and the irradiation control unit may, in the first aspect, cause a first light source to irradiate light onto the pallet, and, in the second aspect, cause a second light source that irradiates light with a narrower wavelength range than the first light source to irradiate light onto the pallet.

[0007] According to this configuration, the position of the pallet can be identified by irradiating the pallet with light from the second light source and acquiring a second captured image showing the characteristics of the pallet. In the above determination device, the light from the second light source may include blue light.

[0008] In the above-mentioned determination device, the identification unit may identify the position of the pallet based on a trained model generated by machine learning using a teacher image showing the pallet and the pallet hole and the position of the pallet in the teacher image as teacher data, and based on the first captured image or the second captured image.

[0009] With this configuration, the position of the pallet can be identified using a trained model that has been trained based on the location of the pallet and the surrounding environment. In the above-mentioned determination device, the teacher image may include at least one of an image in which the pallet is irradiated with light from a third light source, an image in which the pallet is irradiated with light from a fourth light source that irradiates light with a narrower wavelength range than the third light source, an image in which the pallet is irradiated with natural light, and an image in which a shadow of an object present around the forklift is cast on the pallet.

[0010] With this configuration, the position of the pallet can be identified using a trained model that has been trained based on the location of the pallet and the surrounding environment. The above-mentioned determination device may further include a pallet hole position identification unit that identifies the position of the pallet hole based on the position of the pallet identified by the identification unit, and the output unit may output information indicating the position of the pallet hole.

[0011] With this configuration, the position of the pallet hole can be identified regardless of the location of the pallet or the surrounding environment. A forklift that achieves the above object includes any one of the determination devices described above, and a drive control unit that aligns the position of the forks of the forklift with the pallet holes based on the position of the pallet.

[0012] With this configuration, the position of the pallet can be identified and the forks can be inserted into the pallet holes regardless of where the pallet is placed or the surrounding environment. A learning method for achieving the above-mentioned objective involves training a learning model through machine learning using training data including a pallet on which an object to be loaded by a forklift is placed, a teacher image showing pallet holes in the pallet, and the position of the pallet in the teacher image, wherein the training images include at least one of an image in which the pallet is irradiated with light from a third light source, an image in which the pallet is irradiated with light from a fourth light source that irradiates light with a narrower wavelength range than the third light source, an image in which the pallet is irradiated with natural light, and an image in which the shadow of an object present around the forklift is cast on the pallet.

[0013] With this configuration, it is possible to generate a trained model that can identify the position of a pallet regardless of where the pallet is placed or the surrounding environment. Effect of the Invention

[0014] According to the present invention, the position of a pallet can be identified regardless of the location where the pallet is placed or the surrounding environment. [Brief description of the drawings]

[0015] [Figure 1] FIG. 1 is a diagram used to explain a forklift. [Diagram 2] FIG. 2 is a diagram illustrating an example of a configuration of a determination device. [Diagram 3] 4A to 4C are diagrams illustrating an example of a first captured image and a second captured image. [Figure 4] 4 is a flowchart showing an example of an operation of the determination device. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0016] <Embodiment> [About Forklift 20] Hereinafter, specific embodiments of the determination device will be described with reference to the drawings.

[0017] As shown in FIG. 1, the determination device 10 is provided in, for example, a forklift 20. In this embodiment, the forklift 20 travels under automatic driving control. Automatic driving control refers to driving the vehicle by controlling only the steering, only the speed, or both the steering and the speed of the forklift 20, without relying on the driving operation of a driver riding on the forklift 20. The forklift 20 transfers the pallet while traveling under automatic driving control based on information indicating the position of the pallet obtained from the determination device 10. Details of the determination device 10 will be described later.

[0018] The forklift 20 includes, for example, wheels 21, forks 22, a handle 23, an operating lever 24, a travel actuator 25a that drives the wheels 21, a loading actuator 25b that drives the forks 22, a communication unit 26, a control unit 27, a headlight 28, and an imaging unit 29. The handle 23 is used to steer the forklift 20 when a driver is on board the forklift 20. Also, the operating lever 24 is used to operate the forks 22 when a driver is on board the forklift 20.

[0019] As described above, in this embodiment, the forklift 20 is a forklift that travels under automatic driving control. Therefore, the forklift 20 does not need to include the handle 23 and the operation lever 24 that are used when the driver riding on the forklift 20 operates the forklift 20.

[0020] The travel actuator 25a is used for traveling of the forklift 20. Specifically, the travel actuator 25a rotates the wheels 21 and changes the steering angle to change the traveling direction of the forklift 20. The loading actuator 25b is for performing a loading operation using the forks 22. Specifically, the loading actuator 25b performs a lifting operation to raise and lower the forks 22 and a tilting operation to tilt the forks 22. A pallet is loaded on the forks 22. For example, a loading object to be loaded by the forklift 20 is placed on the pallet. In addition, the forks 22 are raised or lowered to load or unload the loading object placed on the pallet.

[0021] The forklift 20 may be, for example, a battery forklift. In this case, the traveling actuator 25a is realized by a traveling electric motor that rotates the wheels 21, an electric motor for steering, etc. Also, the loading actuator 25b is realized by a hydraulic pump, a hydraulic cylinder, etc. for performing loading and unloading operations.

[0022] The communication unit 26 includes, for example, a communication circuit or a communication module. The communication unit 26 transmits and receives various data to and from the determination device 10 and other devices. The communication unit 26 performs communication via wired or wireless communication. In the case of wireless communication, the communication mode of the communication unit 26 may be short-range wireless communication such as Wi-Fi (registered trademark) or Bluetooth (registered trademark). In addition, in the case of wireless communication, the communication mode of the communication unit 26 may be communication via a network such as a LAN (Local Area Network), a provider terminal, a wireless communication network, a wireless base station, or a dedicated line.

[0023] The control unit 27 controls the travel actuator 25a and the loading actuator 25b based on the information indicating the position of the pallet acquired from the determination device 10. More specifically, the control unit 27 controls the travel actuator 25a so as to travel to the position of the pallet indicated by the information indicating the position of the pallet. Furthermore, the control unit 27 controls the loading actuator 25b so as to align the position of the fork 22 with the position of the pallet hole of the pallet based on the information indicating the position of the pallet. The loading actuator 25b is an example of a "drive control unit".

[0024] The headlight 28 irradiates light, for example, forward in the traveling direction of the forklift 20. The headlight 28 also irradiates light to the pallet based on the control of the determination device 10. Hereinafter, the headlight 28 will be described in the case where it has a plurality of light sources, a first light source 28a and a second light source 28b. The first light source 28a and the second light source 28b are, for example, light sources that irradiate light in different wavelength ranges. In detail, the first light source 28a irradiates light in a wider wavelength range than the second light source 28b. Specifically, the first light source 28a is realized by white light. The second light source 28b irradiates light in a narrower wavelength range than the first light source 28a. Specifically, the second light source 28b is realized by blue light. Hereinafter, when the first light source 28a and the second light source 28b are not distinguished from each other, they will be simply referred to as "headlight 28". The headlight 28 is an example of an "irradiation unit".

[0025] The imaging unit 29 is provided, for example, at a position on the forks 22 that does not interfere with the loading and unloading operation, or on a mast supporting the forks 22. The imaging unit 29 captures an image of the surrounding environment of the forklift 20 and generates a captured image. Under the control of the determination device 10, the imaging unit 29 also captures an image of the pallet in a state where the pallet is irradiated with light from the headlights 28 and where the pallet and the pallet holes are included in one captured image, and generates the captured image.

[0026] [Regarding the determination device 10] As shown in FIG. 2, the determination device 10 includes, for example, a communication unit 11, a control unit 100, and a storage unit 200. The communication unit 11 includes, for example, a communication circuit or a communication module. The communication unit 11 transmits and receives various data to and from the communication unit 26 and other devices. The communication unit 11 performs communication by wire or wirelessly. In the case of wireless communication, the communication mode of the communication unit 11 may be short-range wireless communication such as Wi-Fi or Bluetooth. In addition, in the case of wireless communication, the communication mode of the communication unit 11 may be communication via a network such as a LAN, a provider terminal, a wireless communication network, a wireless base station, or a dedicated line.

[0027] The control unit 100 is realized by, for example, a hardware processor such as a CPU (Central Processing Unit) executing a program (software). Some or all of these components may be realized by hardware (including circuitry) such as an LSI (Large Scale Integration), an ASIC (Application Specific Integrated Circuit), an FPGA (Field-Programmable Gate Array), or a GPU (Graphics Processing Unit), or may be realized by a combination of software and hardware. The program may be stored in advance in a storage unit 200 that includes a non-transitory storage medium such as an HDD (Hard Disk Drive) or a flash memory.

[0028] The storage unit 200 may be realized by the above-mentioned various storage devices, or an EEPROM (Electrically Erasable Programmable Read Only Memory), a ROM (Read Only Memory), a RAM (Random Access Memory), etc. In addition to the above-mentioned programs, the storage unit 200 stores a trained model 201. Details of the trained model 201 will be described later.

[0029] The control unit 100 includes, as its functional units, an irradiation control unit 101, an imaging control unit 102, a specifying unit 103, a determining unit 104, and an output unit 105, for example. The illumination control unit 101 controls, for example, the operation of the headlights 28. The operation of the headlights 28 includes turning the headlights 28 on and off.

[0030] The imaging control unit 102 controls, for example, the operation of the imaging unit 29. The operation of the imaging unit 29 includes, for example, capturing an image of an imaging target and generating a captured image, changing settings related to imaging by the imaging unit 29, and changing the range imaged by the imaging unit 29. The settings related to imaging by the imaging unit 29 include, for example, settings of the focus position, aperture value, shutter speed, ISO sensitivity, etc.

[0031] The identification unit 103 identifies the position of the pallet based on the trained model 201 and the captured image generated by the imaging unit 29. Here, the trained model 201 is a computation model that outputs coordinate information indicating the position of the pallet when an image showing a pallet and a pallet hole is input. The coordinates indicating the position of the pallet are, for example, coordinates whose origin is the imaging unit 29 that captured the image.

[0032] The trained model 201 is a trained model generated by machine learning using, for example, a teacher image TP and the position of a palette in the teacher image TP as training data. In an example shown in FIG. 3, a palette PL1 and a palette hole PH1 of the palette PL1 are shown in the teacher image TP. A palette PL2 and a palette hole PH2 of the palette PL2 are also shown in the teacher image TP. The network used to train the trained model 201 may be, for example, a deep neural network (DNN) or a recurrent neural network (RNN). Note that the teacher image TP may show one palette and a palette hole of the palette.

[0033] The teacher image TP includes, for example, at least one of the first teacher image TP1 to the fourth teacher image TP4. The first teacher image TP1 is, for example, an image of a pallet irradiated with white light. The second teacher image TP2 is an image of a pallet irradiated with blue light. The third teacher image TP3 is an image of a pallet irradiated with natural light. The fourth teacher image TP4 is an image of a pallet in which the shadow of an object existing around the forklift is cast on the pallet. The shadow of an object existing around the forklift includes the shadow of a structure constituting the space in which the forklift exists, the shadow of a structure equipped on the forklift, and the like. The white light is an example of a "third light source," and the blue light is an example of a "fourth light source."

[0034] As shown in Fig. 2, the determination unit 104 determines whether or not the position of the palette has been identified by the processing of the identification unit 103. For example, when the coordinate values ​​indicating the position of the palette acquired by the processing of the identification unit 103 are values ​​within a predetermined range, the determination unit 104 determines that the position of the palette has been identified. Furthermore, the determination unit 104 determines that the position of the palette cannot be identified when the position of the palette cannot be acquired by the processing of the identification unit 103, or when the acquired coordinate values ​​indicating the position of the palette are values ​​outside the predetermined range. The value of the predetermined range is, for example, a value indicating the coordinates of a range shown in an image captured by the imaging unit 29.

[0035] When the determination unit 104 determines that the position of the pallet has been identified, the output unit 105 outputs information indicating the position of the pallet identified by the processing of the identification unit 103 to the control unit 27 via the communication unit 11. The control unit 27 controls the traveling actuator 25a and the loading actuator 25b based on the information indicating the position of the pallet obtained from the determination device 10.

[0036] [If the pallet location cannot be determined] Here, when the determination unit 104 determines that the position of the pallet cannot be specified, the illumination control unit 101 operates the headlight 28 in another mode different from the initial mode. Hereinafter, a case where the illumination control unit 101 controls the operation of the headlight 28 to four modes, namely, the first mode to the fourth mode, will be described. The first mode is a state in which, of the first light source 28a and the second light source 28b, only the first light source 28a is turned on. The second mode is a state in which, of the first light source 28a and the second light source 28b, only the second light source 28b is turned on. The third mode is a state in which both the first light source 28a and the second light source 28b are turned on. The fourth mode is a state in which both the first light source 28a and the second light source 28b are turned off.

[0037] Furthermore, when the determination unit 104 determines that the position of the pallet cannot be specified, the imaging control unit 102 causes the imaging unit 29 to capture an image of the pallet and the pallet hole and generate a captured image while the illumination control unit 101 is operating according to the first to fourth modes. Hereinafter, an image generated by the imaging control unit 102 causing the imaging unit 29 to capture an image of the pallet and the pallet hole in the pallet while the headlight 28 is operating according to the first mode and causing the first light source 28a to illuminate the pallet is referred to as a first captured image P1. Furthermore, an image generated by the imaging control unit 102 causing the imaging unit 29 to capture an image of the pallet and the pallet hole in the pallet while the headlight 28 is operating according to the second mode and causing the second light source 28b to illuminate the pallet is referred to as a second captured image P2. When the headlights 28 operate in the third mode and cause the first light source 28a and the second light source 28b to irradiate the pallet with light, the imaging control unit 102 causes the imaging unit 29 to capture an image of the pallet and the pallet holes in the pallet, and the generated image is referred to as a third captured image P3. When the headlights 28 operate in the fourth mode and cause the headlights 28 to be turned off, the imaging control unit 102 causes the imaging unit 29 to capture an image of the pallet and the pallet holes in the pallet, and the generated image is referred to as a fourth captured image P4. Hereinafter, when the first captured image P1 to the fourth captured image P4 are not to be distinguished from one another, they will simply be referred to as "captured images".

[0038] When it is determined that the position of the pallet cannot be identified, the identifying unit 103 may identify the position of the pallet based on a captured image generated by the imaging control unit 102 in the imaging unit 29 in a state in which the illumination control unit 101 operates the headlights 28 in another mode. Hereinafter, when the processing of the identifying unit 103 results in the position of the pallet being unable to be identified based on any of the first captured image P1 to the fourth captured image P4, the determining unit 104 comprehensively determines that the position of the pallet cannot be identified.

[0039] When the determination unit 104 comprehensively determines that the position of the pallet cannot be identified, the output unit 105 outputs information indicating an instruction to execute abnormal operation processing to the control unit 27 of the forklift 20 via the communication unit 11. The control unit 27 causes the forklift 20 to execute abnormal operation processing based on the information acquired from the determination device 10. The abnormal operation processing includes, for example, stopping the power supply to the components of the forklift 20 related to automatic driving control and outputting an abnormal sound to notify the user of the abnormality.

[0040] [Operation of the determination device 10] Hereinafter, the operation of the determination device 10 will be described with reference to FIG. 4. The flowchart shown in FIG. 4 is executed, for example, when the forklift 20 loads a pallet onto the fork 22. First, the illumination control unit 101 controls the operation of the headlight 28 to a first mode (step S100). Next, the imaging control unit 102 causes the imaging unit 29 to capture an image of the pallet and generate a first captured image P1 (step S102). Next, the identification unit 103 executes a process of identifying the position of the pallet based on the first captured image P1 (step S104). Next, the determination unit 104 determines whether or not the position of the pallet has been identified by the identification process of step S104 (step S106). When it is determined by the determination unit 104 that the position of the pallet has been identified, the output unit 105 outputs information indicating the position of the pallet to the control unit 27 (step S108).

[0041] If the determination unit 104 determines in step S106 that the position of the pallet cannot be identified, the illumination control unit 101 controls the operation of the headlights 28 to the second mode (step S110). Next, the imaging control unit 102 causes the imaging unit 29 to capture an image of the pallet and generate a second captured image P2 (step S112). Next, the identification unit 103 executes a process of identifying the position of the pallet based on the second captured image P2 (step S114). Next, the determination unit 104 determines whether or not the position of the pallet has been identified by the identification process of step S114 (step S116). If the determination unit 104 determines that the position of the pallet has been identified, the process proceeds to step S108.

[0042] If the determination unit 104 determines in step S116 that the position of the pallet cannot be identified, the illumination control unit 101 controls the operation of the headlights 28 to the third mode (step S118). Next, the imaging control unit 102 causes the imaging unit 29 to capture an image of the pallet and generate a third captured image P3 (step S120). Next, the identification unit 103 executes a process of identifying the position of the pallet based on the third captured image P3 (step S122). Next, the determination unit 104 determines whether or not the position of the pallet has been identified by the identification process of step S122 (step S124). If the determination unit 104 determines that the position of the pallet has been identified, the process proceeds to step S108.

[0043] If the determination unit 104 determines in step S124 that the position of the pallet cannot be identified, the illumination control unit 101 controls the operation of the headlights 28 to a fourth mode (step S126). Next, the imaging control unit 102 causes the imaging unit 29 to capture an image of the pallet and generate a fourth captured image P4 (step S128). Next, the identification unit 103 executes a process of identifying the position of the pallet based on the fourth captured image P4 (step S130). Next, the determination unit 104 determines whether or not the position of the pallet has been identified by the identification process in step S130 (step S132). If the determination unit 104 determines that the position of the pallet has been identified, the process proceeds to step S108.

[0044] When the determination unit 104 determines in the processes of steps S106, S116, S124, and S132 that the position of the pallet cannot be identified, the output unit 105 assumes that the determination unit 104 has comprehensively determined that the position of the pallet cannot be identified. The output unit 105 outputs information indicating an instruction to execute abnormal operation processing to the control unit 27 via the communication unit 11 (step S134).

[0045] [Effects of the embodiment] According to the above embodiment, the following effects can be obtained. (1) The determination device 10 includes an illumination control unit 101, an imaging control unit 102, a specification unit 103, a determination unit 104, and an output unit 105. The illumination control unit 101 is a control unit that controls the headlight 28 that illuminates the pallet with light, and operates the headlight 28 in at least one of a first mode and a second mode different from the first mode. Specifically, the illumination control unit 101 operates the headlight 28 in any one of the first to fourth modes. The imaging control unit 102 captures an image of the pallet and a pallet hole in the pallet in a state in which the illumination control unit 101 operates the headlight 28 in the first mode, and controls the operation of the imaging unit 29 to generate a first captured image P1. The specification unit 103 specifies the position of the pallet based on the first captured image P1. The determination unit 104 determines whether or not the position of the pallet has been specified based on the first captured image P1. If the determination unit 104 determines that the position of the pallet has been identified, the output unit 105 outputs information indicating the position of the pallet. If the determination unit 104 determines that the position of the pallet cannot be identified based on the first captured image P1, the imaging control unit 102 controls the operation of the imaging unit 29 to capture images of the pallet and the pallet hole in the pallet in a state in which the illumination control unit 101 operates the headlights 28 in the second to fourth modes, and to generate second to fourth captured images P2 to P4. The identification unit 103 identifies the position of the pallet based on the second to fourth captured images P2 to P4. Furthermore, the determination unit 104 further determines whether or not the position of the pallet has been identified based on the second to fourth captured images P2 to P4.

[0046] Here, depending on the position where the pallet is placed and the surrounding environment, even if the pallet is captured, the image may not adequately show the characteristics of the pallet. In this embodiment, when the position of the pallet cannot be identified based on the first captured image P1, the illumination control unit 101 controls the operation of the headlights 28 in another manner. Therefore, the first captured image P1 and the second to fourth captured images P2 to P4 show different ways of showing the characteristics of the pallet in the images, and even if the identification unit 103 cannot identify the position of the pallet in one captured image, it may be possible to identify the position of the pallet in another captured image. With this configuration, the identification unit 103 can identify the position of the pallet regardless of the position where the pallet is placed and the surrounding environment.

[0047] (2) The headlight 28 has a plurality of light sources that irradiate light in different wavelength ranges. In a first mode, the illumination control unit 101 causes the first light source 28a to irradiate the pallet with light. In a second mode, the illumination control unit 101 causes the second light source 28b, which irradiates light with a narrower wavelength range than the first light source 28a, to irradiate the pallet with light.

[0048] According to this configuration, when the position of the palette cannot be identified based on the first captured image P1, the identification unit 103 irradiates the palette with light from the second light source 28b (e.g., blue light) and obtains a second captured image P2 showing the characteristics of the palette. Therefore, even if the position of the palette cannot be identified based on the first captured image P1, the position of the palette can be identified based on the second captured image P2.

[0049] (3) The identification unit 103 may identify the position of the pallet based on the trained model 201 and the first captured image P1 or the second captured image P2. The trained model 201 is generated by machine learning using a teacher image TP showing the pallet and the pallet hole and the position of the pallet in the teacher image TP as teacher data.

[0050] According to this configuration, the identification unit 103 can identify the position of the pallet using the trained model 201 that has been trained according to the position where the pallet is placed and the surrounding environment. (4) The teacher images TP include a first teacher image TP1, a second teacher image TP2, a third teacher image TP3, and a fourth teacher image TP4.

[0051] According to this configuration, the identification unit 103 can identify the position of the pallet using the trained model 201 that has been trained according to the position where the pallet is placed and the surrounding environment. (5) The forklift 20 includes the determination device 10 and a loading actuator 25b.

[0052] With this configuration, the forklift 20 can identify the location of the pallet and insert the forks 22 into the pallet holes regardless of the location of the pallet or the surrounding environment.

[0053] (6) The learning method of the trained model 201 is to train the learning model by machine learning using teacher images TP showing a pallet and a pallet hole in the pallet, and the position of the pallet in the teacher images TP as teacher data, and the teacher images TP include a first teacher image TP1, a second teacher image TP2, a third teacher image TP3, and a fourth teacher image TP4.

[0054] According to this configuration, the learning method can generate a trained model 201 that can identify the position of a pallet regardless of the position where the pallet is placed and the surrounding environment. Also, according to this configuration, the position of a pallet can be identified using the trained model 201 that has been trained according to the position where the pallet is placed and the surrounding environment.

[0055] The above-described embodiments may be modified as follows: The above-described embodiments and the following examples may be combined with each other as long as they are not technically inconsistent. The control unit 100 may further include a pallet hole position identifying unit that identifies the position of the pallet hole based on the position of the pallet identified by the identifying unit 103, for example. In this case, the forklift 20 includes a LIDAR (Light Detection and Ranging). The pallet hole position identifying unit causes the LIDAR to irradiate light near the position of the pallet identified by the identifying unit 103. The pallet hole position identifying unit measures the scattered light of the irradiated light. The pallet hole position identifying unit detects the distance to the target based on the time from light emission to light reception. The irradiated light may be, for example, a pulsed laser light. The pallet hole position identifying unit identifies the range where the distance to the target is far from the range where the light is irradiated as the position of the pallet hole. The output unit 105 outputs information indicating the position of the pallet hole to the control unit 27 instead of (or in addition to) the information on the position of the pallet. With this configuration, the control unit 27 can insert the fork 22 into the pallet hole based on the information indicating the position of the pallet hole.

[0056] The pallet hole position identification unit may identify the positions of the pallet holes using a trained model generated by machine learning using a teacher image showing the pallet holes and the positions of the pallet holes in the teacher image as training data. In this case, the trained model is a computation model that, when an image showing the pallet and the pallet holes is input, outputs information indicating coordinates indicating the positions of the pallet holes on a coordinate system with the imaging unit that captured the image as the origin.

[0057] Immediately before or immediately after step S108, the illumination control unit 101 may execute a process of controlling the operation of the headlight 28 to the first mode. With this configuration, even if the mode of the headlight 28 is controlled to a mode other than the first mode, the headlight 28 can be made to illuminate the light in the traveling direction of the forklift 20 when the forklift 20 moves to the pallet.

[0058] * If it is difficult to realize an environment such as the first teacher image TP1, the second teacher image TP2, the third teacher image TP3, and the fourth teacher image TP4 in the real environment, the teacher image TP may be, for example, an image reproduced in a virtual space.

[0059] The identification unit 103 may identify the position of the palette or the position of the palette hole by a method other than the method using the trained model 201. In this case, the identification unit 103 performs image processing on the first captured image P1 to the fourth captured image P4. The identification unit 103 identifies the position of the palette or the position of the palette hole by identifying feature points of the palette or the palette hole based on the image processing. The image processing is a process that makes it possible to identify the end points of the palette or the palette hole by, for example, edge detection or the like.

[0060] 4, the operation of the headlights 28 is controlled in the order of the first, second, third, and fourth modes, but this is not limited to this. The first process of steps S100 to S106, the second process of steps S110 to S116, the third process of steps S118 to S124, and the fourth process of steps S126 to S132 may be executed in any order. In this case, if the determination unit 104 determines that the position of the pallet cannot be identified in the determination process included in the process executed last among the first to fourth processes, the process proceeds to step S134.

[0061] In the flowchart shown in FIG. 4, the case where the identifying unit 103 identifies the position of the pallet based on the first captured image P1 to the fourth captured image P4 in which the operation of the headlight 28 is generated in the first to fourth modes has been described, but the present invention is not limited to this. The irradiation control unit 101 may, for example, control the operation of the headlight 28 to at least two modes among the first to fourth modes. For example, the irradiation control unit 101 may control the operation of the headlight 28 to only the first and second modes. In this case, the imaging control unit 102 instructs the imaging unit 29 to generate only the first captured image P1 in the first mode and the second captured image P2 in the second mode. The identifying unit 103 executes only two identifying processes, that is, the identifying process of the position of the pallet based on the first captured image P1 and the identifying process of the position of the pallet based on the second captured image P2. If the determining unit 104 determines that the position of the pallet cannot be identified by the two identifying processes, the determining unit 104 determines that the position of the pallet cannot be identified comprehensively.

[0062] The combination of the modes of the headlights 28 described above is an example, and is not limited thereto. The illumination control unit 101 may control the mode of the headlights 28 to at least two different modes among the first mode to the fourth mode, and the combination of modes is not limited to the first mode and the second mode, and may be any combination. In detail, the illumination control unit 101 controls the operation of the headlights 28 to the first mode and the third mode. The imaging control unit 102 generates the first captured image P1 in the first mode. The imaging control unit 102 generates the third captured image P3 in the third mode. In this case, the third mode is an example of the "second mode", and the third captured image P3 is an example of the "second captured image". The illumination control unit 101 also controls the operation of the headlights 28 to the first mode and the fourth mode. The imaging control unit 102 generates the first captured image P1 in the first mode. The imaging control unit 102 generates the fourth captured image P4 in the fourth mode. In this case, the fourth mode is an example of the "second mode", and the fourth captured image P4 is an example of the "second captured image". Also, the illumination control unit 101 controls the operation of the headlights 28 to the second mode and the third mode. The imaging control unit 102 generates the second captured image P2 in the second mode. The imaging control unit 102 generates the third captured image P3 in the third mode. In this case, the second mode is an example of the "first mode", the third mode is an example of the "second mode", the second captured image P2 is an example of the "first captured image", and the third captured image P3 is an example of the "second captured image". Also, the illumination control unit 101 controls the operation of the headlights 28 to the second mode and the fourth mode. The imaging control unit 102 generates the second captured image P2 in the second mode. The imaging control unit 102 generates the fourth captured image P4 in the fourth mode. In this case, the second mode is an example of the "first mode", the fourth mode is an example of the "second mode", the second captured image P2 is an example of the "first captured image", and the fourth captured image P4 is an example of the "second captured image". In addition, the illumination control unit 101 controls the operation of the headlight 28 to the third mode and the fourth mode. The imaging control unit 102 generates the third captured image P3 in the third mode. The imaging control unit 102 generates the fourth captured image P4 in the fourth mode.In this case, the third aspect is an example of the "first aspect", the fourth aspect is an example of the "second aspect", the third captured image P3 is an example of the "first captured image", and the fourth captured image P4 is an example of the "second captured image".

[0063] Alternatively, the illumination control unit 101 may control the operation of the headlights 28 to the "second mode" and then to the "first mode". In this case, the imaging control unit 102 generates a "second captured image" in the "second mode" and then generates a "first captured image" in the "first mode". Furthermore, the determination unit 104 determines whether the position of the pallet has been identified based on the "first captured image" after determining that the position of the pallet cannot be identified based on the "second captured image".

[0064] Although the second light source 28b is realized by blue light in the above description, this is not limited thereto. The second light source 28b may be realized by any light source that emits light having a narrower wavelength range than the light emitted by the first light source 28a. In addition, the first light source 28a may be realized by a light other than white light as long as the light source emits light having a wider wavelength range than the light emitted by the second light source 28b.

[0065] Although the first light source 28a and the second light source 28b are described as being light sources that emit light in different wavelength ranges, the present invention is not limited to this. The first light source 28a and the second light source 28b may be light sources that emit light with different luminances. In this case, the second light source 28b may be realized by any light source that emits light with a lower luminance than the light emitted by the first light source 28a. Furthermore, the first light source 28a may be realized by any light source that emits light with a higher luminance than the light emitted by the second light source 28b.

[0066] The first teacher image TP1 may be, for example, an image of the pallet being illuminated with light from the first light source 28a, or may be an image of the pallet being illuminated with light from a third light source having the same function as the first light source 28a. The second teacher image TP2 may be, for example, an image of the pallet being illuminated with light from the second light source 28b, or may be an image of the pallet being illuminated with light from a fourth light source having the same function as the second light source 28b.

[0067] Although the fourth light source is described as being realized by blue light, this is not limited thereto. The fourth light source may be realized by any light source that emits light having a narrower wavelength range than the light emitted by the third light source. In addition, the third light source may be realized by a light other than white light as long as the light source emits light having a wider wavelength range than the light emitted by the fourth light source.

[0068] Although the third light source and the fourth light source are described as being light sources that irradiate light in different wavelength ranges, the present invention is not limited to this. The third light source and the fourth light source may be light sources that irradiate light with different luminances. In this case, the fourth light source may be realized by any light source that irradiates light with a lower luminance than the light irradiated by the third light source. Also, the third light source may be realized by any light source that irradiates light with a higher luminance than the light irradiated by the fourth light source.

[0069] ○ The first captured image P1 to the fourth captured image P4 may each be realized by a composite image obtained by combining a plurality of captured images. In this case, the imaging control unit 102 generates a plurality of captured images while changing the settings related to imaging, for example, in a state in which the headlights 28 are controlled in each state. Then, the imaging control unit 102 combines the captured images generated in a state in which the headlights 28 are controlled in each state, and generates the first captured image P1 to the fourth captured image P4. In detail, the imaging control unit 102 generates several captured images with different focus positions in a state in which the headlights 28 are controlled in the first state. Then, the imaging control unit 102 combines the several captured images generated to generate the first captured image P1. As a result, the first captured image P1 captures the imaging target shown in the first captured image P1 more clearly than when the first captured image P1 is realized by one image. Therefore, the identification unit 103 can identify the position of the palette based on the clearer captured image, and therefore the accuracy of identifying the position of the palette can be improved.

[0070] In the above, the determination device 10 and the forklift 20 are realized as separate entities, but this is not limited thereto. The determination device 10 and the forklift 20 may be configured as an integrated unit. In this case, the control unit 27 includes the functional units included in the control unit 100. The forklift 20 also includes the storage unit 200. When the determination device 10 and the forklift 20 are configured as an integrated unit, the forklift 20 does not need to include the communication unit 11 or the communication unit 26 as components used for transmitting and receiving information between the control unit 27 and the determination device 10. [Explanation of symbols]

[0071] 10...judgment device, 20...forklift, 21...wheel, 22...fork, 23...handle, 24...operating lever, 25a...traveling actuator, 25b...loading actuator, 11,26...communication unit, 27,100...control unit, 28...headlight, 28a...first light source, 28b...second light source, 29...imaging unit, 101...illumination control unit, 102...imaging control unit, 103...identification unit, 104...judgment unit, 105...output unit, 200...memory unit, 201...learned model, P1...first captured image, P2...second captured image, P3...third captured image, P4...fourth captured image, TP...teacher image, TP1...first teacher image, TP2...second teacher image, TP3...third teacher image, TP4...fourth teacher image.

Claims

1. an illumination control unit that controls an illumination unit that illuminates a pallet on which an object to be handled by a forklift is placed, the illumination control unit operating the illumination unit in at least one of a first mode and a second mode different from the first mode; an imaging control unit that captures an image of the pallet and a pallet hole in the pallet in a state in which the irradiation control unit operates the irradiation unit in the first mode, and controls an operation of the imaging unit to generate a first captured image; An identification unit that identifies a position of the pallet based on the first captured image; a determination unit that determines whether or not a position of the pallet has been identified based on the first captured image; an output unit that outputs information indicating the position of the pallet when the determination unit determines that the position of the pallet has been identified, When the determination unit determines that the position of the pallet cannot be identified based on the first captured image, the imaging control unit images the pallet and a pallet hole in the pallet in a state in which the irradiation control unit operates the irradiation unit in the second mode, and controls the operation of the imaging unit to generate a second captured image, the identification unit identifies the position of the pallet based on the second captured image, and the determination unit further determines whether or not the position of the pallet has been identified based on the second captured image. A determination device characterized by:

2. The irradiation unit has a plurality of light sources that irradiate light of different wavelength ranges, In the first aspect, the irradiation control unit causes a first light source to irradiate light onto the pallet, and in the second aspect, causes a second light source that irradiates light having a narrower wavelength range than the first light source to irradiate light onto the pallet. The determination device according to claim 1 .

3. The light from the second light source includes blue light. The determination device according to claim 2 .

4. The identification unit identifies the position of the pallet based on a trained model generated by machine learning using a teacher image showing the pallet and the pallet hole and the position of the pallet in the teacher image as teacher data, and based on the first captured image or the second captured image. The determination device according to claim 1 .

5. The teacher image includes: At least one of an image of a state in which the pallet is irradiated with light from a third light source, an image of a state in which the pallet is irradiated with light from a fourth light source that irradiates light with a narrower wavelength range than that of the third light source, an image of a state in which the pallet is irradiated with natural light, and an image of a state in which a shadow of an object existing around the forklift is cast on the pallet is included. The determination device according to claim 4.

6. a pallet hole position identifying unit that identifies a position of the pallet hole based on the position of the pallet identified by the identifying unit, The output unit outputs information indicating the position of the pallet hole. The determination device according to claim 1 .

7. A determination device according to any one of claims 1 to 6, and a drive control unit that aligns the position of the forks of the forklift with the pallet holes based on the position of the pallet. A forklift truck characterized by:

8. A learning model is trained by machine learning using a teacher image showing a pallet on which an object to be loaded by a forklift is placed, a pallet hole formed in the pallet, and the position of the pallet in the teacher image as teacher data; The teacher image includes at least one of an image in which the pallet is irradiated with light from a third light source, an image in which the pallet is irradiated with light from a fourth light source that irradiates light with a narrower wavelength range than the third light source, an image in which the pallet is irradiated with natural light, and an image in which a shadow of an object existing around the forklift is cast on the pallet. A learning method comprising:

Images