Intelligent forklift control system based on Internet of Things equipment

Abstract

The invention discloses a forklift intelligent control system based on Internet of Things equipment, and relates to the field of forklift control. Comprising an Internet of Things sensor module which is installed on a forklift body; the virtual driving cabin comprises a driving environment acquisition and display module and a driving intention recognition module; the cloud server is in communication connection with the Internet of Things sensor module and the virtual cab, the cloud server comprises a data storage module, a central calculation module and a control module, and the control module is in control connection with a controller of the forklift and an environment simulation module of the virtual cab. The driving intention recognition module is a combination of an electronic simulation control lever and a control pedal or a gesture recognition camera. The cloud server is used as a medium, the internet of things sensor module is arranged on the forklift body, meanwhile, the virtual cab is arranged, and therefore the remote control function of the forklift can be achieved on the basis of AR.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of forklift control, and particularly relates to a forklift intelligent control system based on Internet of Things equipment. BACKGROUND

[0002] Forklifts are widely used in factories, warehouses, stations, ports and other places, and are mainly used for loading and unloading, transporting, stacking, unstacking and short-distance transportation of goods.

[0003] The forklift uses an electric motor or a fuel engine as a power source and uses hydraulic pressure as a medium to realize the functions of transporting and transporting goods. The forklift is controlled by a joystick and a control pedal in the cockpit to control movement and the lifting of the arm.

[0004] In the prior art, the control system of the forklift is mostly integrated into the controller of the forklift itself, and the driver needs to control it in the cockpit of the forklift. It is not suitable for some work environments with poor working conditions or the possibility of safety accidents.

[0005] Therefore, the present application provides a forklift intelligent control system based on Internet of Things equipment. SUMMARY

[0006] The present application aims to solve the problems in the prior art and provides a forklift intelligent control system based on Internet of Things equipment.

[0007] In order to achieve the above-mentioned purpose, the present application adopts the following technical solutions: A forklift intelligent control system based on Internet of Things equipment, comprising: An Internet of Things sensor module, which is installed on the forklift body; A virtual cockpit, which includes a driving environment acquisition and display module and a driving intention recognition module; A cloud server, which is communicatively connected to the Internet of Things sensor module and the virtual cockpit, and includes a data storage module, a central computing module and a control module, wherein the control module is connected to the controller of the forklift and the environment simulation module of the virtual cockpit.

[0008] Preferably, the Internet of Things sensor module comprises: An environmental camera, which is arranged in multiple and uniformly installed around the forklift body, and acquires video images of the forklift working; A laser radar, which is arranged in multiple and uniformly installed around the forklift body, and is used to acquire data of other objects in the forklift working environment and the forklift, and generate a three-dimensional environment map with distance indication in combination with the image of the environmental camera; A three-axis gyroscope is installed on the forklift body, the three-axis gyroscope senses the three-axis displacement state of the forklift, and the initial attitude of the forklift during installation is combined to obtain the current attitude of the forklift.

[0009] Preferably, the environment acquisition display module is a VR glasses.

[0010] Preferably, the driving intention recognition module is a combination of an electronic simulation joystick and a control pedal or a gesture recognition camera.

[0011] Preferably, the working steps of the forklift intelligent control system include the following steps: S1: The environment camera, laser radar and three-axis gyroscope in the Internet of Things sensor module cooperate to acquire a three-dimensional environment map with distance indication and the position and attitude information of the forklift relative to the map; S2: The cloud server acquires the map information, and after converting the map information into a three-dimensional panoramic map through a central computing module, transmits the three-dimensional panoramic map to a virtual cockpit, and the remote control driver of the virtual cockpit acquires the panoramic information of the map through VR glasses; S3: The remote control driver issues control instructions under the cooperation of the panoramic map information and the task demand to be operated, the driving intention recognition module acquires the control instructions, and transmits the acquired control instructions to the cloud server; S4: The cloud server controls the controller of the forklift through the control module, so that the forklift executes according to the control instructions; S5: During the system operation, the data storage module in the cloud server stores the system operation data.

[0012] Preferably, in the S3 step, when the driving intention recognition module recognizes gestures, the logic is: S31: According to the type of forklift that can be controlled, all forklift control instructions are collected; S32: For each collected instruction, an instruction gesture corresponding to the instruction is matched; S33: The gesture recognition camera acquires the gesture image of the driver, matches the gesture information with the instruction gesture, and obtains the input instruction.

[0013] Preferably, in the S33 step, the matching of the gesture information and the instruction gesture has a single-hand single-mode control mode, a double-hand single-mode control mode and a double-hand double-mode control mode.

[0014] Preferably, the logic of the single-hand single-mode control mode is: A1: The driver inputs a gesture with one hand; A2: The gesture recognition camera acquires the single-hand gesture image of the driver, extracts the gesture features, and acquires the state map of the hand; A3: compare the acquired hand state diagram with the gesture diagram in the instruction gesture, and take the result with the highest similarity as the output.

[0015] Preferably, the logic of the double-hand single-mode control mode is: B1: the driver inputs gestures with both hands; B2: the gesture recognition camera independently acquires gesture images of the driver's two hands respectively, extracts gesture features, and acquires state diagrams of the two hands; B3: compare the state diagrams of the two hands with the instruction gesture respectively, obtain two similarity results S1 and S2, and then calculate the final output similarity S according to the formula Preferably, the logic of the double-hand single-mode control mode is: In the B3 step, k1 and k2 are flexible weights of the two hands, which are input by the driver according to the flexibility of his own hands.

[0016] Preferably, the logic of the double-hand single-mode control mode is: C1: the driver inputs gestures with both hands; C2: the gesture recognition camera independently acquires gesture images of the driver's two hands respectively, extracts gesture features, and acquires state diagrams of the two hands; C3: compare one of the hand state diagrams with the instruction gesture, take the result with the highest similarity as the output instruction one, and then compare the other hand state diagram with the instruction gesture, take the result with the highest similarity as the output instruction two; C4: if the two instructions do not conflict with each other, input the two instructions to the controller of the forklift at the same time, and the forklift executes the two instructions at the same time; if the two instructions conflict with each other, compare the similarity of the two instructions with the instruction gesture, and take the one with the higher similarity as the final output instruction.

[0017] The beneficial effects of the present application are: The present application can realize remote control of the forklift based on AR by taking the cloud server as a medium, setting the Internet of Things sensor module on the forklift body, and setting a virtual cockpit.

[0018] The present application can form a three-dimensional environment map with distance indication and vehicle body posture position by combining the Internet of Things sensor module with the environment camera, laser radar and three-axis gyroscope, thereby increasing the driver's perception accuracy of the environment during remote control and increasing the control accuracy.

[0019] The present application adopts gesture control for driver intention acquisition, and adopts single-hand single-mode control mode, double-hand single-mode control mode and double-hand double-mode control mode based on gesture recognition, thereby making targeted settings according to the driver's own needs and increasing the control effect. BRIEF DESCRIPTION OF DRAWINGS

[0020] Figure 1 A fork truck intelligent control system architecture based on Internet of Things devices is proposed for the present application; Figure 2 A fork truck intelligent control system working logic diagram based on Internet of Things devices is proposed for the present application. DETAILED DESCRIPTION

[0021] The technical solutions of the present application will be further described in detail below in combination with specific embodiments.

[0022] In the description of the present application, it should be noted that unless otherwise explicitly specified and limited, the terms "mounting", "connection", "connecting", "setting" should be understood broadly, for example, it can be fixedly connected, set, or detachably connected, set, or integrally connected, set. For those skilled in the art, the specific meanings of the above terms in the present application can be understood according to the specific circumstances.

[0023] Example 1: A fork truck intelligent control system based on Internet of Things devices, comprising: An Internet of Things sensor module, which is mounted on the body of the fork truck; A virtual cockpit, which comprises a driving environment acquisition and display module and a driving intention recognition module; A cloud server, which is communicatively connected to the Internet of Things sensor module and the virtual cockpit, and comprises a data storage module, a central computing module and a control module, the control module being connected to the controller of the fork truck and the environment simulation module of the virtual cockpit.

[0024] The Internet of Things sensor module comprises: An environmental camera, which is arranged in multiple and uniformly mounted around the body of the fork truck, and acquires video images of the work of the fork truck; A laser radar, which is arranged in multiple and uniformly mounted around the body of the fork truck, and is used to acquire data of other objects in the working environment of the fork truck and the fork truck, and to generate a three-dimensional environment map with distance indication in combination with the images of the environmental camera; A three-axis gyroscope, which is mounted on the body of the fork truck, and senses the three-axis displacement state of the fork truck, and obtains the current attitude of the fork truck in combination with the initial attitude of the fork truck when it is installed.

[0025] The environment acquisition and display module is a VR glasses.

[0026] The driving intention recognition module is a combination of an electronic simulation joystick and a control pedal, or a gesture recognition camera.

[0027] Embodiment 2: The forklift intelligent control system based on the Internet of Things device, the working steps of the forklift intelligent control system based on the Internet of Things device include the following steps: S1: The environment camera, laser radar and three-axis gyroscope in the Internet of Things sensor module cooperate to obtain a three-dimensional environment map with distance indication and the position and attitude information of the forklift relative to the map; S2: The cloud server obtains the map information, and transmits the map information to the virtual cockpit after the map information is converted into a three-dimensional panoramic map by the central computing module, and the remote control driver of the virtual cockpit obtains the panoramic information of the map through the VR glasses; S3: The remote control driver issues a control instruction under the cooperation of the panoramic map information and the task demand to be operated, the driving intention recognition module obtains the control instruction, and transmits the obtained control instruction to the cloud server; S4: The cloud server controls the controller of the forklift through the control module, so that the forklift executes according to the control instruction; S5: During the system operation, the data storage module in the cloud server stores the system operation data.

[0028] In the S3 step, when the driving intention recognition module recognizes gestures, the logic is as follows: S31: According to the type of the forklift that can be controlled, all forklift control instructions (such as forward, backward, lift, lower, left turn, right turn) are collected; S32: For each collected instruction, an instruction gesture corresponding to the instruction is matched; S33: The gesture recognition camera obtains the gesture image of the driver, matches the gesture information with the instruction gesture, and obtains the input instruction.

[0029] In the S33 step, the matching of the gesture information and the instruction gesture is a single-hand single-mode control mode.

[0030] The logic of the single-hand single-mode control mode is as follows: A1: The driver inputs a gesture with one hand; A2: The gesture recognition camera obtains the single-hand gesture image of the driver, extracts the gesture features, and obtains the state map of the hand; A3: The obtained hand state map is compared with the gesture map in the instruction gesture, and the result with the highest similarity is taken as the output.

[0031] Embodiment 3: The forklift intelligent control system based on the Internet of Things device, the working steps of the forklift intelligent control system based on the Internet of Things device include the following steps: S1: The environmental camera, laser radar and three-axis gyroscope in the Internet of Things sensor module cooperate to obtain a three-dimensional environmental map with distance indication and the position and attitude information of the forklift relative to the map; S2: The cloud server obtains the map information and transmits it to the virtual cockpit after converting it into a three-dimensional panoramic map by the central computing module. The remote control driver of the virtual cockpit obtains the panoramic information of the map through the VR glasses. S3: The remote control driver issues control instructions based on the panoramic map information and the task requirements to be operated. The driving intention recognition module obtains the control instructions and transmits them to the cloud server. S4: The cloud server controls the controller of the forklift through the control module, so that the forklift executes according to the control instructions. S5: During the operation of the system, the data storage module in the cloud server stores the system operation data.

[0032] In the S3 step, when the driving intention recognition module recognizes gestures, the logic is as follows: S31: Collect all forklift control instructions (such as forward, backward, lift, lower, left turn, right turn) according to the types that can be controlled by the forklift. S32: Match each collected instruction with a corresponding instruction gesture. S33: The gesture recognition camera obtains the gesture image of the driver, matches the gesture information with the instruction gesture, and obtains the input instruction.

[0033] In the S33 step, the matching of gesture information and instruction gesture is a double-hand single-mode control mode.

[0034] The logic of the double-hand single-mode control mode is as follows: B1: The driver inputs gestures with both hands. B2: The gesture recognition camera independently obtains the gesture image of the driver's two hands and extracts the gesture features to obtain the state map of the two hands. B3: Compare the state map of the two hands with the instruction gesture respectively to obtain two similarity results S1 and S2, and then calculate the final output similarity S according to the formula Then take the result with the highest similarity as the output.

[0035] In the B3 step, k1 and k2 are the flexibility weights of the two hands, which are input by the driver according to their own flexibility.

[0036] Embodiment 4: A forklift intelligent control system based on Internet of Things devices, which makes the following improvements based on Embodiment 1: Embodiment 5: A forklift intelligent control system based on Internet of Things devices.

[0037] The working steps of the forklift intelligent control system include the following steps: S1: The environmental camera, laser radar and three-axis gyroscope in the Internet of Things sensor module cooperate to obtain a three-dimensional environmental map with distance indication and the position and attitude information of the forklift relative to the map; S2: The cloud server obtains the map information, and after converting the map information into a three-dimensional panoramic map through a central computing module, transmits the three-dimensional panoramic map to a virtual cockpit, and the remote control driver of the virtual cockpit obtains the panoramic information of the map through VR glasses; S3: The remote control driver issues a control instruction based on the panoramic map information and the task demand to be operated, the driving intention recognition module obtains the control instruction, and transmits the obtained control instruction to the cloud server; S4: The cloud server controls the controller of the forklift through the control module, so that the forklift executes according to the control instruction; S5: During the system operation, the data storage module in the cloud server stores the system operation data.

[0038] In the S3 step, when the driving intention recognition module recognizes gestures, the logic is as follows: S31: According to the types of forklifts that can be controlled, all forklift control instructions (such as forward, backward, lift, lower, left turn, right turn) are collected; S32: For each collected instruction, match an instruction gesture corresponding to it; S33: The gesture recognition camera obtains the gesture image of the driver, matches the gesture information with the instruction gesture, and obtains the input instruction.

[0039] In the S33 step, the matching of gesture information and instruction gesture is a double-hand double-mode control mode.

[0040] The logic of the double-hand double-mode control mode is as follows: C1: The driver inputs gestures with both hands; C2: The gesture recognition camera independently obtains the gesture image of the driver's two hands, respectively, and extracts gesture features to obtain the state map of two hands; C3: Compare one of the hand state maps with the instruction gesture to output instruction one with the highest similarity, and then compare the other hand state map with the instruction gesture to output instruction two with the highest similarity; C4: If two instructions do not conflict with each other, two instructions are input to the controller of the forklift at the same time, the forklift executes two instructions (such as forward and rising, left turn and rising, etc.) at the same time, if two instructions conflict with each other (such as rising and falling, forward and backward, left turn and right turn), the similarity of the two instructions and the instruction gesture is compared, and the instruction with the largest similarity is output as the final instruction.

[0041] The forklift remote control function can be realized based on AR by taking the cloud server as a medium, setting the Internet of Things sensor module on the forklift body, and setting a virtual cockpit.

[0042] The Internet of Things sensor module is set to an environmental camera, a laser radar and a three-axis gyroscope, which can be combined to form a three-dimensional environmental map with distance indication and vehicle body posture position, so that the perception accuracy of the driver to the environment during remote control is increased, and the control accuracy is increased.

[0043] In addition, gesture control is adopted for driver intention acquisition, and three control modes of single-hand single-mode control mode, double-hand single-mode control mode and double-hand double-mode control mode are adopted on the basis of gesture recognition, so that targeted setting is made according to the needs of the driver, and the control effect is increased.

[0044] The above is only a preferred specific embodiment of the present application, but the protection scope of the present application is not limited to this, any person skilled in the art can make equivalent replacement or change according to the technical scheme and the inventive concept of the present application within the technical range disclosed by the present application, which should be covered in the protection scope of the present application.

Claims

1. A forklift intelligent control system based on Internet of Things equipment, characterized in that, The application relates to a forklift intelligent control system. The forklift intelligent control system comprises an Internet of Things sensor module, a virtual cockpit, and a cloud server. The Internet of Things sensor module is installed on a forklift body. The virtual cockpit comprises a driving environment acquisition display module and a driving intention recognition module.

2. The forklift intelligent control system based on Internet of Things devices of claim 1, wherein, The cloud server is communicatively connected to the Internet of Things sensor module and the virtual cockpit, and comprises a data storage module, a central computing module, and a control module. The control module is connected to a controller of the forklift and an environment simulation module of the virtual cockpit. The Internet of Things sensor module comprises: environmental cameras arranged around the forklift body and used to acquire video images of the forklift working environment; 3. The forklift intelligent control system based on Internet of Things devices of claim 1, wherein, laser radars arranged around the forklift body and used to acquire data of other objects in the forklift working environment and the forklift, and generate a three-dimensional environment map with distance indication in combination with the images of the environmental cameras; 4. The forklift intelligent control system based on the Internet of Things device of claim 1, wherein, a three-axis gyroscope installed on the forklift body and used to sense three-axis displacement states of the forklift and obtain a current attitude of the forklift in combination with an initial attitude of the forklift during installation.

5. The forklift intelligent control system based on the Internet of Things device according to any one of claims 1-4, characterized in that, The environment acquisition display module is a VR (Virtual Reality) glasses. The driving intention recognition module is a combination of an electronic simulation joystick and a control pedal or a gesture recognition camera. The working steps of the forklift intelligent control system comprise the following steps: S1: the environmental cameras, the laser radars, and the three-axis gyroscope in the Internet of Things sensor module cooperate to acquire a three-dimensional environment map with distance indication and position and attitude information of the forklift relative to the map; S2: the cloud server acquires the map information, converts the map information into a three-dimensional panoramic map through the central computing module, and transmits the three-dimensional panoramic map to the virtual cockpit, so that a remote control driver of the virtual cockpit can acquire panoramic information of the map through the VR glasses; S3: the remote control driver issues a control instruction in combination with the panoramic map information and a task demand to be operated, the driving intention recognition module acquires the control instruction, and transmits the acquired control instruction to the cloud server; 6. The forklift intelligent control system based on an Internet of Things device according to claim 5, characterized in that, S4: the cloud server controls the controller of the forklift through the control module, so that the forklift executes the control instruction; S5: during the system operation, the data storage module in the cloud server stores system operation data. In the S3 step, when the driving intention recognition module recognizes gestures, the logic is as follows: S31: according to the types of the forklift that can be controlled, all forklift control instructions are collected; 7. The forklift intelligent control system based on Internet of Things devices of claim 6, wherein, S32: for each collected instruction, a corresponding instruction gesture is matched; 8. The forklift intelligent control system based on an Internet of Things device according to claim 7, characterized in that, S33: a gesture recognition camera acquires gesture images of the driver, matches the gesture information with the instruction gesture, and obtains an input instruction. In the S33 step, the matching of the gesture information with the instruction gesture has a single-hand single-mode control mode, a double-hand single-mode control mode, and a double-hand double-mode control mode. The logic of the single-hand single-mode control mode is as follows: A1: the driver inputs a gesture with a single hand; 9. The forklift intelligent control system based on the Internet of Things device of claim 7, wherein, A2: the gesture recognition camera acquires a single-hand gesture image of the driver, extracts gesture features, and acquires a state map of the hand; A3: the acquired state map of the hand is compared with gesture maps in the instruction gesture, and the result with the highest similarity is taken as an output. The logic of the double-hand single-mode control mode is as follows: B1: Driver's both hands input gesture; B2: Gesture recognition camera respectively independently acquires driver's both hands gesture image, and extracts gesture feature, acquires two hand state diagram; B3: compare the state diagram of the two hands with the instruction gesture respectively, obtain two similarity results S1 and S2, and then according to the formula Calculate the final output similarity S, and then take the result with the highest similarity as the output; In the B3 step, k1 and k2 are the flexible weights of both hands, which are input by the driver according to the flexibility of his own hands.

10. The forklift intelligent control system based on an Internet of Things device of claim 7, wherein, The logic of the double-hand double-mode control mode is as follows: C1: Driver's both hands input gesture; C2: Gesture recognition camera respectively independently acquires driver's both hands gesture image, and extracts gesture feature, acquires two hand state diagram; C3: Compare one of the hand state diagrams with the instruction gesture, output instruction one with the highest similarity, and then compare the other hand state diagram with the instruction gesture, output instruction two with the highest similarity; C4: If the two instructions do not conflict with each other, input two instructions to the forklift controller at the same time, and the forklift executes the two instructions at the same time, if the two instructions conflict with each other, compare the similarity of the two instructions and the instruction gesture, and output the final instruction with the highest similarity.

Images