Forklift real-time map building and positioning method based on machine vision technology

Abstract

The invention relates to the technical field of forklift positioning, in particular to a forklift real-time map building and positioning method based on a machine vision technology, and the method comprises the steps: collecting three-dimensional modeling data in a factory, and building a three-dimensional visual model of a factory environment; key nodes in the three-dimensional visualization model are extracted, and a topology flow direction association graph of the factory environment is constructed; the real-time movement state of the forklift is obtained, and the real-time movement state of the forklift is mapped to the topology flow direction association diagram of the factory environment; according to the real-time movement state of the forklift, the actual behavior mode probability distribution of the forklift is recognized, and the similarity between the actual behavior mode probability distribution of the forklift and the preset behavior mode probability distribution of the forklift is calculated; and when the similarity is lower than a similarity threshold value, forklift early warning is given to a remote management terminal.

Description

Background Art

[0002] With the rapid development of the logistics industry, forklifts play a vital role in warehousing and logistics. However, the traditional forklift management method has many problems, such as inaccurate positioning, irregular management, low efficiency, etc. These problems seriously affect the operational efficiency and competitiveness of enterprises. Therefore, in order to improve the management level and work efficiency of forklifts, forklift positioning technology came into being.

[0003] Forklift positioning technology is widely used in logistics, manufacturing, warehousing and other industries. Through forklift positioning technology, enterprises can achieve real-time tracking and positioning of forklifts, improve the accuracy and efficiency of warehouse management. At the same time, forklift positioning technology can also help enterprises optimize forklift scheduling and management processes, reduce forklift resource waste and maintenance costs, and improve the company's operational efficiency and market competitiveness.

[0004] Problems in the existing technology: When positioning a forklift, there are currently few methods that can analyze the behavior pattern of the forklift while positioning the forklift, and obtain the gap between the forklift behavior pattern and the ideal behavior pattern for early warning management. Summary of the invention

[0005] The main purpose of the present invention is to provide a method for building and positioning a forklift real-time map based on machine vision technology, by collecting three-dimensional modeling data in the factory, establishing a three-dimensional visualization model of the factory environment; extracting key nodes in the three-dimensional visualization model, and constructing a topological flow association diagram of the factory environment; obtaining the real-time motion state of the forklift, and mapping the real-time motion state of the forklift to the topological flow association diagram of the factory environment; according to the real-time motion state of the forklift, identifying the probability distribution of the actual behavior pattern of the forklift, and calculating the similarity between the probability distribution of the actual behavior pattern of the forklift and the probability distribution of the preset behavior pattern of the forklift; and presenting a forklift warning to the remote management terminal when the similarity is lower than the similarity threshold. Based on analyzing the behavior pattern of the forklift and obtaining the gap between the behavior pattern of the forklift and the ideal behavior pattern for early warning management, the above-mentioned problems mentioned in the background technology are effectively solved.

[0006] The technical solution of the present invention is as follows:

[0007] In the first aspect, a forklift real-time map construction and positioning method based on machine vision technology is proposed, and the method comprises the following steps:

[0008] S1. Collect 3D modeling data in the factory and build a 3D visualization model of the factory environment;

[0009] S2, extract key nodes in the 3D visualization model and construct a topological flow association diagram of the factory environment;

[0010] S3, obtaining the real-time motion state of the forklift, and mapping the real-time motion state of the forklift to a topological flow association diagram of the factory environment;

[0011] S4. According to the real-time motion state of the forklift, identify the actual behavior pattern probability distribution of the forklift, and calculate the similarity between the actual behavior pattern probability distribution of the forklift and the preset behavior pattern probability distribution of the forklift;

[0012] S5. When the similarity is lower than the similarity threshold, a forklift warning is issued to the remote management terminal.

[0013] A further improvement of the present invention is that S1 comprises the following specific steps:

[0014] S11. Arrange a 3D laser scanning station inside the factory according to the factory layout and scanning plan, and use a high-precision 3D laser scanner to scan inside and outside the factory to obtain the first point cloud data;

[0015] S12, using a total station or a precision level to measure the elevation control points between the scanning sites, and introducing the elevation data into the first point cloud data, correcting the first point cloud data in the vertical direction, outputting the second point cloud data, performing splicing, denoising, and registration processing on the second point cloud data, and outputting a three-dimensional point cloud model of the factory;

[0016] S13. Obtain the factory's construction drawings, equipment layout diagrams, process flow diagrams, and material storage diagrams, and use Revit software to build a three-dimensional model of the factory's buildings and equipment components;

[0017] S14. Use GPS positioning to obtain the GCJ-02 coordinates of factory buildings and equipment components, and integrate the three-dimensional point cloud model, three-dimensional model, and GCJ-02 coordinates into a complete three-dimensional visualization model of the factory.

[0018] A further improvement of the present invention is that the specific content of S13 is: constructing a three-dimensional model of the factory building and equipment components according to the factory's construction drawings, equipment layout diagrams, process flow diagrams and material storage diagrams, and assigning material properties and structural parameter information to the factory building and equipment components, while adding equipment number, manufacturer, and maintenance record information to each equipment component, and exporting the three-dimensional model to an ifc format file through the IFC data exchange standard.

[0019] A further improvement of the present invention is that S2 comprises the following specific steps:

[0020] S21, extracting key nodes in the three-dimensional visualization model, wherein the key nodes include a forklift charging station, a material storage area, and a logistics channel entrance and exit, and creating a key node set A; the key node set A includes the GCJ-02 coordinates of the key nodes of the factory environment;

[0021] S22. Determine the actual logistics path according to the process flow chart and material storage diagram of the factory, use the actual logistics path as the movement path of the forklift, and create a directed edge set B; the directed edge set B includes the movement path of the forklift along the key nodes of the factory environment.

[0022] S23, construct a directed graph C = (A, B), the directed graph is a topological flow association graph of the factory environment.

[0023] A further improvement of the present invention is that the real-time motion state of the forklift in S3 includes: the real-time GCJ-02 coordinates, real-time acceleration, real-time steering wheel angle, and real-time speed direction of the forklift.

[0024] A further improvement of the present invention is that S4 comprises:

[0025] S41, combining the real-time GCJ-02 coordinates, real-time acceleration, real-time steering wheel angle, and real-time speed direction of the forklift into a feature vector form;

[0026] S42. Input the set of feature vectors into a machine learning model, wherein the machine learning model takes the behavior pattern probability predicted by each set of feature vectors as output, takes the actual behavior pattern probability corresponding to each set of feature vectors as a prediction target, and takes minimizing the sum of the prediction accuracies of all predicted behavior pattern probabilities as a training target, and stops training when the sum of the prediction accuracies converges, and outputs the actual behavior pattern probability distribution of the forklift.

[0027] A further improvement of the present invention is that the behavior mode in S42 includes stationary, uniform motion, turning, acceleration to braking, and braking to acceleration; and the machine learning model is any one of a random forest, a support vector machine and a neural network model.

[0028] A further improvement of the present invention is that the S4 further includes:

[0029] S43, according to the real-time motion state of the forklift, setting the preset probability of each behavior mode of the forklift, and forming a probability distribution of the preset behavior mode of the forklift;

[0030] S44, calculating the similarity between the actual behavior pattern probability distribution of the forklift and the preset behavior pattern probability distribution of the forklift, the similarity calculation formula is: Among them, a i is the actual behavior pattern probability of the forklift, b i is the probability of the preset behavior pattern of the forklift.

[0031] The technical effects of the present invention are as follows:

[0032] A method for building and locating a forklift real-time map based on machine vision technology is constructed. By collecting three-dimensional modeling data in the factory, a three-dimensional visualization model of the factory environment is established; key nodes in the three-dimensional visualization model are extracted to construct a topological flow association diagram of the factory environment; the real-time motion state of the forklift is obtained, and the real-time motion state of the forklift is mapped to the topological flow association diagram of the factory environment; according to the real-time motion state of the forklift, the probability distribution of the actual behavior pattern of the forklift is identified, and the similarity between the probability distribution of the actual behavior pattern of the forklift and the probability distribution of the preset behavior pattern of the forklift is calculated; when the similarity is lower than the similarity threshold, a forklift warning is issued to the remote management terminal. Based on analyzing the behavior pattern of the forklift and obtaining the gap between the behavior pattern of the forklift and the ideal behavior pattern for early warning management, the above-mentioned problems mentioned in the background technology are effectively solved. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Other features, objects and advantages of the present invention will become more apparent from the detailed description of non-limiting embodiments made with reference to the following drawings:

[0034] Figure 1 This is a flow chart of a method for constructing and positioning a forklift real-time map based on machine vision technology according to Embodiment 1 of the present invention. DETAILED DESCRIPTION

[0035] Example 1

[0036] This embodiment proposes a method for building and locating a forklift real-time map based on machine vision technology. By collecting three-dimensional modeling data in the factory, a three-dimensional visualization model of the factory environment is established; key nodes in the three-dimensional visualization model are extracted to build a topological flow association diagram of the factory environment; the real-time motion state of the forklift is obtained, and the real-time motion state of the forklift is mapped to the topological flow association diagram of the factory environment; according to the real-time motion state of the forklift, the actual behavior pattern probability distribution of the forklift is identified, and the similarity between the actual behavior pattern probability distribution of the forklift and the preset behavior pattern probability distribution of the forklift is calculated; when the similarity is lower than the similarity threshold, a forklift warning is issued to the remote management terminal. Warning management is performed based on analyzing the behavior pattern of the forklift and obtaining the gap between the behavior pattern of the forklift and the ideal behavior pattern.

[0037] Specifically, Figure 1 As shown, the method for constructing and positioning a forklift real-time map based on machine vision technology proposed in this embodiment includes the following specific steps:

[0038] S1. Collect 3D modeling data in the factory and build a 3D visualization model of the factory environment;

[0039] S2, extract key nodes in the 3D visualization model and construct a topological flow association diagram of the factory environment;

[0040] S3, obtaining the real-time motion state of the forklift, and mapping the real-time motion state of the forklift to a topological flow association diagram of the factory environment;

[0041] S4. According to the real-time motion state of the forklift, identify the actual behavior pattern probability distribution of the forklift, and calculate the similarity between the actual behavior pattern probability distribution of the forklift and the preset behavior pattern probability distribution of the forklift;

[0042] S5. When the similarity is lower than the similarity threshold, a forklift warning is issued to the remote management terminal.

[0043] In this embodiment, S1 includes the following specific steps:

[0044] S11. Arrange a 3D laser scanning station inside the factory according to the factory layout and scanning plan, and use a high-precision 3D laser scanner to scan inside and outside the factory to obtain the first point cloud data;

[0045] S12, using a total station or a precision level to measure the elevation control points between the scanning sites, and introducing the elevation data into the first point cloud data, correcting the first point cloud data in the vertical direction, outputting the second point cloud data, performing splicing, denoising, and registration processing on the second point cloud data, and outputting a three-dimensional point cloud model of the factory;

[0046] S13. Obtain the factory's construction drawings, equipment layout diagrams, process flow diagrams, and material storage diagrams, and use Revit software to build a three-dimensional model of the factory's buildings and equipment components;

[0047] S14. Use GPS positioning to obtain the GCJ-02 coordinates of factory buildings and equipment components, and integrate the three-dimensional point cloud model, three-dimensional model, and GCJ-02 coordinates into a complete three-dimensional visualization model of the factory.

[0048] In this embodiment, the specific content of S13 is: construct a three-dimensional model of the factory building and equipment components according to the factory's construction drawings, equipment layout diagrams, process flow diagrams and material storage diagrams, and assign material properties and structural parameter information to the factory buildings and equipment components. At the same time, add equipment number, manufacturer, and maintenance record information for each equipment component, and export the three-dimensional model as an ifc format file through the IFC data exchange standard.

[0049] In this embodiment, S2 includes the following specific steps:

[0050] S21, extracting key nodes in the three-dimensional visualization model, wherein the key nodes include a forklift charging station, a material storage area, and a logistics channel entrance and exit, and creating a key node set A; the key node set A includes the GCJ-02 coordinates of the key nodes of the factory environment;

[0051] S22. Determine the actual logistics path according to the process flow chart and material storage diagram of the factory, use the actual logistics path as the movement path of the forklift, and create a directed edge set B; the directed edge set B includes the movement path of the forklift along the key nodes of the factory environment.

[0052] S23, construct a directed graph C = (A, B), the directed graph is a topological flow association graph of the factory environment.

[0053] In this embodiment, the real-time motion state of the forklift in S3 includes: the real-time GCJ-02 coordinates, real-time acceleration, real-time steering wheel angle, and real-time speed direction of the forklift.

[0054] In this embodiment, S4 includes:

[0055] S41, combining the real-time GCJ-02 coordinates, real-time acceleration, real-time steering wheel angle, and real-time speed direction of the forklift into a feature vector form;

[0056] S42. Input the set of feature vectors into a machine learning model, wherein the machine learning model takes the behavior pattern probability predicted by each set of feature vectors as output, takes the actual behavior pattern probability corresponding to each set of feature vectors as a prediction target, and takes minimizing the sum of the prediction accuracies of all predicted behavior pattern probabilities as a training target, and stops training when the sum of the prediction accuracies converges, and outputs the actual behavior pattern probability distribution of the forklift.

[0057] In this embodiment, the behavior modes in S42 include stationary, uniform motion, turning, acceleration to braking, and braking to acceleration; and the machine learning model is any one of a random forest, a support vector machine, and a neural network model.

[0058] In this embodiment, S4 further includes:

[0059] S43, according to the real-time motion state of the forklift, setting the preset probability of each behavior mode of the forklift, and forming a probability distribution of the preset behavior mode of the forklift;

[0060] S44, calculating the similarity between the actual behavior pattern probability distribution of the forklift and the preset behavior pattern probability distribution of the forklift, the similarity calculation formula is: Among them, a i is the actual behavior pattern probability of the forklift, b i is the probability of the preset behavior pattern of the forklift.

[0061] Example 2

[0062] This embodiment provides an electronic device, including: a processor and a memory, wherein the memory stores a computer program that can be called by the processor; the processor executes the above-mentioned forklift real-time map construction and positioning method based on machine vision technology by calling the computer program stored in the memory.

[0063] The electronic device may have relatively large differences due to different configurations or performances, and may include one or more processors (Central Processing Units, CPU) and one or more memories, wherein the memory stores at least one computer program, and the computer program is loaded and executed by the processor to implement a forklift real-time map construction and positioning method based on machine vision technology provided by the above method embodiment. The electronic device may also include other components for realizing the functions of the device, for example, the electronic device may also have components such as a wired or wireless network interface and an input and output interface to input and output data. This embodiment will not be described in detail here.

[0064] Those skilled in the art know that the present invention can be implemented as a system, method or computer program product. Therefore, the present disclosure can be specifically implemented in the following forms, namely: it can be complete hardware, it can be complete software (including firmware, resident software, microcode, etc.), and it can also be a combination of hardware and software, which is generally referred to as "circuit", "module" or "system" herein. In addition, in some embodiments, the present invention can also be implemented in the form of a computer program product in one or more computer-readable media, and the computer-readable medium contains computer-readable program code.

[0065] Any combination of one or more computer-readable media may be used. A computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium. A computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or device, or any combination thereof. More specific examples of computer-readable storage media (a non-exhaustive list) include: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In this document, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in conjunction with an instruction execution system, device, or device.

[0066] The present invention is described with reference to flowcharts and block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process or block in the flowchart and block diagram, as well as the combination of processes and blocks in the flowchart or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowcharts. Figure 1 Process or multiple processes and boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0067] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 Process or multiple processes and boxes Figure 1 The steps for the functions specified in one or more boxes.

[0068] The embodiments of the present invention are described above in conjunction with the accompanying drawings, but the present invention is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the enlightenment of the present invention, ordinary technicians in this field can also make many forms without departing from the scope of protection of the purpose of the present invention and the claims, which all fall within the protection of the present invention.

Claims

1. A forklift real-time map construction and positioning method based on machine vision technology, characterized in that: The specific steps include: S1. Collect 3D modeling data in the factory and build a 3D visualization model of the factory environment; S2, extract key nodes in the 3D visualization model and construct a topological flow association diagram of the factory environment; S3, obtaining the real-time motion state of the forklift, and mapping the real-time motion state of the forklift to a topological flow association diagram of the factory environment; S4. According to the real-time motion state of the forklift, identify the actual behavior pattern probability distribution of the forklift, and calculate the similarity between the actual behavior pattern probability distribution of the forklift and the preset behavior pattern probability distribution of the forklift; S5. When the similarity is lower than the similarity threshold, a forklift warning is issued to the remote management terminal.

2. The method for constructing and positioning a forklift real-time map based on machine vision technology according to claim 1, characterized in that: The S1 comprises the following specific steps: S11. Arrange a 3D laser scanning station inside the factory according to the factory layout and scanning plan, and use a high-precision 3D laser scanner to scan inside and outside the factory to obtain the first point cloud data; S12, using a total station or a precision level to measure the elevation control points between the scanning sites, and introducing the elevation data into the first point cloud data, correcting the first point cloud data in the vertical direction, outputting the second point cloud data, performing splicing, denoising, and registration processing on the second point cloud data, and outputting a three-dimensional point cloud model of the factory; S13. Obtain the factory's construction drawings, equipment layout diagrams, process flow diagrams, and material storage diagrams, and use Revit software to build a three-dimensional model of the factory's buildings and equipment components; S14. Use GPS positioning to obtain the GCJ-02 coordinates of factory buildings and equipment components, and integrate the three-dimensional point cloud model, three-dimensional model, and GCJ-02 coordinates into a complete three-dimensional visualization model of the factory.

3. The method for constructing and positioning a forklift real-time map based on machine vision technology according to claim 2, characterized in that: The specific content of S13 is: constructing a three-dimensional model of the factory building and equipment components according to the factory's construction drawings, equipment layout diagrams, process flow diagrams and material storage diagrams, and assigning material properties and structural parameter information to the factory building and equipment components, while adding equipment number, manufacturer, and maintenance record information to each equipment component, and exporting the three-dimensional model to an ifc format file through the IFC data exchange standard.

4. The method for constructing and positioning a forklift real-time map based on machine vision technology according to claim 3 is characterized in that: The S2 comprises the following specific steps: S21, extracting key nodes in the three-dimensional visualization model, wherein the key nodes include a forklift charging station, a material storage area, and a logistics channel entrance and exit, and creating a key node set A; the key node set A includes the GCJ-02 coordinates of the key nodes of the factory environment; S22. Determine the actual logistics path according to the process flow chart and material storage diagram of the factory, use the actual logistics path as the movement path of the forklift, and create a directed edge set B; the directed edge set B includes the movement path of the forklift along the key nodes of the factory environment. S23, construct a directed graph C = (A, B), the directed graph is a topological flow association graph of the factory environment.

5. The method for constructing and positioning a forklift real-time map based on machine vision technology according to claim 4 is characterized in that: The real-time motion state of the forklift in S3 includes: the real-time GCJ-02 coordinates, real-time acceleration, real-time steering wheel angle, and real-time speed direction of the forklift.

6. The method for constructing and positioning a forklift real-time map based on machine vision technology according to claim 5, characterized in that: The S4 includes: S41, combining the real-time GCJ-02 coordinates, real-time acceleration, real-time steering wheel angle, and real-time speed direction of the forklift into a feature vector form; S42. Input the set of feature vectors into a machine learning model, wherein the machine learning model takes the behavior pattern probability predicted by each set of feature vectors as output, takes the actual behavior pattern probability corresponding to each set of feature vectors as a prediction target, and takes minimizing the sum of the prediction accuracies of all predicted behavior pattern probabilities as a training target, and stops training when the sum of the prediction accuracies converges, and outputs the actual behavior pattern probability distribution of the forklift.

7. The method for constructing and positioning a forklift real-time map based on machine vision technology according to claim 6, characterized in that: The behavior modes in S42 include stationary, uniform motion, turning, acceleration-to-braking, and braking-to-acceleration; The machine learning model is any one of a random forest, a support vector machine and a neural network model.

8. The method for constructing and positioning a forklift real-time map based on machine vision technology according to claim 7, characterized in that: The S4 further comprises: S43, according to the real-time motion state of the forklift, setting the preset probability of each behavior mode of the forklift, and forming a probability distribution of the preset behavior mode of the forklift; S44, calculating the similarity between the actual behavior pattern probability distribution of the forklift and the preset behavior pattern probability distribution of the forklift, the similarity calculation formula is: Among them, a i is the actual behavior pattern probability of the forklift, b i is the probability of the preset behavior pattern of the forklift.

Images