A method for real-time forklift map construction and positioning based on machine vision technology

Abstract

The present invention relates to the technical field of forklift positioning, and particularly to a method for real-time map construction and positioning of a forklift based on machine vision technology. The method includes: collecting three-dimensional modeling data in a factory to establish a three-dimensional visualization model of the factory environment; extracting key nodes in the three-dimensional visualization model to construct a topological flow association graph of the factory environment; obtaining the real-time motion state of the forklift and mapping the real-time motion state of the forklift to the topological flow association graph of the factory environment; identifying the probability distribution of the actual behavior pattern of the forklift according to the real-time motion state of the forklift, and calculating the similarity between the probability distribution of the actual behavior pattern of the forklift and the preset behavior pattern probability distribution of the forklift; and sending a forklift warning to a remote management terminal when the similarity is lower than a similarity threshold.

Description

Background Art

[0002] With the rapid development of the logistics industry, forklifts play a crucial role in warehousing logistics. However, there are many problems in the traditional forklift management methods, such as inaccurate positioning, irregular management, low efficiency, etc. These problems seriously affect the operation efficiency and competitiveness of enterprises. Therefore, in order to improve the management level and working efficiency of forklifts, forklift positioning technology has emerged as the times require.

[0003] Forklift positioning technology is widely used in industries such as logistics, manufacturing, and warehousing. Through forklift positioning technology, enterprises can achieve real-time tracking and positioning of forklifts, improving the accuracy and efficiency of warehouse management. At the same time, forklift positioning technology can also help enterprises optimize forklift scheduling and management processes, reduce forklift resource waste and maintenance costs, and improve the operation efficiency and market competitiveness of enterprises.

[0004] Problems existing in the prior art: When positioning forklifts, there are currently few methods that analyze the behavior patterns of forklifts while positioning them, and obtain the gap between the forklift behavior patterns and the ideal behavior patterns for early warning management. Summary of the Invention

[0005] The main purpose of the present invention is to provide a forklift real-time map construction and positioning method based on machine vision technology. By collecting three-dimensional modeling data in the factory, a three-dimensional visualization model of the factory environment is established; key nodes in the three-dimensional visualization model are extracted to construct a topological flow association graph of the factory environment; the real-time motion state of the forklift is obtained, and the real-time motion state of the forklift is mapped to the topological flow association graph of the factory environment; according to the real-time motion state of the forklift, the probability distribution of the actual behavior pattern of the forklift is identified, and the similarity between the probability distribution of the actual behavior pattern of the forklift and the preset probability distribution of the behavior pattern of the forklift is calculated; a forklift early warning is sent to the remote management terminal when the similarity is lower than the similarity threshold. Based on analyzing the behavior pattern of the forklift and obtaining the gap between the forklift behavior pattern and the ideal behavior pattern for early warning management, the above problems mentioned in the background art are effectively solved.

[0006] The technical solution of the present invention is as follows:

[0007] In the first aspect, a forklift real-time map construction and positioning method based on machine vision technology is proposed, and the method includes the following steps:

[0008] S1. Collect three-dimensional modeling data in the factory and establish a three-dimensional visualization model of the factory environment;

[0009] S2. Extract key nodes in the three-dimensional visualization model and construct a topological flow association graph of the factory environment;

[0010] S3. Obtain the real-time motion state of the forklift and map the real-time motion state of the forklift to the topological flow correlation diagram of the factory environment;

[0011] S4. According to the real-time motion state of the forklift, identify the probability distribution of the actual behavior pattern of the forklift, and calculate the similarity between the probability distribution of the actual behavior pattern of the forklift and the preset behavior pattern probability distribution of the forklift;

[0012] S5. Raise a forklift warning to the remote management terminal for the situation where the similarity is lower than the similarity threshold.

[0013] A further improvement of the present invention is that the S1 includes the following specific steps:

[0014] S11. According to the factory layout and scanning plan, arrange 3D laser scanning stations inside the factory, and use a high-precision 3D laser scanner to scan inside and outside the factory to obtain the first point cloud data;

[0015] S12. Use a total station or precise level to measure the elevation control points between each scanning station, introduce the elevation data into the first point cloud data, correct the first point cloud data in the vertical direction, output the second point cloud data, and perform stitching, denoising, and registration processing on the second point cloud data to output the 3D point cloud model of the factory;

[0016] S13. Obtain the construction drawings, equipment layout drawings, process flow charts, and material storage drawings of the factory, and use Revit software to construct 3D models of factory buildings and equipment components;

[0017] S14. Use GPS positioning to obtain the GCJ 02 coordinates of factory buildings and equipment components, and integrate the 3D point cloud model, 3D model, and GCJ 02 coordinates into a complete 3D visualization model of the factory.

[0018] A further improvement of the present invention is that the specific content of the S13 is: construct 3D models of factory buildings and equipment components according to the construction drawings, equipment layout drawings, process flow charts, and material storage drawings of the factory, endow the factory buildings and equipment components with material properties and structural parameter information, and at the same time add equipment numbers, manufacturers, and maintenance record information to each equipment component, and export the 3D model as an ifc format file through the IFC data exchange standard.

[0019] A further improvement of the present invention is that the S2 includes the following specific steps:

[0020] S21. Extract the key nodes in the 3D visualization model. The key nodes include forklift charging stations, material stacking areas, and logistics channel entrances and exits, and create a key node set A; the key node set A contains the GCJ 02 coordinates of the key nodes in the factory environment;

[0021] S22. Determine the actual logistics path according to the process flow chart and material storage diagram of the factory, use the actual logistics path as the movement path of the forklift, and create a set of directed edges B; the set of directed edges B includes the movement paths of the forklift along the key nodes of the factory environment.

[0022] S23. Construct a directed graph C=(A,B), and this directed graph is the topological flow association graph of the factory environment.

[0023] A further improvement of the present invention is that the real-time movement state of the forklift in S3 includes: the real-time GCJ02 coordinates of the forklift, the real-time acceleration, the real-time steering wheel angle, and the real-time speed direction.

[0024] A further improvement of the present invention is that S4 includes:

[0025] S41. Combine the real-time GCJ 02 coordinates, real-time acceleration, real-time steering wheel angle, and real-time speed direction of the forklift into the form of a feature vector.

[0026] S42. Input the set of feature vectors into a machine learning model. The machine learning model takes the probability of the behavior pattern predicted by each group of feature vectors as the output, takes the probability of the actual behavior pattern corresponding to each group of feature vectors as the prediction target, and takes minimizing the sum of the prediction accuracies of all predicted behavior pattern probabilities as the training target. Stop training until the sum of the prediction accuracies reaches convergence, and output the probability distribution of the actual behavior pattern of the forklift.

[0027] A further improvement of the present invention is that the behavior patterns in S42 include stationary, uniform motion, turning, accelerating to braking, and braking to accelerating; the machine learning model is any one of a random forest, a support vector machine, and a neural network model.

[0028] A further improvement of the present invention is that S4 further includes:

[0029] S43. Set the preset probability of each behavior pattern of the forklift according to the real-time movement state of the forklift, and form the preset probability distribution of the behavior pattern of the forklift.

[0030] S44. Calculate the similarity between the probability distribution of the actual behavior pattern of the forklift and the preset probability distribution of the behavior pattern of the forklift. The similarity calculation formula is: ; where is the probability of the actual behavior pattern of the forklift, is the preset probability of the behavior pattern of the forklift.

[0031] The technical effects of the present invention are as follows:

[0032] A real-time map construction and positioning method for forklifts based on machine vision technology is constructed. By collecting three-dimensional modeling data in the factory, a three-dimensional visualization model of the factory environment is established; key nodes in the three-dimensional visualization model are extracted to construct a topological flow association graph of the factory environment; the real-time motion state of the forklift is obtained and mapped to the topological flow association graph of the factory environment; according to the real-time motion state of the forklift, the probability distribution of the actual behavior pattern of the forklift is identified, and the similarity between the probability distribution of the actual behavior pattern of the forklift and the preset behavior pattern probability distribution of the forklift is calculated; when the similarity is lower than the similarity threshold, a forklift warning is sent to the remote management terminal. Based on the analysis of the behavior pattern of the forklift and obtaining the gap between the forklift behavior pattern and the ideal behavior pattern for warning management, the above problems mentioned in the background technology are effectively solved. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Other features, objects, and advantages of the present invention will become more apparent by reading the following detailed description of non-limiting embodiments with reference to the accompanying drawings:

[0034] Figure 1 It is a flowchart of a real-time map construction and positioning method for forklifts based on machine vision technology according to Embodiment 1 of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0035] Embodiment 1

[0036] This embodiment proposes a real-time map construction and positioning method for forklifts based on machine vision technology. By collecting three-dimensional modeling data in the factory, a three-dimensional visualization model of the factory environment is established; key nodes in the three-dimensional visualization model are extracted to construct a topological flow association graph of the factory environment; the real-time motion state of the forklift is obtained and mapped to the topological flow association graph of the factory environment; according to the real-time motion state of the forklift, the probability distribution of the actual behavior pattern of the forklift is identified, and the similarity between the probability distribution of the actual behavior pattern of the forklift and the preset behavior pattern probability distribution of the forklift is calculated; when the similarity is lower than the similarity threshold, a forklift warning is sent to the remote management terminal. Based on the analysis of the behavior pattern of the forklift and obtaining the gap between the forklift behavior pattern and the ideal behavior pattern for warning management.

[0037] Specifically, as Figure 1 shown, a real-time map construction and positioning method for forklifts based on machine vision technology proposed in this embodiment includes the following specific steps:

[0038] S1. Collect three-dimensional modeling data in the factory and establish a three-dimensional visualization model of the factory environment;

[0039] S2. Extract key nodes in the three-dimensional visualization model and construct a topological flow association graph of the factory environment;

[0040] S3. Obtain the real-time motion state of the forklift and map the real-time motion state of the forklift to the topological flow correlation graph of the factory environment;

[0041] S4. According to the real-time motion state of the forklift, identify the probability distribution of the actual behavior pattern of the forklift, and calculate the similarity between the probability distribution of the actual behavior pattern of the forklift and the preset behavior pattern probability distribution of the forklift;

[0042] S5. Raise a forklift warning to the remote management terminal for the situation where the similarity is lower than the similarity threshold.

[0043] In this embodiment, the S1 includes the following specific steps:

[0044] S11. According to the factory layout and scanning plan, arrange 3D laser scanning stations inside the factory, use a high-precision 3D laser scanner to scan inside and outside the factory, and obtain the first point cloud data;

[0045] S12. Use a total station or precise level to measure the elevation control points between each scanning station, introduce the elevation data into the first point cloud data, correct the first point cloud data in the vertical direction, output the second point cloud data, and perform stitching, denoising, and registration processing on the second point cloud data to output the 3D point cloud model of the factory;

[0046] S13. Obtain the construction drawings, equipment layout drawings, process flow charts, and material storage drawings of the factory, and use Revit software to construct 3D models of factory buildings and equipment components;

[0047] S14. Use GPS positioning to obtain the GCJ 02 coordinates of factory buildings and equipment components, and integrate the 3D point cloud model, 3D model, and GCJ 02 coordinates into a complete 3D visualization model of the factory.

[0048] In this embodiment, the specific content of the S13 is: construct 3D models of factory buildings and equipment components according to the construction drawings, equipment layout drawings, process flow charts, and material storage drawings of the factory, endow the factory buildings and equipment components with material properties and structural parameter information, and at the same time add equipment numbers, manufacturers, and maintenance record information to each equipment component, and export the 3D model as an ifc format file through the IFC data exchange standard.

[0049] In this embodiment, the S2 includes the following specific steps:

[0050] S21. Extract the key nodes in the 3D visualization model. The key nodes include forklift charging stations, material stacking areas, and logistics channel entrances and exits, and create a key node set A; the key node set A contains the GCJ 02 coordinates of the key nodes of the factory environment;

[0051] S22. Determine the actual logistics path according to the process flow chart and material storage diagram of the factory, use the actual logistics path as the movement path of the forklift, and create a set of directed edges B; the set of directed edges B includes the movement paths of the forklift along the key nodes of the factory environment.

[0052] S23. Construct a directed graph C=(A,B), and this directed graph is the topological flow association graph of the factory environment.

[0053] In this embodiment, the real-time movement state of the forklift in S3 includes: the real-time GCJ 02 coordinates of the forklift, the real-time acceleration, the real-time steering wheel angle, and the real-time speed direction.

[0054] In this embodiment, S4 includes:

[0055] S41. Combine the real-time GCJ 02 coordinates, real-time acceleration, real-time steering wheel angle, and real-time speed direction of the forklift into the form of a feature vector.

[0056] S42. Input the set of feature vectors into a machine learning model. The machine learning model takes the probability of the behavior pattern predicted by each group of feature vectors as the output, takes the actual probability of the behavior pattern corresponding to each group of feature vectors as the prediction target, and takes minimizing the sum of the prediction accuracies of all predicted behavior pattern probabilities as the training target. Stop training until the sum of the prediction accuracies reaches convergence, and output the actual behavior pattern probability distribution of the forklift.

[0057] In this embodiment, the behavior patterns in S42 include stationary, uniform motion, turning, accelerating to braking, and braking to accelerating; the machine learning model is any one of a random forest, a support vector machine, and a neural network model.

[0058] In this embodiment, S4 further includes:

[0059] S43. Set the preset probability of each behavior pattern of the forklift according to the real-time movement state of the forklift, and form the preset behavior pattern probability distribution of the forklift.

[0060] S44. Calculate the similarity between the actual behavior pattern probability distribution of the forklift and the preset behavior pattern probability distribution of the forklift. The similarity calculation formula is: ; where is the actual behavior pattern probability of the forklift, is the preset behavior pattern probability of the forklift.

[0061] Embodiment 2

[0062] This embodiment provides an electronic device, including: a processor and a memory, wherein the memory stores a computer program that can be called by the processor; the processor executes the above-mentioned method for real-time forklift map construction and positioning based on machine vision technology by calling the computer program stored in the memory.

[0063] This electronic device may vary greatly due to different configurations or performances, and can include one or more processors (Central Processing Units, CPUs) and one or more memories. Among them, at least one computer program is stored in the memory, and this computer program is loaded and executed by the processor to implement the method for real-time forklift map construction and positioning based on machine vision technology provided by the above-mentioned method embodiment. This electronic device can also include other components for implementing the functions of the device. For example, this electronic device can also have components such as wired or wireless network interfaces and input / output interfaces for data input and output. This embodiment will not be elaborated here.

[0064] Those skilled in the art know that the present invention can be implemented as a system, a method, or a computer program product. Therefore, the present disclosure can be specifically implemented in the following forms: it can be completely hardware, can also be completely software (including firmware, resident software, microcode, etc.), or can also be a combination of hardware and software, generally referred to as "circuit", "module", or "system" in this article. In addition, in some embodiments, the present invention can also be implemented in the form of a computer program product in one or more computer-readable media, and the computer-readable media contains computer-readable program code.

[0065] Any combination of one or more computer-readable media can be adopted. The computer-readable media can be a computer-readable signal medium or a computer-readable storage medium. The computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or component, or any combination of the above. More specific examples (non-exhaustive list) of the computer-readable storage medium include: an electrical connection having one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this document, the computer-readable storage medium can be any tangible medium that contains or stores a program, and this program can be used by or in combination with an instruction execution system, device, or component.

[0066] The present invention will be described with reference to the flowcharts and block diagrams of methods, apparatuses (systems), and computer program products according to embodiments of the present invention. It should be understood that each flow or block in the flowcharts and block diagrams, and the combination of flows and blocks in the flowcharts or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing devices to generate a machine, such that the instructions executed by the processor of the computer or other programmable data processing devices generate means for implementing the functions specified in one or more of the flows Figure 1 one or more flows and blocks Figure 1 or one or more of the blocks.

[0067] These computer program instructions can also be loaded onto a computer or other programmable data processing device, such that a series of operation steps are executed on the computer or other programmable device to generate a computer-implemented process, so that the instructions executed on the computer or other programmable device provide steps for implementing the functions specified in one or more of the flows Figure 1 one or more flows and blocks Figure 1 or one or more of the blocks.

[0068] The embodiments of the present invention have been described above in conjunction with the accompanying drawings. However, the present invention is not limited to the above specific embodiments. The above specific embodiments are merely illustrative and not restrictive. Under the inspiration of the present invention, those of ordinary skill in the art can also make many forms without departing from the spirit and scope of the present invention as protected by the claims. All of these are within the protection scope of the present invention.

Claims

1. A forklift real-time map construction and positioning method based on machine vision technology, characterized in that: The specific steps include: S1. Collect 3D modeling data in the factory and build a 3D visualization model of the factory environment; S2, extract key nodes in the 3D visualization model and construct a topological flow association diagram of the factory environment; S3, obtaining the real-time motion state of the forklift, and mapping the real-time motion state of the forklift to a topological flow association diagram of the factory environment; S4. According to the real-time motion state of the forklift, identify the actual behavior pattern probability distribution of the forklift, and calculate the similarity between the actual behavior pattern probability distribution of the forklift and the preset behavior pattern probability distribution of the forklift; S5. When the similarity is lower than the similarity threshold, a forklift warning is issued to the remote management terminal.

2. The method for constructing and positioning a forklift real-time map based on machine vision technology according to claim 1, characterized in that: The S1 comprises the following specific steps: S11. Arrange a 3D laser scanning station inside the factory according to the factory layout and scanning plan, and use a high-precision 3D laser scanner to scan inside and outside the factory to obtain the first point cloud data; S12, using a total station or a precision level to measure the elevation control points between the scanning sites, and introducing the elevation data into the first point cloud data, correcting the first point cloud data in the vertical direction, outputting the second point cloud data, performing splicing, denoising, and registration processing on the second point cloud data, and outputting a three-dimensional point cloud model of the factory; S13. Obtain the factory's construction drawings, equipment layout diagrams, process flow diagrams, and material storage diagrams, and use Revit software to build a three-dimensional model of the factory's buildings and equipment components; S14. Use GPS positioning to obtain GCJ 02 coordinates of factory buildings and equipment components, and integrate the three-dimensional point cloud model, three-dimensional model, and GCJ 02 coordinates into a complete three-dimensional visualization model of the factory.

3. The method for constructing and positioning a forklift real-time map based on machine vision technology according to claim 2, characterized in that: The specific content of S13 is: constructing a three-dimensional model of the factory building and equipment components according to the factory's construction drawings, equipment layout diagrams, process flow diagrams and material storage diagrams, and assigning material properties and structural parameter information to the factory building and equipment components, while adding equipment number, manufacturer, and maintenance record information to each equipment component, and exporting the three-dimensional model to an ifc format file through the IFC data exchange standard.

4. The method for constructing and positioning a forklift real-time map based on machine vision technology according to claim 3 is characterized in that: The S2 comprises the following specific steps: S21, extracting key nodes in the three-dimensional visualization model, wherein the key nodes include a forklift charging station, a material storage area, and a logistics channel entrance and exit, and creating a key node set A; the key node set A includes the GCJ02 coordinates of the key nodes of the factory environment; S22, determining the actual logistics path according to the process flow chart and material storage diagram of the factory, taking the actual logistics path as the movement path of the forklift, and creating a directed edge set B; the directed edge set B includes the movement path of the forklift along the key nodes of the factory environment; S23. Construct a directed graph C=(A, B), which is a topological flow association graph of the factory environment.

5. The method for constructing and positioning a forklift real-time map based on machine vision technology according to claim 4 is characterized in that: The real-time motion state of the forklift in S3 includes: the real-time GCJ 02 coordinates, real-time acceleration, real-time steering wheel angle, and real-time speed direction of the forklift.

6. The method for constructing and positioning a forklift real-time map based on machine vision technology according to claim 5, characterized in that: The S4 includes: S41, combining the real-time GCJ 02 coordinates, real-time acceleration, real-time steering wheel angle, and real-time speed direction of the forklift into a feature vector form; S42. Input the set of feature vectors into a machine learning model, wherein the machine learning model takes the behavior pattern probability predicted by each set of feature vectors as output, takes the actual behavior pattern probability corresponding to each set of feature vectors as a prediction target, and takes minimizing the sum of the prediction accuracies of all predicted behavior pattern probabilities as a training target, and stops training when the sum of the prediction accuracies converges, and outputs the actual behavior pattern probability distribution of the forklift.

7. The method for constructing and positioning a forklift real-time map based on machine vision technology according to claim 6, characterized in that: The behavior modes in S42 include stationary, uniform motion, turning, acceleration-to-braking, and braking-to-acceleration; The machine learning model is any one of a random forest, a support vector machine and a neural network model.

8. The method for constructing and positioning a forklift real-time map based on machine vision technology according to claim 7, characterized in that: The S4 further comprises: S43, according to the real-time motion state of the forklift, setting the preset probability of each behavior mode of the forklift, and forming a probability distribution of the preset behavior mode of the forklift; S44, calculating the similarity between the actual behavior pattern probability distribution of the forklift and the preset behavior pattern probability distribution of the forklift, the similarity calculation formula is: ;in, is the actual behavior pattern probability of the forklift, is the probability of the preset behavior pattern of the forklift.

Images