Automatic guided vehicle navigation positioning method and system based on look-around semantic image

Abstract

The invention discloses an automatic guided vehicle navigation positioning method and system based on a look-around semantic image, and relates to the technical field of intelligent transportation, and the method comprises the steps: generating a vehicle operation path and a vehicle operation reference trajectory according to a vehicle operation task; the method comprises the following steps: when a vehicle works, collecting a scene look-around image in real time, and converting the scene look-around image into a real-time semantic image through a semantic network; obtaining the accurate pose of the vehicle according to the real-time semantic image; and establishing a vehicle kinematic model, and performing motion control on the vehicle in combination with the accurate pose of the vehicle and the vehicle running reference trajectory to complete a vehicle operation task. By combining semantic feature information, the influence of factors such as deficiency and singleness of textures in a workshop environment and the like which enable the effect of a traditional visual positioning method to be poorer is avoided, accumulated errors are eliminated by combining a global map, a centimeter-level positioning effect is obtained, the cost is low, the precision is high, and the method can adapt to more complex operation tasks and operation environments.

Description

Technical Field

[0001] The invention relates to the field of intelligent transportation technology, and in particular to an automatic guided vehicle navigation and positioning method based on surround semantic images. Background Art

[0002] An automated guided vehicle (AGV) is an unmanned vehicle that can autonomously navigate and transport materials. It uses various sensing technologies, such as lidar and cameras, to sense the surrounding environment and make decisions. It is an important development direction of intelligent transportation in the intelligent development of manufacturing. In the field of intelligent warehousing, AGV can realize the automated storage and retrieval of goods, improve the utilization rate of storage space and the efficiency of goods management. In the logistics and distribution link, AGV can accurately travel between different logistics nodes, realize the rapid sorting and transportation of goods, reduce logistics costs, and improve the quality and speed of logistics services. AGV can greatly improve production efficiency, ensure the efficiency and smoothness of the production process through automated material handling and optimized work processes; it can reduce labor costs, reduce the risk of damage to goods and improve space utilization, while improving quality and safety, ensuring stable product quality, and is a very important part of the intelligent manufacturing system. However, traditional AGVs rely on magnetic tapes, magnetic nails, QR codes, etc. for navigation and positioning, and their accuracy and routes are often greatly limited, and cannot adapt to more diverse operational needs.

[0003] In recent years, autonomous driving technology has developed rapidly. It has transitioned from traditional visual simultaneous positioning and mapping methods (such as ORB-SLAM2) to diversified positioning technologies that utilize multiple sensors (such as lidar, GPS, radar, and IMU) and multi-sensor fusion methods (such as GPS, radar, and IMU). Each positioning method has obvious advantages and disadvantages. For example, although lidar has high accuracy, it is expensive and lacks the ability to recognize color semantics and other information. In contrast, vision-based methods are cost-effective, information-rich, and benefit from rapid technological advances.

[0004] With the development of autonomous driving technology, AGVs that use lidar and visual sensors for free navigation and positioning have gradually developed, which have obvious advantages in perception, navigation and safety, making them more suitable for complex and dynamic manufacturing environments. Visual AGVs have the advantages of low cost, large amount of information and accurate target detection, and have broad development prospects. However, industrial environments often lack GPS signals, have many closed and narrow areas, lack of wall textures, many repeated structures, unstable lighting and easily changing environments, which often cause visual positioning to fail or have poor results.

[0005] Therefore, technicians in this field are committed to developing an automatic guided vehicle navigation and positioning method and system based on surround semantic images, which has low cost, high positioning accuracy, and can adapt to more complex work tasks and working environments. Summary of the invention

[0006] In view of the above-mentioned defects of the prior art, the technical problem to be solved by the present invention is that the traditional visual method has low positioning accuracy, high cost and cannot cope with more complex environments.

[0007] To achieve the above object, the present invention provides an automatic guided vehicle navigation and positioning method based on surround semantic images, the method comprising the following steps:

[0008] Step 1: Generate a vehicle operation path and a vehicle operation reference trajectory according to the vehicle operation task;

[0009] Step 2: When the vehicle is operating, a scene surround image is collected in real time, and the scene surround image is converted into a real-time semantic image through a semantic network;

[0010] Step 3: obtaining the precise position and posture of the vehicle according to the real-time semantic image;

[0011] Step 4: Establish a vehicle kinematic model, and control the vehicle motion based on the precise position and posture of the vehicle and the reference trajectory of the vehicle to complete the vehicle operation task.

[0012] Furthermore, the step 1 further comprises:

[0013] Step 1.1, set key nodes on the global map, store the coordinates and status information of the key nodes, and generate a topological node map;

[0014] Step 1.2, according to the operation task, a series of node sequences from the starting point to the pickup point and then to the delivery point are planned in the topological node map as the vehicle operation path;

[0015] Step 1.3: After obtaining the task node sequence, use straight lines and curves to connect the nodes to obtain a series of continuous reference trajectories as the vehicle operation reference trajectories.

[0016] Furthermore, the key nodes include road intersections, turning points, parking points, garage points, starting points, pickup points, and delivery points.

[0017] Furthermore, in step 2, the semantic network segments the scene surround image using the semantics including lane lines, arrows, garage lines, and crosswalks.

[0018] Furthermore, the step 3 also includes:

[0019] Step 3.1: for the acquired real-time semantic image, perform ICP matching on the real-time semantic images of adjacent frames to obtain the relative position and posture between the two frames;

[0020] Step 3.2: Use the relative pose as a visual odometer and combine it with the vehicle pose of the previous frame to obtain a rough estimate of the vehicle pose in the current frame.

[0021] Step 3.3: Use the roughly estimated pose as the initial value, select a local map from the global map, match it with the real-time semantic image of the current frame, and obtain the precise pose of the vehicle in the current frame.

[0022] Furthermore, the step 4 also includes:

[0023] Step 4.1, establish vehicle kinematic model;

[0024] Step 4.2, obtaining a vehicle error state according to the vehicle running reference trajectory obtained in step 1 and the vehicle precise posture obtained in step 3;

[0025] Step 4.3: Input the vehicle error state into the vehicle kinematics model, and the vehicle kinematics model outputs control of vehicle speed and angular velocity.

[0026] Furthermore, the vehicle error state includes: a real-time angle and a vertical distance error between the actual operation of the vehicle and the reference trajectory of the vehicle operation.

[0027] Furthermore, the method for constructing the global map includes: the vehicle collects site images by circling the site in advance, and with the help of a visual odometer and loop optimization correction, the collected data are spliced ​​together to form a global map.

[0028] A navigation and positioning system for an automatic guided vehicle based on surround view semantic images, characterized in that it includes a surround view module, a mapping module, a positioning module, and a control module, wherein the surround view module is used to obtain scene surround view images in real time, and convert the scene surround view images into real-time semantic images through a semantic network; the mapping module optimizes the real-time semantic images through visual odometer and loop optimization, and then synthesizes a global map; the positioning module estimates the relative pose by matching the real-time semantic images of adjacent frames, uses the relative pose as the visual odometer to obtain a rough estimated pose of the vehicle, then uses the rough estimated pose as an initial value, selects a local map from the global map, and matches it with the real-time semantic image of the current frame to obtain the precise pose of the vehicle in the current frame.

[0029] Furthermore, the control module is used to generate a topological node map, and generate a vehicle operation path and a vehicle operation reference trajectory through the topological node map; the control module is used to establish a vehicle kinematic model, and obtain a vehicle error state according to the vehicle operation reference trajectory and the vehicle's precise posture output by the positioning module, and input the vehicle error state into the vehicle kinematic model to obtain a vehicle control quantity output, and control the vehicle to move according to a given reference trajectory, thereby completing the entire operation process.

[0030] Compared with the prior art, the present invention has at least the following beneficial technical effects:

[0031] The present invention converts surround images into semantic maps by training semantic networks, which can more effectively express scene features, avoid the failure of traditional image feature positioning, and adapt to factory environments with single textures and changeable scenes; by correcting the global map, the accumulated error of the semantic feature odometer is eliminated, and a high-precision visual positioning effect can be obtained; combined with the vehicle motion model and positioning module, a controller is used to control the vehicle to move along a standard trajectory to complete the vehicle operation task, and the control error is controlled at the centimeter level.

[0032] The present invention can complete the operation task under the condition of low-cost hardware, has better adaptability to various operations, and can meet the needs of more complex tasks.

[0033] The concept, specific structure and technical effects of the present invention will be further described below in conjunction with the accompanying drawings to fully understand the purpose, characteristics and effects of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 It is a schematic diagram of the system structure of a preferred embodiment of the present invention;

[0035] Figure 2 It is a schematic diagram of a vehicle running reference trajectory of a preferred embodiment of the present invention;

[0036] Figure 3 It is a schematic diagram of the positioning process of a preferred embodiment of the present invention;

[0037] Figure 4 This is a schematic diagram of a kinematic model of a single-steering wheel AGV according to a preferred embodiment of the present invention;

[0038] Figure 5 It is a schematic diagram of the kinematic model of an improved single-steering wheel AGV according to a preferred embodiment of the present invention. DETAILED DESCRIPTION

[0039] The following describes several preferred embodiments of the present invention with reference to the drawings in the specification, so that the technical content is clearer and easier to understand. The present invention can be embodied in many different forms of embodiments, and the protection scope of the present invention is not limited to the embodiments mentioned in the text.

[0040] In the drawings, components with the same structure are indicated by the same numerical reference numerals, and components with similar structures or functions are indicated by similar numerical reference numerals. The size and thickness of each component shown in the drawings are arbitrarily shown, and the present invention does not limit the size and thickness of each component. In order to make the illustration clearer, the thickness of the components is appropriately exaggerated in some places in the drawings.

[0041] This embodiment provides an automatic guided vehicle navigation and positioning method based on surround semantic images. The method is implemented on an automatic guided vehicle navigation and positioning system based on surround semantic images. Figure 1 As shown, the system includes a surround view module, a mapping module, a positioning module, and a control module, wherein the surround view module is used to obtain the surround view image of the AGV scene in real time, convert the scene surround view image into a real-time semantic image through a semantic network, and send the real-time semantic image to the mapping module and the positioning module; the mapping module optimizes the pose of the real-time semantic image collected by the surround view module through visual odometer and loop optimization, and then synthesizes a global map; the positioning module estimates the relative pose by matching the real-time semantic images of adjacent frames, and uses the relative pose as the visual odometer to obtain a rough estimated pose of the vehicle, and then uses the rough estimated pose as the initial value, selects a local map in the global map, matches it with the real-time semantic image of the current frame, and obtains the precise pose of the vehicle in the current frame; the control module is used to generate a topological node map, and generates a vehicle operation path and a vehicle operation reference trajectory through the topological node map; the control module is used to establish a vehicle kinematic model, obtain a vehicle error state according to the vehicle operation reference trajectory and the precise pose of the vehicle output by the positioning module, input the vehicle error state into the vehicle kinematic model to obtain the vehicle control quantity output, and control the vehicle to move according to a given reference trajectory, thereby completing the entire operation process.

[0042] The navigation positioning method includes the following steps:

[0043] Step 1: Generate vehicle operation path and vehicle operation reference trajectory according to vehicle operation tasks. Specifically include:

[0044] Step 1.1, set key nodes on the global map, store the coordinates and status information of the key nodes, and generate a topological node map; key nodes include road junctions, turning points, parking points, garage points, starting points, pickup points, and delivery points;

[0045] Step 1.2: According to the operation task, a series of node sequences from the starting point to the pickup point and then to the delivery point are planned in the topological node map as the vehicle operation path;

[0046] Step 1.3: After obtaining the task node sequence, use straight lines and curves to connect the nodes to obtain a series of continuous reference trajectories as the vehicle operation reference trajectory, such as Figure 2 shown.

[0047] Step 2: When the vehicle is operating, real-time surround view images are collected, and the surround view images are converted into real-time semantic images through the semantic network. The use of semantic features can handle scenes that are difficult to accurately locate with traditional features, and enhance the positioning accuracy of visual methods. The construction of the semantic network includes: using data annotation software to annotate scene semantics such as lane lines, arrows, garage lines, crosswalks, etc. on the collected images; after obtaining the annotated data set, use the semantic network for training to obtain a semantic network that can segment common semantics in factory scenes.

[0048] Step 3: Obtain the precise position and posture of the vehicle based on the real-time semantic image. Figure 3 As shown, specifically including:

[0049] Step 3.1: For the acquired real-time semantic image, perform ICP matching on the real-time semantic images of adjacent frames to obtain the relative position and posture between the two frames;

[0050] Step 3.2: Use the relative pose as a visual odometer and combine it with the vehicle pose of the previous frame to obtain a rough estimate of the vehicle pose in the current frame.

[0051] Step 3.3: Use the roughly estimated pose as the initial value, select a local map from the global map, match it with the semantic image of the current frame, and obtain the precise pose of the vehicle in the current frame.

[0052] Among them, due to the cumulative error of odometer estimation, the vehicle posture error will gradually increase, and the odometer estimation error will be large when the vehicle speed changes too quickly. Using global map correction can effectively reduce or eliminate the odometer error. The vehicle collects site images by circling the site in advance, and with the help of visual odometer and loop optimization correction, the collected data is spliced ​​together to form a global map. The introduction of the global map can eliminate the odometer estimation error caused by factors such as rapid speed changes, and on the other hand, it can also eliminate the odometer's cumulative error problem and avoid the odometer estimation drift problem.

[0053] Step 4: Establish a vehicle kinematic model, and control the vehicle motion based on the vehicle’s precise position and reference trajectory to complete the vehicle’s operation task. Specifically, it includes:

[0054] Step 4.1, establish vehicle kinematic model;

[0055] like Figure 4 As shown, the kinematic model of the single steering wheel AGV is:

[0056]

[0057] Among them, x, y, z represent the coordinates of the AGV in the world coordinate system, θ represents the body angle, v is the vehicle linear velocity, and w is the vehicle angular velocity. d is the vehicle driving wheel speed, δ is the wheel deflection angle, and L is the distance between the vehicle driving wheel and the driven wheel.

[0058] It can be seen that the vehicle has a driving wheel to provide power and steering, and two sets of driven wheels for stability.

[0059] like Figure 5 As shown in the figure, the kinematic model of the single steering wheel AGV is improved to obtain:

[0060]

[0061] Where e is the error of the vehicle in x, y and the yaw angle direction, v r 、w r is the reference linear velocity and reference angular velocity corresponding to the reference trajectory, k 2 , k 3 is the controller parameter, v out 、w out are the controller output linear velocity and angular velocity respectively.

[0062] Step 4.2, according to the vehicle running reference trajectory obtained in step 1 and the vehicle precise posture obtained in step 3, the vehicle error state is obtained, and the vehicle error state is the real-time angle and vertical distance error between the actual operation of the vehicle and the vehicle running reference trajectory;

[0063] Step 4.3: Input the vehicle error state into the vehicle kinematics model, and the vehicle kinematics model outputs the control of the vehicle speed and angular velocity, where v 0 is the reference speed, w 0 is the reference angular velocity. By reasonably designing the parameters of the vehicle kinematic model, the control error can be controlled within a small range, so that the vehicle can travel along the reference trajectory.

[0064] In this embodiment, the model implementation platform is a single-steering wheel AGV forklift. The experimental platform is equipped with an industrial computer with an Ubuntu operating system, a GeForce GTX 1050ti GPU graphics card and an i7-8700 CPU. The platform is equipped with four fisheye cameras at the front, rear, left and right as surround view system perception sensors. The front is equipped with active wheels for driving and steering, and the rear is equipped with two sets of driven wheels to maintain balance. The surround view module uses the classic U-net as the semantic network, and the training set is obtained through on-site collection and annotation. The positioning module uses ICP matching as the semantic image matching method, with 1000 iterations. The site size is 14m×16m, equipped with a parking garage and six work garages.

[0065] First, test the positioning accuracy of the system. Repeatedly park the vehicle in the garage, record the distance between the center of the vehicle and the center line of the garage, and actually measure the distance between the center of the vehicle and the center line of the garage to compare and obtain the positioning error of the vehicle. Repeatedly park in the garage 20 times and measure. The positioning error results are shown in Table 1:

[0066] Table 1

[0067] Error average cm Maximum error cm 3.07 5.17

[0068] Analysis of experimental results: As shown in Table 1, the average positioning error is about 3 cm, and the maximum error is about 5 cm. It can be seen that the vehicle positioning accuracy reaches the centimeter level, which can meet the positioning requirements of vehicle operations. At the same time, centimeter-level positioning is sufficient to meet most factory operation requirements.

[0069] After that, the control accuracy of the system was tested. Various instructions were executed on the vehicle, including long straight road movement, detour and operation tasks. The deviation between the position of the vehicle platform and the reference line in various tasks was recorded to obtain the control accuracy of the vehicle platform. The control accuracy results are shown in Table 2:

[0070] Table 2

[0071] Scenario Running length m Average distance error cm Average angle error° Long straight road 11.23 3.55 1.18 Detour site 67.19 4.21 3.71 Operation 24.57 4.06 2.97

[0072] Experimental results analysis: The vehicle has the smallest error on the long straight road, with an average vertical distance error of 3.55cm and an angle error of about 1°; the vehicle detoured a long distance and made many turns, with an average distance error of 4.21cm and an angle error of less than 4°; the average distance error of the vehicle in normal operation was about 4cm, and the angle error was about 3°. Overall, the vehicle deviated from the reference line by about 4cm when performing tasks, had a good fit with the reference line, had a small angle error, and basically moved along the reference line. The vehicle was able to perform tasks well according to the given reference line.

[0073] The preferred specific embodiments of the present invention are described in detail above. It should be understood that a person skilled in the art can make many modifications and changes based on the concept of the present invention without creative work. Therefore, any technical solution that can be obtained by a person skilled in the art through logical analysis, reasoning or limited experiments based on the concept of the present invention on the basis of the prior art should be within the scope of protection determined by the claims.

Claims

1. A navigation and positioning method for an automatic guided vehicle based on surround semantic images, characterized in that: The method comprises the following steps: Step 1: Generate a vehicle operation path and a vehicle operation reference trajectory according to the vehicle operation task; Step 2: When the vehicle is operating, a scene surround image is collected in real time, and the scene surround image is converted into a real-time semantic image through a semantic network; Step 3: obtaining the precise position and posture of the vehicle according to the real-time semantic image; Step 4: Establish a vehicle kinematic model, and control the vehicle motion based on the precise position and posture of the vehicle and the reference trajectory of the vehicle to complete the vehicle operation task.

2. The method for automatic guided vehicle navigation and positioning based on surround semantic images according to claim 1, characterized in that: The step 1 also includes: Step 1.1, set key nodes on the global map, store the coordinates and status information of the key nodes, and generate a topological node map; Step 1.2, according to the operation task, a series of node sequences from the starting point to the pickup point and then to the delivery point are planned in the topological node map as the vehicle operation path; Step 1.3: After obtaining the task node sequence, use straight lines and curves to connect the nodes to obtain a series of continuous reference trajectories as the vehicle operation reference trajectories.

3. The method for navigation and positioning of an automatic guided vehicle based on surround semantic images as claimed in claim 2, characterized in that: The key nodes include road junctions, turning points, parking points, garage points, starting points, pickup points, and delivery points.

4. The method for navigation and positioning of an automatic guided vehicle based on surround semantic images according to claim 1, characterized in that: In the step 2, the semantic network segments the scene surround image with semantics including: lane lines, arrows, garage lines, and crosswalks.

5. The method for navigation and positioning of an automatic guided vehicle based on surround semantic images according to claim 1, characterized in that: The step 3 also includes: Step 3.1: for the acquired real-time semantic image, perform ICP matching on the real-time semantic images of adjacent frames to obtain the relative position and posture between the two frames; Step 3.2: Use the relative pose as a visual odometer and combine it with the vehicle pose of the previous frame to obtain a rough estimate of the vehicle pose in the current frame. Step 3.3: Use the roughly estimated pose as the initial value, select a local map from the global map, match it with the real-time semantic image of the current frame, and obtain the precise pose of the vehicle in the current frame.

6. The method for navigation and positioning of an automatic guided vehicle based on surround semantic images according to claim 1, characterized in that: The step 4 also includes: Step 4.1, establish vehicle kinematic model; Step 4.2, obtaining a vehicle error state according to the vehicle running reference trajectory obtained in step 1 and the vehicle precise posture obtained in step 3; Step 4.3: Input the vehicle error state into the vehicle kinematics model, and the vehicle kinematics model outputs control of vehicle speed and angular velocity.

7. The method for navigation and positioning of an automatic guided vehicle based on surround semantic images according to claim 6, characterized in that: The vehicle error state includes: a real-time angle and a vertical distance error between the actual operation of the vehicle and the reference trajectory of the vehicle operation.

8. The method for automatic guided vehicle navigation and positioning based on surround semantic images according to claim 2 or 5, characterized in that: The method for constructing the global map includes: the vehicle collects site images by circling the site in advance, and with the help of a visual odometer and loop optimization correction, the collected data are spliced ​​together to form a global map.

9. An automatic guided vehicle navigation and positioning system based on surround semantic images, characterized in that: It includes a surround view module, a mapping module, a positioning module, and a control module, wherein the surround view module is used to obtain scene surround view images in real time, and convert the scene surround view images into real-time semantic images through a semantic network; the mapping module optimizes the real-time semantic images through a visual odometer and loop optimization, and then synthesizes a global map; the positioning module estimates the relative posture by matching the real-time semantic images of adjacent frames, uses the relative posture as a visual odometer to obtain a rough estimated posture of the vehicle, and then uses the rough estimated posture as an initial value, selects a local map from the global map, and matches it with the real-time semantic image of the current frame to obtain the precise posture of the vehicle in the current frame.

10. The automatic guided vehicle navigation and positioning system based on surround semantic images according to claim 9, characterized in that: The control module is used to generate a topological node map, and generate a vehicle operation path and a vehicle operation reference trajectory through the topological node map; the control module is used to establish a vehicle kinematic model, and obtain a vehicle error state according to the vehicle operation reference trajectory and the vehicle's precise posture output by the positioning module, and input the vehicle error state into the vehicle kinematic model to obtain a vehicle control quantity output, and control the vehicle to move according to a given reference trajectory, thereby completing the entire operation process.

Images