Intelligent logistics warehouse guide line visual detection method based on deep learning

Abstract

The application discloses a deep learning-based intelligent logistics warehouse guide line visual detection method, and relates to the technical field of visual detection.The method comprises the following steps: configuring sensors at fixed points or mobile devices in a warehouse, and synchronously collecting images and data; pre-processing the images and data, extracting features, and splicing the features into feature vectors; manually labeling RGB images, labeling feature vectors according to the labeling results, integrating data, and constructing a data set; training a U-Net model to perform guide line semantic segmentation, identifying guide line information according to a prediction result; fusing multi-source data to construct a grid map, and realizing device positioning by fusing vision and laser radar; planning a path by using an A* algorithm, controlling the travel of the device according to the planned path, and adjusting the state in real time according to the guide line information.The method can effectively improve the insufficient detection accuracy in the prior art.

Description

Technical Field

[0001] This invention relates to the field of visual inspection technology, specifically a visual inspection method for intelligent logistics warehouse guide lines based on deep learning. Background Technology

[0002] In intelligent logistics warehouses, visual inspection of guide lines is crucial for achieving precise navigation and efficient operation of logistics equipment. With the rapid development of the logistics industry, the requirements for automation and intelligence are constantly increasing. Deep learning, with its powerful feature learning capabilities, can uncover complex patterns from massive amounts of data and is widely used in fields such as image recognition and object detection.

[0003] In traditional visual inspection of guide lines in logistics warehouses, many solutions rely on a single sensor to acquire image information and identify guide lines through simple threshold segmentation and feature matching algorithms. However, warehouse environments are complex and variable, with unstable lighting conditions. Under direct sunlight, guide lines will reflect light, and shadowed areas will change the color and contrast of the guide lines, resulting in blurred guide line features in the image and making it difficult to achieve high-precision detection. Summary of the Invention

[0004] The purpose of this invention is to provide a deep learning-based visual detection method for intelligent logistics warehouse guide lines to solve the problems raised in the prior art.

[0005] To achieve the above objectives, the present invention provides the following technical solution: a deep learning-based visual detection method for intelligent logistics warehouse guide lines, the method comprising the following steps: Step 1: Configure sensors at fixed points in the warehouse or on mobile devices to synchronously collect RGB images, infrared images, and point cloud data; Step 2: Preprocess the RGB image, infrared image, and point cloud data, extract features, and concatenate them into a feature vector; Step 3: Manually label RGB images, label feature vectors based on the labeling results, and integrate data to construct a dataset; Step 4: Train the U-Net model based on the dataset to perform semantic segmentation of the guide lines, and identify the guide line information based on the prediction results; Step 5: Based on the guide line information, construct a grid map by integrating multi-source data, and use the fusion of vision and LiDAR to realize the positioning of logistics handling equipment; Step 6: Use the A* algorithm to plan the path, control the movement of the logistics handling equipment according to the planned path, and adjust the status in real time according to the guide line information.

[0006] In step 1, the sensors include an RGB camera, an infrared camera, and a lidar; the sensors are configured at fixed locations in the warehouse or on logistics mobile devices to collect data. RGB and infrared cameras are used to capture guide lines and surrounding objects in the warehouse environment, acquiring RGB and infrared images respectively; the infrared camera works in conjunction with the RGB camera to acquire images of the same area; the RGB image is represented as: [I1,I2,…,I… a The infrared image is represented as: [i1,i2,…,i b LiDAR is used to acquire spatial depth information of the warehouse and collect point cloud data, represented as: [D1, D2, ..., D...]. c The infrared camera, RGB camera, and LiDAR all have the same acquisition time and interval; where a is a positive integer representing the number of RGB cameras, I1~I a These represent the first to a RGB images captured at a certain moment; b is a positive integer representing the number of infrared cameras, i1 to i... b These represent the first to b infrared images acquired at a given time; c is a positive integer representing the number of lidar sensors, D1 to D... c These represent the point cloud data for groups 1 through c, respectively.

[0007] In step 2, the RGB image is stored in RGB format, with each pixel represented by the values ​​of the red, green, and blue channels; the infrared image is stored in grayscale format, with each pixel representing the infrared radiation intensity at that point; and the point cloud data is stored in binary format, with each point containing three-dimensional coordinates and reflection intensity information. The RGB images are converted to grayscale and normalized; the infrared images are normalized; the point cloud data is downsampled using a voxel grid filtering algorithm, and the point cloud data coordinates are transformed to the world coordinate system in combination with the LiDAR acquisition position and situation. Features of RGB and infrared images are extracted using convolutional neural networks, and point cloud data features are extracted using point cloud feature extraction algorithms; for RGB images I at the same time... o Infrared image i p and point cloud data D q The features are concatenated to form a feature vector; a × b × c feature vectors are formed in total; where o ∈ {1, 2, ..., a}, p ∈ {1, 2, ..., b}, q ∈ {1, 2, ..., c}; o, p, and q represent the sequences of RGB image, infrared image, and point cloud data, respectively. o Represents the o-th RGB image, i p Let D represent the p-th infrared image. q This represents the q-th group of point cloud data.

[0008] In step 3, the acquired RGB image is manually annotated using an image annotation tool. The annotator uses the drawing tools provided by the annotation tool to draw the annotation area along the outline of the guide line to obtain the annotation result. The annotation result includes the annotation category information and pixel coordinate information. The category information includes: the category information includes guide line and non-guide line, the guide line is represented as 1, and the non-guide line is represented as 0; the pixel coordinate information is represented as a series of coordinate points; the pixel within the annotation area is marked as 1, and the pixel outside the annotation area is marked as 0. Identify the RGB images in the dataset before feature vector fusion, and label the feature vectors according to the annotations corresponding to the RGB images; integrate the feature vectors at different times and their corresponding annotation results to construct the dataset.

[0009] In step 4, the U-Net model is used for semantic segmentation detection of the guide lines: The generated fused feature vectors are used as input data and grouped according to a certain batch size. The input data and ground truth labels are normalized. The normalized fused feature vectors are then input into the input layer of the U-Net model, where they receive the feature vectors. The fused feature vectors enter the encoder, where convolutional layers perform convolution operations with the input data to extract feature information at different scales and the features of the guide lines. Pooling layers are used to reduce the size of the feature maps, decrease computation, and compress feature information. After processing by convolutional and pooling layers, the input data is progressively downsampled, and feature information is continuously extracted and compressed. After each layer of the encoder, the feature map of that layer is compared with the corresponding layer of the decoder. The feature maps are skipped through the encoder. In the decoder, upsampling and deconvolution operations are used to gradually restore the feature information extracted by the encoder to the original input size. The deconvolution layer performs convolution operations on the feature maps, which is the opposite of the convolution layer, to further extract and restore the image's detailed information. In each decoder layer, the feature maps passed from the skip connections are combined to fuse and refine the feature information. After processing by the decoder, the feature maps are input to the output layer. The output layer uses the Softmax function to classify each pixel, calculating the probability that each pixel belongs to a guide line or not, and the class with the highest probability is used as the predicted class of that pixel. The dataset is divided to train the model. The output layer results are post-processed by using a threshold segmentation method to convert the probability map into a binary image. A threshold A is set, and when the probability of a pixel belonging to a guide line is greater than the threshold A, the pixel is marked as a guide line; otherwise, it is marked as a non-guide line. A contour detection algorithm is used to extract the contour of the guide line and determine its position, shape, and direction.

[0010] Based on the extracted contour coordinate information, combined with the previously established world coordinate system, the points on the contour are converted into actual coordinates in the world coordinate system to determine the precise position of the guide line in the warehouse space. The shape information of the guide line is obtained based on the geometric features of the contour; Analyze the order and direction of points on the contour, calculate the vectors between adjacent points along the contour, and determine the direction of the guide line based on the direction of the vectors.

[0011] In step 5, the logistics handling equipment acquires guide line information, including its location, shape, and direction; A warehouse map is constructed using point cloud data, RGB images, and infrared image data collected by LiDAR: The point cloud data from the LiDAR is processed, and point cloud stitching and fusion technology is used to integrate point cloud data collected at different locations and times into a complete point cloud map; the location information of the guide lines is marked on the map to form a complete map containing the warehouse environment and guide line information. In the map building process, rasterization technology is used to divide the continuous map space into discrete raster units. Each raster unit represents a fixed-size area for subsequent path planning and search algorithms.

[0012] During operation, logistics handling equipment determines its real-time position on the warehouse map; a positioning method based on the fusion of vision and LiDAR is used, combined with the position information of the guide line and map information to achieve precise positioning of the equipment; In terms of visual positioning, the device uses RGB and infrared cameras installed on it to collect images of the surrounding environment. The image matching algorithm is then used to find the area in the map that is most similar to the currently collected image, thereby determining the device's location on the map. In terms of lidar positioning, the point cloud data collected by the lidar on the device is matched with the point cloud data in the map, and the position and attitude of the device are calculated by iterating the nearest point ICP algorithm. The results of visual positioning and lidar positioning are fused to obtain the final positioning result of the device.

[0013] In step 6, after determining the current location and target location of the device, the A* algorithm is used for path planning: Define an open list and a closed list; the open list stores nodes to be evaluated, and the closed list stores nodes that have already been evaluated; add the grid node corresponding to the current location of the device as the starting node to the open list, and set its g value to 0, the h value to be calculated using the heuristic function Euclidean distance, and the f value to be the sum of the g and h values; where the g value represents the actual cost from the starting node to the current node; and the h value represents the estimated cost from the current node to the target node. Select the node with the smallest f-value from the open list as the current node, remove it from the open list, and add it to the closed list; expand the neighboring nodes of the current node, checking whether the neighboring nodes are within the map range or already in the closed list; when the neighboring node meets the conditions, calculate its g-value, which is the current node's g-value plus the movement cost from the current node to the neighboring node, the h-value is calculated using a heuristic function, and the f-value is the sum of the g-value and the h-value; when the neighboring node is not in the open list, add it to the open list; when the neighboring node is already in the open list, and the newly calculated g-value is less than the original g-value, update its g-value, h-value, and f-value. Repeat the node expansion process described above until the target node is found or the open list is empty. When the target node is found, the optimal driving path is obtained by backtracking from the target node to the starting node. During the backtracking process, the parent node of each node is recorded. Starting from the target node, the path is backtracked step by step along the pointers of the parent nodes to the starting node to obtain the complete path. During path planning, the information of the guide lines is fully considered, and paths along the guide lines are prioritized for searching. This is achieved by setting the weights of the guide lines. The cost of moving along the guide lines is relatively small, making the algorithm more inclined to choose paths related to the guide lines. The weights can be set by staff based on experience. According to the planned route, the logistics handling equipment controls its own speed and steering angle in real time; during the journey, the equipment receives guide line information in real time and adjusts its driving status to ensure that it always travels along the guide line.

[0014] Compared with existing technologies, the beneficial effects of this invention are as follows: This invention integrates multi-source data from RGB cameras, infrared cameras, and LiDAR to acquire guide line information from multiple dimensions. Combined with the U-Net model for semantic segmentation and detection, it can accurately extract guide line features. Even when faced with blurred guide line features due to changes in lighting, stains, or wear, it can accurately identify the position, shape, and direction of the guide line, significantly improving detection accuracy. This invention employs a positioning method that integrates vision and LiDAR, combined with warehouse maps and guide line information, enabling logistics handling equipment to determine its own position in the warehouse in real time and accurately, reducing positioning deviations, ensuring stable equipment operation, and improving the safety and accuracy of logistics operations. Attached Figure Description

[0015] Figure 1 This is a schematic diagram illustrating the steps of the visual detection method for intelligent logistics warehouse guide lines based on deep learning, as described in this invention. Detailed Implementation

[0016] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0017] Example: Figure 1 As shown, this invention provides a technical solution: a deep learning-based visual detection method for intelligent logistics warehouse guide lines. The method includes the following steps: Step 1: Configure sensors at fixed points in the warehouse or on mobile devices to synchronously collect RGB images, infrared images, and point cloud data; Step 2: Preprocess the RGB image, infrared image, and point cloud data, extract features, and concatenate them into a feature vector; Step 3: Manually label RGB images, label feature vectors based on the labeling results, and integrate data to construct a dataset; Step 4: Train the U-Net model based on the dataset to perform semantic segmentation of the guide lines, and identify the guide line information based on the prediction results; Step 5: Based on the guide line information, construct a grid map by integrating multi-source data, and use the fusion of vision and LiDAR to realize the positioning of logistics handling equipment; Step 6: Use the A* algorithm to plan the path, control the movement of the logistics handling equipment according to the planned path, and adjust the status in real time according to the guide line information.

[0018] In step 1, the sensors include an RGB camera, an infrared camera, and a lidar; the sensors are configured at fixed locations in the warehouse or on logistics mobile devices to collect data. RGB and infrared cameras are used to capture guide lines and surrounding objects in the warehouse environment, acquiring RGB and infrared images respectively; the infrared camera works in conjunction with the RGB camera to acquire images of the same area; the RGB image is represented as: [I1,I2,…,I… a The infrared image is represented as: [i1,i2,…,i b LiDAR is used to acquire spatial depth information of the warehouse and collect point cloud data, represented as: [D1, D2, ..., D...]. c The infrared camera, RGB camera, and LiDAR all have the same acquisition time and interval; where a is a positive integer representing the number of RGB cameras, I1~I a These represent the first to a RGB images captured at a certain moment; b is a positive integer representing the number of infrared cameras, i1 to i... bThese represent the first to b infrared images acquired at a given time; c is a positive integer representing the number of lidar sensors, D1 to D... c These represent the point cloud data for groups 1 through c, respectively.

[0019] In step 2, the RGB image is stored in RGB format, with each pixel represented by the values ​​of the red, green, and blue channels; the infrared image is stored in grayscale format, with each pixel representing the infrared radiation intensity at that point; and the point cloud data is stored in binary format, with each point containing three-dimensional coordinates and reflection intensity information. The RGB images are converted to grayscale and normalized; the infrared images are normalized; the point cloud data is downsampled using a voxel grid filtering algorithm, and the point cloud data coordinates are transformed to the world coordinate system in combination with the LiDAR acquisition position and situation. Features of RGB and infrared images are extracted using convolutional neural networks, and point cloud data features are extracted using point cloud feature extraction algorithms; for RGB images I at the same time... o Infrared image i p and point cloud data D q The features are concatenated to form a feature vector; a × b × c feature vectors are formed in total; where o ∈ {1, 2, ..., a}, p ∈ {1, 2, ..., b}, q ∈ {1, 2, ..., c}; o, p, and q represent the sequences of RGB image, infrared image, and point cloud data, respectively. o Represents the o-th RGB image, i p Let D represent the p-th infrared image. q This represents the q-th group of point cloud data.

[0020] In step 3, the acquired RGB image is manually annotated using an image annotation tool. The annotator uses the drawing tools provided by the annotation tool to draw the annotation area along the outline of the guide line to obtain the annotation result. The annotation result includes the annotation category information and pixel coordinate information. The category information includes: the category information includes guide line and non-guide line, the guide line is represented as 1, and the non-guide line is represented as 0; the pixel coordinate information is represented as a series of coordinate points; the pixel within the annotation area is marked as 1, and the pixel outside the annotation area is marked as 0. Identify the RGB images in the dataset before feature vector fusion, and label the feature vectors according to the annotations corresponding to the RGB images; integrate the feature vectors at different times and their corresponding annotation results to construct the dataset.

[0021] In step 4, the U-Net model is used for semantic segmentation detection of the guide lines: The generated fused feature vectors are used as input data and grouped according to a certain batch size. The input data and ground truth labels are normalized. The normalized fused feature vectors are then input into the input layer of the U-Net model, where they receive the feature vectors. The fused feature vectors enter the encoder, where convolutional layers perform convolution operations with the input data to extract feature information at different scales and the features of the guide lines. Pooling layers are used to reduce the size of the feature maps, decrease computation, and compress feature information. After processing by convolutional and pooling layers, the input data is progressively downsampled, and feature information is continuously extracted and compressed. After each layer of the encoder, the feature map of that layer is compared with the corresponding layer of the decoder. The feature maps are skipped through the encoder. In the decoder, upsampling and deconvolution operations are used to gradually restore the feature information extracted by the encoder to the original input size. The deconvolution layer performs convolution operations on the feature maps, which is the opposite of the convolution layer, to further extract and restore the image's detailed information. In each decoder layer, the feature maps passed from the skip connections are combined to fuse and refine the feature information. After processing by the decoder, the feature maps are input to the output layer. The output layer uses the Softmax function to classify each pixel, calculating the probability that each pixel belongs to a guide line or not, and the class with the highest probability is used as the predicted class of that pixel. The dataset is divided to train the model. The output layer results are post-processed by using a threshold segmentation method to convert the probability map into a binary image. A threshold A is set, and when the probability of a pixel belonging to a guide line is greater than the threshold A, the pixel is marked as a guide line; otherwise, it is marked as a non-guide line. A contour detection algorithm is used to extract the contour of the guide line and determine its position, shape, and direction.

[0022] Based on the extracted contour coordinate information, combined with the previously established world coordinate system, the points on the contour are converted into actual coordinates in the world coordinate system to determine the precise position of the guide line in the warehouse space. The shape information of the guide line is obtained based on the geometric features of the contour; Analyze the order and direction of points on the contour, calculate the vectors between adjacent points along the contour, and determine the direction of the guide line based on the direction of the vectors.

[0023] In step 5, the logistics handling equipment acquires guide line information, including its location, shape, and direction; A warehouse map is constructed using point cloud data, RGB images, and infrared image data collected by LiDAR: The point cloud data from the LiDAR is processed, and point cloud stitching and fusion technology is used to integrate point cloud data collected at different locations and times into a complete point cloud map; the location information of the guide lines is marked on the map to form a complete map containing the warehouse environment and guide line information. In the map building process, rasterization technology is used to divide the continuous map space into discrete raster units. Each raster unit represents a fixed-size area for subsequent path planning and search algorithms.

[0024] During operation, logistics handling equipment determines its real-time position on the warehouse map; a positioning method based on the fusion of vision and LiDAR is used, combined with the position information of the guide line and map information to achieve precise positioning of the equipment; In terms of visual positioning, the device uses RGB and infrared cameras installed on it to collect images of the surrounding environment. The image matching algorithm is then used to find the area in the map that is most similar to the currently collected image, thereby determining the device's location on the map. In terms of lidar positioning, the point cloud data collected by the lidar on the device is matched with the point cloud data in the map, and the position and attitude of the device are calculated by iterating the nearest point ICP algorithm. The results of visual positioning and lidar positioning are fused to obtain the final positioning result of the device.

[0025] In step 6, after determining the current location and target location of the device, the A* algorithm is used for path planning: Define an open list and a closed list; the open list stores nodes to be evaluated, and the closed list stores nodes that have already been evaluated; add the grid node corresponding to the current location of the device as the starting node to the open list, and set its g value to 0, the h value to be calculated using the heuristic function Euclidean distance, and the f value to be the sum of the g and h values; where the g value represents the actual cost from the starting node to the current node; and the h value represents the estimated cost from the current node to the target node. Select the node with the smallest f-value from the open list as the current node, remove it from the open list, and add it to the closed list; expand the neighboring nodes of the current node, checking whether the neighboring nodes are within the map range or already in the closed list; when the neighboring node meets the conditions, calculate its g-value, which is the current node's g-value plus the movement cost from the current node to the neighboring node, the h-value is calculated using a heuristic function, and the f-value is the sum of the g-value and the h-value; when the neighboring node is not in the open list, add it to the open list; when the neighboring node is already in the open list, and the newly calculated g-value is less than the original g-value, update its g-value, h-value, and f-value. Repeat the node expansion process described above until a target node is found or the open list is empty. When a target node is found, the optimal driving path is obtained by backtracking from the target node to the starting node. During the backtracking process, the parent node of each node is recorded. Starting from the target node, the path is backtracked step by step along the pointers of the parent nodes to the starting node to obtain the complete path. During the path planning process, the information of the guide lines is fully considered, and the path along the guide lines is prioritized for search. This is achieved by setting the weight of the guide lines. The cost of moving along the guide lines is relatively small, making the algorithm more inclined to choose the paths related to the guide lines. According to the planned route, the logistics handling equipment controls its own speed and steering angle in real time; during the journey, the equipment receives guide line information in real time and adjusts its driving status to ensure that it always travels along the guide line.

[0026] In this embodiment, a deep learning-based intelligent logistics warehouse guide line visual detection solution is introduced in a 5,000-square-meter intelligent logistics warehouse to achieve efficient handling and storage of goods. Multiple guide lines are installed inside the warehouse to guide logistics vehicles to move accurately within the warehouse.

[0027] Three RGB cameras (a=3), two infrared cameras (b=2), and one lidar (c=1) were installed on the logistics transport vehicle; these sensors started data acquisition at the same time, with the acquisition interval set to 0.5 seconds.

[0028] At a certain time t, the RGB camera acquires 3 RGB images, denoted as I1, I2, and I3; the infrared camera acquires 2 infrared images, denoted as i1 and i2; and the lidar acquires 1 set of point cloud data, denoted as D1. Each RGB image has a resolution of 1920×1080 pixels and is stored in RGB format. Each pixel is represented by the values ​​of the red, green, and blue channels, with a value range of 0-255. The infrared image is also 1920×1080 pixels and is stored in grayscale format. The value of each pixel represents the infrared radiation intensity of that point, with a value range of 0-255. The point cloud data contains approximately 10,000 points, each containing three-dimensional coordinates (x, y, z) and reflection intensity information. The coordinate unit is meters, and the reflection intensity value ranges from 0 to 100. RGB image preprocessing: The RGB image is converted to grayscale using the formula Gray=0.2989×R+0.5870×G+0.1140×B; then normalization is performed to map the pixel values ​​of the grayscale image to the range of 0-1. Infrared image preprocessing: Directly normalize the infrared image, mapping pixel values ​​to between 0 and 1; Point cloud data preprocessing: The point cloud data is downsampled using a voxel grid filtering algorithm. The side length of the voxel grid is set to 0.05 meters. After downsampling, the point cloud data is reduced to about 2000 points. Then, combined with the acquisition position (x0, y0, z0) and attitude (rotation angles around the x, y, and z axes are α, β, and γ, respectively), the point cloud data coordinates are transformed to the world coordinate system. Feature extraction: Features of RGB and infrared images are extracted using a convolutional neural network VGG16. After multiple rounds of convolution and pooling operations, feature maps are obtained. For point cloud data, the PointNet algorithm is used to extract features. The features of the RGB image Io (o=1,2,3), the infrared image ip (p=1,2), and the point cloud data D1 at the same time are concatenated to form feature vectors, resulting in a total of 3×2×1=6 feature vectors, each with a dimension of 1024. The acquired RGB images were manually annotated using the LabelImg image annotation tool. Annotators drew the annotation area along the outline of the guide line. The annotation results included the annotation category information (1 for guide line, 0 for non-guide line) and pixel coordinate information. For image I1, the pixel coordinates in the annotation area are [(x1,y1),(x2,y2),...,(xn,yn)], and these coordinate points constitute the outline of the guide line. According to the annotation classification of the RGB images, the corresponding feature vectors were annotated. The feature vectors and annotation results at different times were integrated to construct a dataset. The final dataset contains 1000 sets of data, of which 800 sets were used for training and 200 sets were used for testing.

[0029] Model training: The generated fused feature vectors were used as input data and grouped into batches of 32. The input data and the true labels were normalized to be distributed between 0 and 1. The normalized fused feature vectors were then input into the input layer of the U-Net model. The model used the Adam optimizer with a learning rate of 0.001 and a cross-entropy loss function. After 50 epochs of training, the model achieved an accuracy of 95% on the validation set.

[0030] Prediction and Post-processing: The test set data is input into the trained model. The output layer uses the Softmax function to classify each pixel, calculating the probability that each pixel belongs to a guide line and not a guide line. A threshold A=0.6 is set. When the probability of a pixel belonging to a guide line is greater than the threshold A, the pixel is marked as a guide line; otherwise, it is marked as a non-guide line. The contour of the guide line is extracted using the findContours function in OpenCV to obtain the position, shape, and direction information of the guide line.

[0031] Map Building: The logistics handling vehicle uses point cloud data, RGB images, and infrared image data collected by LiDAR to build a warehouse map; through point cloud stitching and fusion technology, point cloud data collected at different locations and times are integrated into a complete point cloud map; using rasterization technology, the map space is divided into raster units with a side length of 0.1 meters, and each raster unit represents an area of ​​a fixed size; the location information of the guide lines is marked on the map to form a complete map containing warehouse environment and guide line information.

[0032] Equipment Positioning: During operation, the logistics transport vehicle employs a fusion positioning method combining visual and LiDAR technologies. For visual positioning, images of the surrounding environment are captured by the vehicle's RGB and infrared cameras, with a resolution of 1920×1080 pixels. The ORB feature matching algorithm is used to find the region in the map most similar to the currently captured image, thus determining the vehicle's position on the map. For LiDAR positioning, point cloud data collected by the LiDAR on the vehicle is matched with point cloud data in the map. The nearest point (ICP) algorithm is used to calculate the vehicle's position and orientation. Finally, the positioning results from both methods are fused to obtain the final positioning of the equipment.

[0033] Path planning: After determining the current position (10,20) and target position (50,80) of the logistics transport vehicle, the A* algorithm is used for path planning. An open list and a closed list are defined. The grid node corresponding to the current position of the device is added to the open list as the starting node, and its g value is set to 0. The h value is calculated using the heuristic function Euclidean distance: h = sqrt((target x - current x)^2 + (target y - current y)^2). The f value is the sum of the g and h values. The node with the smallest f value is selected from the open list as the current node, removed from the open list, and added to the closed list. The adjacent nodes of the current node are expanded, and it is checked whether the adjacent nodes are... Within the map area or whether it is already in the closed list; when an adjacent node meets the conditions, calculate its g value, the current node's g value plus the movement cost from the current node to the adjacent node (the movement cost is set to 1), the h value is calculated through a heuristic function, and the f value is the sum of the g value and the h value; when an adjacent node is not in the open list, add it to the open list; when an adjacent node is already in the open list and the newly calculated g value is less than the original g value, update its g value, h value, and f value; repeat the above node expansion process until the target node is found or the open list is empty; after multiple iterations, the target node is finally found, and the optimal driving path is obtained by backtracking from the target node to the starting node.

[0034] Driving Control: Based on the planned path, the logistics transport vehicle controls its speed and steering angle in real time. During operation, the equipment receives guide line information and adjusts its driving status accordingly to ensure it always follows the guide line. Assuming the maximum speed of the logistics transport vehicle is 1 meter per second and the steering angle range is -90° to 90°, when the vehicle deviates from the guide line, the steering angle is adjusted according to the degree of deviation to bring the vehicle back onto the guide line.

[0035] Through the above embodiments, this solution can effectively improve the guide line detection accuracy and positioning accuracy of logistics handling vehicles in practical applications, realize efficient path planning and driving control, and improve the operational efficiency of intelligent logistics warehouses.

[0036] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the invention can be implemented in other specific forms without departing from its spirit or essential characteristics. Therefore, the embodiments should be considered in all respects as exemplary and non-limiting, and the scope of the invention is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be included within the present invention. No reference numerals in the claims should be construed as limiting the scope of the claims.

Claims

1. A deep learning-based visual inspection method for guide lines in intelligent logistics warehouses, characterized by: The method includes the following steps: Step 1: Configure sensors at fixed points in the warehouse or on mobile devices to synchronously collect RGB images, infrared images, and point cloud data; Step 2: Preprocess the RGB image, infrared image, and point cloud data, extract features, and concatenate them into a feature vector; Step 3: Manually label RGB images, label feature vectors based on the labeling results, and integrate data to construct a dataset; Step 4: Train the U-Net model based on the dataset to perform semantic segmentation of the guide lines, and identify the guide line information based on the prediction results; Step 5: Based on the guide line information, construct a grid map by integrating multi-source data, and use the fusion of vision and LiDAR to realize the positioning of logistics handling equipment; Step 6: Use the A* algorithm to plan the path, control the movement of the logistics handling equipment according to the planned path, and adjust the status in real time according to the guide line information; In step 6, after determining the current location and target location of the device, the A* algorithm is used for path planning: Define an open list and a closed list; the open list stores nodes to be evaluated, and the closed list stores nodes that have already been evaluated; add the grid node corresponding to the current location of the device as the starting node to the open list, and set its g value to 0, the h value to be calculated using the heuristic function Euclidean distance, and the f value to be the sum of the g and h values; where the g value represents the actual cost from the starting node to the current node; and the h value represents the estimated cost from the current node to the target node. Select the node with the smallest f-value from the open list as the current node, remove it from the open list, and add it to the closed list; expand the neighboring nodes of the current node, checking whether the neighboring nodes are within the map range or already in the closed list; when the neighboring node meets the conditions, calculate its g-value, which is the current node's g-value plus the movement cost from the current node to the neighboring node, the h-value is calculated using a heuristic function, and the f-value is the sum of the g-value and the h-value; when the neighboring node is not in the open list, add it to the open list; when the neighboring node is already in the open list, and the newly calculated g-value is less than the original g-value, update its g-value, h-value, and f-value. Repeat the node expansion process described above until a target node is found or the open list is empty. When a target node is found, the optimal driving path is obtained by backtracking from the target node to the starting node. During the backtracking process, the parent node of each node is recorded. Starting from the target node, the path is backtracked step by step along the pointers of the parent nodes to the starting node to obtain the complete path. During the path planning process, the information of the guide lines is fully considered, and the path along the guide lines is prioritized for search. This is achieved by setting the weight of the guide lines. The cost of moving along the guide lines is relatively small, making the algorithm more inclined to choose the paths related to the guide lines. According to the planned route, the logistics handling equipment controls its own speed and steering angle in real time; during the journey, the equipment receives guide line information in real time and adjusts its driving status to ensure that it always travels along the guide line.

2. The visual inspection method for intelligent logistics warehouse guide lines based on deep learning according to claim 1, characterized in that: In step 1, the sensors include an RGB camera, an infrared camera, and a lidar; Sensors are configured in fixed locations in the warehouse or on logistics mobile devices to collect data; RGB and infrared cameras are used to capture guide lines and surrounding objects in the warehouse environment, acquiring RGB and infrared images respectively; the infrared camera works in conjunction with the RGB camera to acquire images of the same area; the RGB image is represented as: [I1,I2,…,I… a The infrared image is represented as: [i1,i2,…,i b LiDAR is used to acquire spatial depth information of the warehouse and collect point cloud data, represented as: [D1, D2, ..., D...]. c The infrared camera, RGB camera, and LiDAR all have the same acquisition time and interval; where a is a positive integer representing the number of RGB cameras, I1~I a These represent the first to a RGB images captured at a certain moment; b is a positive integer representing the number of infrared cameras, i1 to i... b These represent the first to b infrared images acquired at a given time; c is a positive integer representing the number of lidar sensors, D1 to D... c These represent the point cloud data for groups 1 through c, respectively.

3. The deep learning-based visual inspection method for intelligent logistics warehouse guide lines according to claim 2, characterized in that: In step 2, the RGB image is stored in RGB format, with each pixel represented by the values ​​of the red, green, and blue channels; the infrared image is stored in grayscale format, with each pixel representing the infrared radiation intensity at that point; and the point cloud data is stored in binary format, with each point containing three-dimensional coordinates and reflection intensity information. Perform grayscale conversion and normalization on RGB images; Normalize the infrared image; The point cloud data is downsampled using a voxel grid filtering algorithm, and the coordinates of the point cloud data are transformed to the world coordinate system by combining the location and situation of the lidar acquisition. Features of RGB and infrared images are extracted using convolutional neural networks, and point cloud data features are extracted using point cloud feature extraction algorithms; for RGB images I at the same time... o Infrared image i p and point cloud data D q Their features are then concatenated to form a feature vector; A total of a×b×c feature vectors are formed; where o∈{1,2,…,a}, p∈{1,2,…,b}, q∈{1,2,…,c}; o, p, and q represent the sequences of RGB image, infrared image, and point cloud data, respectively. o Represents the o-th RGB image, i p Let D represent the p-th infrared image. q This represents the q-th group of point cloud data.

4. The deep learning-based visual inspection method for intelligent logistics warehouse guide lines according to claim 3, characterized in that: In step 3, the acquired RGB image is manually annotated using an image annotation tool. The annotator uses the drawing tools provided by the annotation tool to draw the annotation area along the outline of the guide line to obtain the annotation result. The annotation result includes the annotation category information and pixel coordinate information. The category information includes guide lines and non-guide lines, with guide lines represented as 1 and non-guide lines represented as 0. The pixel coordinate information is represented as a series of coordinate points. Pixels within the annotation area are marked as 1, and pixels outside the annotation area are marked as 0. Identify the RGB images in the dataset before feature vector fusion, and label the feature vectors according to the annotations corresponding to the RGB images; integrate the feature vectors at different times and their corresponding annotation results to construct the dataset.

5. The deep learning-based visual detection method for intelligent logistics warehouse guide lines according to claim 4, characterized in that: In step 4, the U-Net model is used for semantic segmentation detection of the guide lines: The generated fused feature vectors are used as input data and grouped according to a certain batch size; the input data and the real labels are normalized; the normalized fused feature vectors are input into the input layer of the U-Net model, and the input layer receives the feature vectors. The fused feature vectors are fed into the encoder. In the convolutional layers, convolution operations are performed between the kernel and the input data to extract feature information at different scales and the features of the guide lines. Pooling layers are used to reduce the size of the feature maps, reduce computation, and compress feature information. After processing by the convolutional and pooling layers, the input data is progressively downsampled, and feature information is continuously extracted and compressed. After each layer of the encoder, the feature map of that layer is skip-connected with the feature map of the corresponding layer in the decoder. In the decoder, upsampling and deconvolution operations are used to progressively restore the feature information extracted by the encoder to the original input size. The deconvolutional layer performs convolution operations on the feature maps through the opposite operation of the convolutional layers to further extract and restore the image's detailed information. In each layer of the decoder, the feature maps passed from the skip connections are combined to fuse and refine the feature information. After processing by the decoder, the feature maps are input to the output layer. The output layer uses the Softmax function to classify each pixel, calculating the probability that each pixel belongs to a guide line or not, and the class with the highest probability is used as the predicted class for that pixel. The dataset is divided to train the model. The output layer results are post-processed by using a threshold segmentation method to convert the probability map into a binary image; A threshold A is set. When the probability of a pixel belonging to a guide line is greater than the threshold A, the pixel is marked as a guide line; otherwise, it is marked as a non-guide line. A contour detection algorithm is used to extract the contour of the guide line and determine its position, shape, and direction.

6. The deep learning-based visual inspection method for intelligent logistics warehouse guide lines according to claim 5, characterized in that: In step 4, based on the extracted contour coordinate information and combined with the previously established world coordinate system, the points on the contour are converted into actual coordinates in the world coordinate system to determine the precise position of the guide line in the warehouse space. The shape information of the guide line is obtained based on the geometric features of the contour; Analyze the order and direction of points on the contour, calculate the vectors between adjacent points along the contour, and determine the direction of the guide line based on the direction of the vectors.

7. The visual inspection method for intelligent logistics warehouse guide lines based on deep learning according to claim 6, characterized in that: In step 5, the logistics handling equipment acquires guide line information, including its location, shape, and direction; A warehouse map is constructed using point cloud data, RGB images, and infrared image data collected by LiDAR: The point cloud data from the LiDAR is processed, and point cloud stitching and fusion technology is used to integrate point cloud data collected at different locations and times into a complete point cloud map; the location information of the guide lines is marked on the map to form a complete map containing the warehouse environment and guide line information. In the map building process, rasterization technology is used to divide the continuous map space into discrete raster units. Each raster unit represents a fixed-size area for subsequent path planning and search algorithms.

8. The deep learning-based visual detection method for intelligent logistics warehouse guide lines according to claim 7, characterized in that: In step 5, the logistics handling equipment determines its position on the warehouse map in real time during its operation; a positioning method based on vision and LiDAR fusion is used, which combines the position information of the guide line and map information to achieve precise positioning of the equipment; In terms of visual positioning, the device uses RGB and infrared cameras installed on it to collect images of the surrounding environment. The image matching algorithm is then used to find the area in the map that is most similar to the currently collected image, thereby determining the device's location on the map. In terms of lidar positioning, the point cloud data collected by the lidar on the device is matched with the point cloud data in the map, and the position and attitude of the device are calculated by iterating the nearest point ICP algorithm. The results of visual positioning and lidar positioning are fused to obtain the final positioning result of the device.

Images