Digital grid map generation method and system based on deep learning

Abstract

The invention discloses a digital grid map generation method and system based on deep learning, and the method comprises the steps: obtaining a grid data set M1 with actual position coordinate data of an original ground data set H corresponding to each camera, carrying out the coordinate assignment of the remaining grids without the actual position coordinate data, obtaining an original grid data set U shot by the camera, aggregating grid coordinates and grid labels of the original grid data set U shot by the camera, and establishing a grid coordinate data set K; generating a warehouse static grid map based on grid coordinates and grid types of the grid coordinate data set K; based on an improved YOLOv8 algorithm model, a camera shoots an original grid data set U and a warehouse static grid map to obtain real-time operation state information of all forklifts in a warehouse, a dynamic grid map is generated based on the real-time operation state information, and the problem that in the prior art, the cost is high in intelligent transformation is solved.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a method and system for generating a digital raster map based on deep learning. Background Art

[0002] In the development process of today's warehousing industry, many warehousing companies are still stuck in the dilemma of traditional operation mode; on the one hand, a large number of companies still rely on traditional forklifts to carry out cargo handling and adopt the method of ground-pile cargo management, which not only consumes a lot of manpower, but also lacks scientificity in cargo storage layout, and has low efficiency in searching and handling. On the other hand, although the trend of intelligent transformation of warehouses is becoming increasingly prominent, advanced equipment such as smart forklifts and smart shelves have great potential to improve operating efficiency and optimize management processes, but companies are facing high cost thresholds. The purchase of smart forklifts ranges from hundreds of thousands of yuan to millions of yuan per unit. The construction of smart shelves and the deployment of supporting systems are also expensive. The high initial investment and subsequent maintenance costs have discouraged many warehousing companies, and the intelligent upgrade plan has been repeatedly shelved.

[0003] At the same time, digital raster maps (DRGs), as a type of graphic data that has been digitally scanned and computer-processed and presented in raster form, have great application value in the field of intelligent warehousing and can provide accurate digital models for warehousing spaces.

[0004] Therefore, there is an urgent need for a method that focuses on the existing equipment and sites of warehousing enterprises, generates digital raster maps by combining advanced image recognition technology, and then unlocks multiple scenario applications such as forklift intelligent navigation, warehouse intelligent operation scheduling management, and warehouse digital twins, striving to resolve the current problems of high cost and low operating efficiency of intelligent transformation of traditional warehousing enterprises in a low-cost and high-efficiency manner. Summary of the invention

[0005] In view of this, the present invention proposes a digital grid map generation method and system based on deep learning, which can utilize existing warehousing equipment and sites to generate static and dynamic grid maps at low cost, improve warehouse operation efficiency, and help traditional warehousing companies achieve intelligent management.

[0006] To achieve the above object, the present invention provides the following technical solutions:

[0007] A method for generating a digital raster map based on deep learning, comprising:

[0008] The warehouse is divided into areas and cameras are installed and fixed, and the original ground image dataset H before and after the small flag is placed by each camera is obtained;

[0009] Based on the improved YOLOv8 algorithm model, identify the grids in the ground image dataset F, obtain the pixel area of ​​each grid in the image and generate a unique serial number, and mark the grid as a lane grid or a storage location grid based on the grid color;

[0010] Differences are made between the original ground image dataset H and the ground image dataset F corresponding to each camera before and after the small flag is placed, to determine the pixel area and coordinate data of the small flag;

[0011] Record the grid number set, adjacent grid number set and adjacent relationship set of the images in the ground image dataset F;

[0012] The pixel area of ​​each grid is summed up, and the difference is compared with the pixel area of ​​the image in the original ground image data set H, to obtain at least one blank pixel area, define the blank pixel area as a new grid and generate a unique serial number, determine the adjacent relationship of the new grid, obtain the small flag coordinate data of the small flag pixel area contained in the blank pixel area, use the small flag coordinate data as the actual position coordinate data of the grid corresponding to the blank pixel area, and assign all the small flag coordinate data to the corresponding grid, so as to obtain all the grids, grid serial numbers, grid adjacent relationships and actual position coordinate data of some grids of the original ground data set H captured by each camera;

[0013] Obtain a grid data set M1 with actual position coordinate data of the original ground data set H corresponding to each camera, assign coordinates to the remaining grids without actual position coordinate data to obtain the original grid data set U photographed by the camera, aggregate the grid coordinates and grid labels of the original grid data set U photographed by the camera, deduplicate the grid coordinate data and grid labels of the aggregated data to establish a grid coordinate data set K, and establish an association relationship between the grid coordinate data set K and the original grid data set U photographed by the camera;

[0014] Generate a static grid map of the warehouse based on the grid coordinates and grid type of the grid coordinate data set K, and generate a static grid map of each camera based on the association between the grid coordinate data set K and the original grid data set U captured by each camera;

[0015] Based on the improved YOLOv8 algorithm model, the original grid data set U captured by the camera and the static grid map of the warehouse, the real-time operating status information of all forklifts in the warehouse is obtained, and a dynamic grid map is generated based on the real-time operating status information.

[0016] On the basis of the above technical solution, the present invention can also be improved as follows:

[0017] Optionally, the improved YOLOv8 algorithm model includes an input end, a backbone network, a Neck end and an output end;

[0018] A channel prior convolutional attention module is added to the backbone network, wherein the channel prior convolutional attention module includes a channel attention mechanism and a spatial attention mechanism connected in sequence, a channel attention map is generated through the channel attention mechanism, and a spatial attention map is generated through the spatial attention mechanism;

[0019] The Neck end includes a bottleneck module, and the convolution module in the bottleneck module is replaced by DCNv3;

[0020] The spatial attention map is received by DCNv3 for channel mixing to obtain the channel mixing result, and the channel mixing structure is element-wise multiplied with the channel prior to obtain the feature representation of the image content.

[0021] Optionally, generating a channel attention map by a channel attention mechanism and generating a spatial attention map by spatial attention includes:

[0022] Calculate channel attention using formula (1);

[0023] A c (F)=σ(mlp(avgpool(F))+mlp(maxpool(F))) Formula (1);

[0024] In the formula, A c (F) is channel attention, F is a given feature map F, avgpool(F) is the average pooling operation on the feature map F, maxpool(F) is the maximum pooling operation on the feature map F, mlp is a multi-layer perceptron, and σ is the activation function;

[0025] Calculate spatial attention using formula (2);

[0026]

[0027] In the formula, A t (F) is spatial attention, dwconv is depth-wise separable convolution; branch i is the i-th branch.

[0028] Optionally, receiving the spatial attention map through DCNv3 to perform channel mixing to obtain a channel mixing result, and multiplying the channel mixing structure by the channel prior element by element to obtain a feature representation of the image content, including:

[0029] The feature representation of the image content is calculated by formula (3);

[0030]

[0031] In the formula, y(p0) is the feature representation of the image content, M is the number of clusters, N is the dimension, and for the mth group, w m ∈R is the mth group of position-independent projection weights, h mn ∈R is the modulation scalar of the nth sampling point of the mth group, x m ∈R is the slice input feature map, P0 is the current pixel, p n is the grid sampling position, is the mth group of grid sampling positions p n The corresponding offset.

[0032] Optionally, the step of obtaining a grid data set M1 having actual position coordinate data of the original ground data set H corresponding to each camera, and assigning coordinates to the remaining grids without actual position coordinate data to obtain the original grid data set U captured by the camera includes:

[0033] For coordinates (X M11 , Y M11 ) of the grid M11, obtain the left adjacent grid of the grid M11, and determine whether the left adjacent grid has coordinates. If yes, no processing is performed; if no, define (X M11 -1, Y M11 ) is the coordinate of the left adjacent grid, and the left adjacent grid is recorded in the data set M2;

[0034] Similarly, the right adjacent grid, the upper adjacent grid and the lower adjacent grid of the grid M11 are obtained, and the assignment of the actual position coordinate data of the grid in the grid data set M1 to the adjacent grid is completed;

[0035] The actual position coordinate data of the grids in the newly generated data set M2 are assigned to adjacent grids, and the assigned grids are recorded in the data set M3 until the coordinate assignment of the grids in the original ground image of each camera is completed. All grid numbers, grid coordinates, grid labels, pixel areas corresponding to the grids, and camera numbers in the shooting area of ​​each camera are obtained to obtain the original grid data set U shot by the camera.

[0036] Optionally, the real-time operation status information of all forklifts in the warehouse is obtained based on the improved YOLOv8 algorithm model, the original grid data set U captured by the camera, and the static grid map of the warehouse, and the dynamic grid map is generated based on the real-time operation status information, including:

[0037] The pixel area of ​​each forklift in the current camera image and the marking number of the forklift head direction are obtained by improving the YOLOv8 algorithm model;

[0038] Compare the pixel area of ​​the forklift in the current camera image with the pixel area corresponding to the grid in the original grid data set U captured by the current camera, match the corresponding grids, and if the match is a grid, record the pixel area value of the matching grid to obtain the pixel area to be compared;

[0039] If multiple grids are matched, the pixel areas of the multiple grids are merged to obtain the pixel area to be compared;

[0040] Obtaining coordinate data and head direction data of each forklift based on the pixel area to be compared;

[0041] Generate a dynamic grid map based on the current warehouse static grid map and the coordinate data and head direction data of each forklift.

[0042] Optionally, obtaining coordinate data of each forklift based on the pixel area to be compared includes:

[0043] Calculate the forklift horizontal coordinate by formula (4);

[0044]

[0045] In the formula, X n is the horizontal coordinate of the forklift, X is the horizontal coordinate of the pixel area to be compared, X1 is the distance ratio between the overlap area and the left side, and X2 is the distance ratio between the overlap area and the right side;

[0046] Calculate the forklift ordinate by formula (5);

[0047]

[0048] Where Y n is the ordinate of the forklift, Y is the ordinate of the pixel area to be compared, Y1 is the ratio of the distance between the overlapping area and the upper edge, and Y2 is the ratio of the distance between the overlapping area and the lower edge.

[0049] A digital raster map generation system based on deep learning, comprising:

[0050] The data acquisition module is used to divide the warehouse into areas and install and fix cameras, and obtain the original ground image dataset H of each camera before placing the small flag and the ground image dataset F after placing the small flag;

[0051] The grid recognition module is used to identify the grids in the images of the ground image dataset F based on the improved YOLOv8 algorithm model, obtain the pixel area of ​​each grid in the image and generate a unique serial number, and mark the grid as a lane grid or a storage location grid based on the grid color;

[0052] The difference calculation module is used to make a difference between the original ground image data set H and the ground image data set F before and after the small flag is placed corresponding to each camera, and determine the pixel area and coordinate data of the small flag;

[0053] A recording module, used to record the grid sequence number set, adjacent grid sequence number set and adjacent relationship set of the images in the ground image dataset F;

[0054] The actual position coordinate data acquisition module is used to sum up the pixel area of ​​each grid, perform difference comparison with the pixel area of ​​the image in the original ground image data set H, obtain at least one blank pixel area, define the blank pixel area as a new grid and generate a unique serial number, determine the adjacent relationship of the new grid, obtain the small flag coordinate data of the small flag pixel area contained in the blank pixel area, use the small flag coordinate data as the actual position coordinate data of the grid corresponding to the blank pixel area, and assign all the small flag coordinate data to the corresponding grid, so as to obtain the actual position coordinate data of all grids, grid serial numbers, grid adjacent relationships and part of the grids captured by each camera in the original ground data set H;

[0055] A coordinate assignment module is used to obtain a grid data set M1 with actual position coordinate data of the original ground data set H corresponding to each camera, and to assign coordinates to the remaining grids without actual position coordinate data to obtain an original grid data set U captured by the camera, aggregate the grid coordinates and grid labels of the original grid data set U captured by the camera, perform deduplication processing on the grid coordinate data and grid labels of the aggregated data to establish a grid coordinate data set K, and establish an association relationship between the grid coordinate data set K and the original grid data set U captured by the camera;

[0056] A static map generation module, used to generate a static grid map of the warehouse based on the grid coordinates and grid type of the grid coordinate data set K, and to generate a static grid map of each camera based on the association between the grid coordinate data set K and the original grid data set U captured by each camera;

[0057] The dynamic map generation module is used to obtain the real-time operation status information of all forklifts in the warehouse based on the improved YOLOv8 algorithm model, the original grid data set U captured by the camera and the static grid map of the warehouse, and generate a dynamic grid map based on the real-time operation status information.

[0058] An electronic device comprises a memory, a processor and a computer program stored in the memory and running on the processor, wherein the processor implements the steps of the method when executing the computer program.

[0059] A non-transitory computer-readable storage medium stores a computer program, which implements the steps of the method when executed by a processor.

[0060] The present invention has the following advantages:

[0061] The digital raster map generation method based on deep learning in the present invention can carry out map generation work by utilizing the existing space in the warehouse and basic equipment such as conventional cameras, without the need to purchase new expensive intelligent hardware on a large scale. This effectively reduces the high cost of hardware facility investment for traditional warehousing companies to move towards intelligent management, enabling more warehousing companies to embark on the road of intelligent transformation at a lower cost.

[0062] The digital grid map generation method based on deep learning in the present invention, with the help of the improved YOLOv8 algorithm model, can accurately identify the grids in the ground picture, accurately distinguish the lane grids from the storage location grids, carefully process the blank pixel areas into new grids and reasonably determine their related attributes and coordinates. The static grid map generated through a rigorous data processing flow can highly restore the actual layout of the warehouse and provide accurate digital space reference for subsequent management.

[0063] The digital grid map generation method based on deep learning in the present invention further generates a dynamic grid map based on the generated static grid map and the acquisition of real-time operating status information of the forklift. It can grasp the forklift dynamics, cargo flow, etc. in real time, realize intelligent navigation and intelligent operation scheduling management, optimize the forklift driving path, reduce cargo handling time, avoid operation conflicts, significantly improve the overall operation efficiency of the warehouse, and enhance the fluency and orderliness of warehousing operations.

[0064] The digital grid map generation method based on deep learning in the present invention promotes the transformation of traditional warehousing enterprises from the traditional management mode to the digital twin intelligent management mode, which facilitates enterprises to better carry out inventory monitoring, resource allocation, planning and decision-making, etc., improves the level of refinement and intelligence of enterprises in warehousing management, and enhances market competitiveness. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] For the purpose of illustration and not limitation, the present invention is now described in conjunction with the embodiments of the present invention and the accompanying drawings, in which:

[0066] Figure 1 A schematic diagram of a process for generating a digital grid map based on deep learning in an embodiment of the present invention;

[0067] Figure 2 A schematic diagram of the main components of a digital grid map generation system based on deep learning in an embodiment of the present invention;

[0068] Figure 3A first schematic diagram of dividing a warehouse into regions and installing and fixing cameras in an embodiment of the present invention;

[0069] Figure 4 A second schematic diagram of the area division and camera installation and fixation of a warehouse in an embodiment of the present invention;

[0070] Figure 5 A schematic diagram of the server environment configuration used in the experiment in the embodiment of the present invention;

[0071] Figure 6 A schematic diagram of a model improvement ablation test in an embodiment of the present invention;

[0072] Figure 7 It is a schematic diagram of comparison of warehouse forklift detection model results in an embodiment of the present invention;

[0073] Figure 8 A schematic diagram of a static grid map in an embodiment of the present invention;

[0074] Fig. 9 A schematic diagram of a forklift roof marking design in an embodiment of the present invention;

[0075] Fig.10 A schematic diagram of a dynamic grid map in an embodiment of the present invention;

[0076] Fig.11 This is a schematic diagram of the physical structure of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0077] In order to enable those skilled in the art to better understand the solution of the present invention, the technical solution in the implementation of the present invention will be clearly and completely described below in conjunction with the drawings in the implementation of the present invention. Obviously, the described implementation is only a part of the implementation of the present invention, not all of the implementations. Based on the implementation of the present invention, all other implementations obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0078] It should be noted that the terms "first", "second", etc. in the specification of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so as to describe the embodiments of the present invention described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0079] It should be noted that, in the absence of conflict, the embodiments of the present invention and the features in the embodiments can be combined with each other. The embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0080] Figure 1 Schematic diagram of the process of the method for generating a digital grid map based on deep learning in an embodiment of the present invention. Figure 1 As shown, the deep learning-based digital grid map generation method provided in an embodiment of the present invention includes the following steps S101 to S108.

[0081] S101, divide the warehouse into areas and install and fix cameras, and obtain the original ground image dataset H of each camera before placing a small flag and the ground image dataset F after placing the small flag.

[0082] Specifically, Figure 3 As shown in the figure, the warehouse is divided into areas and the cameras are installed and fixed: For the camera installed on the top of the warehouse, For the storage location of goods, close to the storage location A lane for AGV and other forklifts to travel;

[0083] The camera installation is determined based on the site's floor height, area, environment, etc. The installation requirements are: the area that all cameras can capture includes all warehouse goods, each lane, and other areas, and the shooting angle of each camera is finally determined and fixed, so that the camera can subsequently obtain valid images for generating warehouse digital twins;

[0084] The entire warehouse area is gridded and divided into storage areas, and different colors are used to distinguish storage areas, lanes, etc. The left and bottom of the warehouse grid are marked as follows: Figure 3 The numbers are used to facilitate the generation of coordinate comparisons of the grids in each region.

[0085] S102, identifying the grids of the image in the ground image dataset F based on the improved YOLOv8 algorithm model, obtaining the pixel area of ​​each grid in the image and generating a unique serial number, and marking the grid as a lane grid or a storage location grid based on the grid color.

[0086] Specifically, in actual warehouse application scenarios, due to the influence of uncontrollable factors such as light, angle, and surrounding environment, the quality of the collected images is not high, resulting in difficulty in identifying some targets, generally including: image blur caused by changes in the lighting environment, etc.; due to shooting angle problems, dense targets cause objects to be blocked; due to reasons such as target movement and acquisition distance, the size and shape of the target are not fixed, etc. To solve these problems, the present invention improves the algorithm based on YOLOv8n; the improvement is mainly reflected in two points: on the one hand, a channel prior convolutional attention (Channel Prior Convolutional Attention, CPCA) module is added to the backbone network, which is an attention mechanism for enhancing feature representation and dynamically allocating attention weights. CPCA combines channel attention and spatial attention, effectively extracts spatial relationships and retains channel priors through multi-scale deep separable convolution modules; on the other hand, DCNv3 (Deformable Convolutional Networks v3) is an extension of the traditional convolutional neural network, aiming to improve the flexibility and accuracy of feature extraction. DCNv3 introduces deformable convolutions to enable convolution kernels to adapt to different shapes and scales, thereby better capturing non-rigid objects and complex structures in images. DCNv3 enhances the adaptability of convolution kernels by learning adaptive sampling positions in each convolution layer, thereby improving the robustness to object deformation and the network's expressiveness. This method performs well in image detection and segmentation tasks.

[0087] In warehouse image recognition application scenarios, due to factors such as changes in lighting environment or angle, problems such as image blur or target occlusion may occur, increasing the difficulty of image recognition. Channel Prior Convolutional Attention (CPCA) has the ability to focus on information channels and important areas. By adopting a multi-scale deep convolution module, it can effectively extract spatial relationships while retaining the prior channels.

[0088] The overall structure of CPCA consists of the sequential placement of channel attention (CA) and spatial attention (SA). The spatial information of the feature map is aggregated by the channel attention through operations such as average pooling and max pooling. Subsequently, the spatial information is processed by a shared MLP (Multi-layer Perceptron) and added to generate the channel attention map. The channel prior is obtained by element-wise multiplication of the input features and the channel attention map. Subsequently, the channel prior is input into the deep convolution module to generate the spatial attention map. The convolution module receives the spatial attention map for channel mixing. Finally, the refined features are obtained as output by element-wise multiplication of the channel mixing result and the channel prior. The channel mixing process helps to enhance the representation of the features.

[0089] The input is an intermediate feature map F∈R C×H×W , First, the one-dimensional channel attention map M is calculated by CA c ∈R C×1×1 ; Secondly, M c Multiply the input feature F element by element to obtain the refined feature F with channel attention c ∈R C×H×W ; Then, the 3D spatial attention map M is calculated through SA c ∈R C×H×W ; Finally, M t With F c Multiply element by element to get the output features The overall attention calculation formula is as follows:

[0090]

[0091] Where: Indicates element-by-element multiplication; A c represents channel attention; A t represents spatial attention; F represents a given feature map; Represents the output feature map.

[0092] The main function of CA is to generate channel attention maps, which is achieved by aggregating channel relationships in feature maps. Average pooling and maximum pooling are used to aggregate spatial information in feature maps. The spatial information is processed and added through a shared multi-layer perceptron to generate channel attention maps.

[0093] Calculate channel attention using formula (1);

[0094] A c (F)=σ(mlp(avgpool(F))+mlp(maxpool(F))) Formula (1);

[0095] In the formula, A c (F) is channel attention, F is a given feature map F, avgpool(F) is the average pooling operation on the feature map F, maxpool(F) is the maximum pooling operation on the feature map F, mlp is a multi-layer perceptron, and σ is the activation function;

[0096] The main function of SA is to extract spatial feature information. In this structure, a multi-scale structure is used to enhance the ability of convolution operations to capture spatial relationships. Channel mixing is performed by using a 1×1 convolution at the tail of the spatial attention module.

[0097] Calculate spatial attention using formula (2);

[0098]

[0099] In the formula, A t (F) is spatial attention, dwconv is depthwise separable convolution; branch i is the i-th branch.

[0100] Since the convolution kernel is regular, ordinary convolution cannot effectively adapt to the geometric changes of objects when the objects are rotated, symmetrical, or scaled, and the feature extraction effect will deteriorate. Its ability to model geometric transformations mainly comes from extensive data enhancement, large model capacity, and some simple hand-crafted modules (for example, maximum pooling for small translation invariance). A key challenge in visual recognition is how to adapt to geometric changes or model geometric transformations in object scale, posture, viewpoint, and part deformation. Fixed-size convolution kernel operations cannot achieve this effect well.

[0101] In the warehouse recognition application scenario, most targets are not fixed in size and shape due to movement, acquisition distance and other reasons, so the traditional convolution does not work well for feature extraction of images with deformation and different target sizes. Variable convolution can improve the model's ability to handle irregular shapes and deformation problems by predicting the offset for the sampling point and adaptively changing the sampling position. At the same time, the receptive field size can be adaptively changed according to the target size, which effectively solves the problems of traditional convolution in processing images with deformation.

[0102] The present invention uses DCNv3 (deformable convolution v3) to replace the convolution module in the bottleneck module in the YOLOv8 model. As an extension of the DCN series, DCNv3 has the following three advantages:

[0103] This operator makes up for the shortcomings of regular convolution in long-range dependency and adaptive spatial aggregation; compared with common attention-based operators such as MHSA and the closely related deformable attention, this operator inherits the inductive bias of convolution, making the model more efficient with less training data and shorter training time;

[0104] The operator is based on sparse sampling and is more computationally and memory efficient than previous methods such as MHSA and reparameterized large kernels.

[0105] For the current pixel point P0, the feature representation of the image content is calculated by formula (3);

[0106]

[0107] In the formula, y(p0) is the feature representation of the image content, M is the number of clusters, N is the dimension, and for the mth group, w m∈R is the mth group of position-independent projection weights, h mn ∈R is the modulation scalar of the nth sampling point of the mth group, x m ∈R is the slice input feature map, P0 is the current pixel, p n is the grid sampling position, is the mth group of grid sampling positions p n The corresponding offset.

[0108] The dataset used in this paper is the experimental warehouse camera detection image dataset. In order to solve the forklift classification and detection problems in complex environments, a dataset with different angles, light, distance, object intrusion and other scenarios was constructed. Through manual screening and annotation, it is divided into two formats of image classification and target detection. There are 13 categories and a total of 2957 images, which can ultimately provide basic data for tasks such as forklift image classification and target detection. The images are randomly divided into training set, validation set and test set;

[0109] The server operating system used in the experiment is Ubuntu 20.04, the computing resource is a single NVIDIA RTX A5000 graphics card, and the deep learning framework is Pytorch 1.11.0. The detailed configuration is as follows Figure 5 shown.

[0110] After 300 rounds of training on the original model and the improved model, the original model performed better at the beginning because it could load pre-trained weights, and only needed about 200 rounds of training to reach convergence. However, the improved model can only load some pre-trained weights due to the change in model structure, so its effect is equivalent to retraining, and more training rounds are required to reach convergence. By comparing the two curves, it can be found that the improved model improves the mAP50 index by 3.7 percentage points from 0.669 to 0.706 by introducing the CPCA attention mechanism and using DCNv3 instead of traditional convolution.

[0111] mAP is an important evaluation indicator, which refers to the average value of average precision (AP). For each category, its AP value can be calculated, that is, the area under the curve of precision and recall rate when segmented with different thresholds in all samples of the category. Then the average AP value of all categories is the mAP value. The relevant formula is as follows:

[0112]

[0113] In order to verify the effectiveness of the improvements made in the present invention, the following ablation test was performed on the YOLOv8n model. Figure 6It can be seen that the designed DCNv3_C2f Block increases the model mAP value by 3.7 points while reducing the model size by 0.4MB, achieving a double improvement in model accuracy and speed.

[0114] The improved model of the present invention is compared with the target detection models YOLOv3-tiny, YOLOv5, YOLOv6 and YOLOv8n, and the improved model of the present invention is verified to be superior to the YOLOv3-tiny, YOLOv5, YOLOv6 and YOLOv8n models in terms of accuracy and speed, and the model parameter volume and inference time are also lower than the other four models. The effectiveness of the improved method proposed in the present invention is further proved. The improved forklift recognition model is used in the forklift detection of the actual warehouse, and the forklift recognition accuracy and computing efficiency are greatly improved. Figure 7 shown.

[0115] In order to verify the detection results of the algorithm in different environments in actual warehouses, some images with different backgrounds were selected from the experimental test set. It can be found that the improved algorithm can accurately identify forklifts in complex environments such as different light, different angles, and different image backgrounds, and can also compare and identify the direction of the forklift head. The improved algorithm can accurately identify forklifts and their specific information, and can meet the actual scene application.

[0116] S103, taking a difference between the original ground image dataset H and the ground image dataset F corresponding to each camera before and after the small flag is placed, and determining the pixel area of ​​the small flag and the coordinate data of the small flag.

[0117] Specifically, Figure 4 As shown, small flags of different colors are placed in each grid area of ​​the warehouse, and the coordinate data of the position where each small flag is placed is recorded according to the numbers on the left and bottom. The ground image dataset F (F1, F2...Fn) with small flags taken by each camera is obtained.

[0118] S104, recording the grid sequence number set, adjacent grid sequence number set and adjacent relationship set of the images in the ground image dataset F.

[0119] S105, sum up the pixel areas of each grid, perform difference comparison with the pixel areas of the image in the original ground image data set H, obtain at least one blank pixel area, define the blank pixel area as a new grid and generate a unique serial number, determine the adjacent relationship of the new grid, obtain the small flag coordinate data of the small flag pixel area contained in the blank pixel area, use the small flag coordinate data as the actual position coordinate data of the grid corresponding to the blank pixel area, assign all the small flag coordinate data to the corresponding grid, so as to obtain all the grids, grid serial numbers, grid adjacent relationships and actual position coordinate data of some grids captured by each camera in the original ground data set H.

[0120] S106, obtain the grid data set M1 with actual position coordinate data of the original ground data set H corresponding to each camera, assign coordinates to the remaining grids without actual position coordinate data to obtain the original grid data set U photographed by the camera, aggregate the grid coordinates and grid labels of the original grid data set U photographed by the camera, deduplicate the grid coordinate data and grid labels of the aggregated data to establish a grid coordinate data set K, and establish an association relationship between the grid coordinate data set K and the original grid data set U photographed by the camera.

[0121] Specifically, for coordinates (X M11 , Y M11 ) of the grid M11, obtain the left adjacent grid of the grid M11, and determine whether the left adjacent grid has coordinates. If yes, no processing is performed; if no, define (X M11 -1, Y M11 ) is the coordinate of the left adjacent grid, and the left adjacent grid is recorded in the data set M2;

[0122] Similarly, the right adjacent grid, the upper adjacent grid and the lower adjacent grid of the grid M11 are obtained, and the assignment of the actual position coordinate data of the grid in the grid data set M1 to the adjacent grid is completed;

[0123] The actual position coordinate data of the grids in the newly generated data set M2 are assigned to adjacent grids, and the assigned grids are recorded in the data set M3 until the coordinate assignment of the grids in the original ground image of each camera is completed. All grid numbers, grid coordinates, grid labels, pixel areas corresponding to the grids, and camera numbers in the shooting area of ​​each camera are obtained to obtain the original grid data set U shot by the camera.

[0124] S107, generating a static grid map of the warehouse based on the grid coordinates and grid types of the grid coordinate dataset K, and generating a static grid map of each camera based on the association between the grid coordinate dataset K and the original grid dataset U captured by each camera.

[0125] Specifically, the static grid map of the warehouse is the basis for the generation of dynamic grid maps and warehouse digital twins. The static grid map of the present invention is generated based on drawing grids and storage areas in the empty warehouse, obtaining the ground image information captured by each camera, and finally generating the coordinate information of the grids of the area that each camera can capture based on the grid map generation algorithm. Define the actual length and width of each grid based on the actual situation, and then generate forklift driving routes and other applications based on the scheduling task planning; Figure 8 As shown, the light color is the lane grid and the dark color is the storage location grid.

[0126] S108, based on the improved YOLOv8 algorithm model, the original grid data set U captured by the camera and the static grid map of the warehouse, the real-time operation status information of all forklifts in the warehouse is obtained, and a dynamic grid map is generated based on the real-time operation status information.

[0127] Specifically, the pixel area of ​​each forklift in the current camera image and the marking number of the forklift head direction are obtained by improving the YOLOv8 algorithm model;

[0128] Since warehouse forklifts are similar in appearance, the present invention fixes an arrow with the direction of the head of the vehicle and a number or symbol representing the unique forklift on the roof of each forklift to ensure the accuracy and effectiveness of identification, so that each forklift can be identified more accurately with minimal resource cost. Fig. 9 As shown, the roof of each forklift in the warehouse is marked with pattern information that can clearly distinguish different numbers or other symbols to identify the identity information of each forklift; at the same time, the roof is marked with an arrow pattern for identifying the direction of the forklift's head to identify the direction in which the forklift is traveling or parked.

[0129] Compare the pixel area of ​​the forklift in the current camera image with the pixel area corresponding to the grid in the original grid data set U captured by the current camera, match the corresponding grids, and if the match is a grid, record the pixel area value of the matching grid to obtain the pixel area to be compared;

[0130] If multiple grids are matched, the pixel areas of the multiple grids are merged to obtain the pixel area to be compared;

[0131] Obtaining coordinate data and head direction data of each forklift based on the pixel area to be compared;

[0132] Based on the current warehouse static grid map and the coordinate data and head direction data of each forklift, a dynamic grid map is generated, such as Fig.10 shown.

[0133] Calculate the forklift horizontal coordinate by formula (4);

[0134]

[0135] Where, X n is the horizontal coordinate of the forklift, X is the horizontal coordinate of the pixel area to be compared, X1 is the distance ratio between the overlap area and the left side, and X2 is the distance ratio between the overlap area and the right side;

[0136] Calculate the forklift ordinate by formula (5);

[0137]

[0138] Where Y n is the ordinate of the forklift, Y is the ordinate of the pixel area to be compared, Y1 is the ratio of the distance between the overlapping area and the upper edge, and Y2 is the ratio of the distance between the overlapping area and the lower edge.

[0139] Figure 2 Schematic diagram of the main components of the deep learning-based digital grid map generation system in the embodiment of the present invention. Figure 2 As shown, the deep learning-based digital grid map generation system 1 provided in an embodiment of the present invention includes a data acquisition module 10, a grid recognition module 20, a difference calculation module 30, a recording module 40, an actual position coordinate data acquisition module 50, a coordinate assignment module 60, a static map generation module 70 and a dynamic map generation module 80.

[0140] The data acquisition module 10 is used to divide the warehouse into regions and install and fix cameras, and obtain the original ground image dataset H of each camera before placing the small flag and the ground image dataset F after placing the small flag;

[0141] A grid recognition module 20 is used to recognize grids in the image in the ground image dataset F based on the improved YOLOv8 algorithm model, obtain the pixel area of ​​each grid in the image and generate a unique serial number, and mark the grid as a lane grid or a storage location grid based on the grid color;

[0142] The difference calculation module 30 is used to make a difference between the original ground image data set H and the ground image data set F corresponding to each camera before and after the small flag is placed, and determine the pixel area of ​​the small flag and the coordinate data of the small flag;

[0143] The recording module 40 is used to record the grid sequence number set, adjacent grid sequence number set and adjacent relationship set of the image in the ground image dataset F;

[0144] The actual position coordinate data acquisition module 50 is used to sum up the pixel area of ​​each grid, perform difference comparison with the pixel area of ​​the image in the original ground image data set H, obtain at least one blank pixel area, define the blank pixel area as a new grid and generate a unique serial number, determine the adjacent relationship of the new grid, obtain the small flag coordinate data of the small flag pixel area contained in the blank pixel area, use the small flag coordinate data as the actual position coordinate data of the grid corresponding to the blank pixel area, and assign all the small flag coordinate data to the corresponding grid, so as to obtain all the grids, grid serial numbers, grid adjacent relationships and actual position coordinate data of some grids of the original ground data set H captured by each camera;

[0145] A coordinate assignment module 60 is used to obtain a grid data set M1 with actual position coordinate data of the original ground data set H corresponding to each camera, and to assign coordinates to the remaining grids without actual position coordinate data to obtain an original grid data set U captured by the camera, aggregate the grid coordinates and grid labels of the original grid data set U captured by the camera, perform deduplication processing on the grid coordinate data and grid labels of the aggregated data to establish a grid coordinate data set K, and establish an association relationship between the grid coordinate data set K and the original grid data set U captured by the camera;

[0146] A static map generation module 70, for generating a static grid map of the warehouse based on the grid coordinates and grid type of the grid coordinate data set K, and generating a static grid map of each camera based on the association between the grid coordinate data set K and the original grid data set U captured by each camera;

[0147] The dynamic map generation module 80 is used to obtain the real-time operation status information of all forklifts in the warehouse based on the improved YOLOv8 algorithm model, the original grid data set U captured by the camera and the static grid map of the warehouse, and generate a dynamic grid map based on the real-time operation status information.

[0148] Fig.11 A schematic diagram of the physical structure of an electronic device provided in an embodiment of the present invention, such as Fig.11 As shown, the electronic device 90 includes: a processor 901 (processor), a memory 902 (memory) and a bus 903;

[0149] The processor 901 and the memory 902 communicate with each other via the bus 903;

[0150] The processor 901 is used to call the program instructions in the memory 902 to execute the methods provided by the above-mentioned method embodiments, so as to execute the methods provided by the implementation mode of the present invention.

[0151] This embodiment provides a non-transitory computer-readable storage medium, which stores computer instructions. The computer instructions enable a computer to execute the method provided by the embodiment of the present invention.

[0152] A person skilled in the art can understand that all or part of the steps of implementing the above method embodiment can be completed by hardware related to program instructions, and the aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps of the above method embodiment; and the aforementioned storage medium includes: ROM, RAM, magnetic disk or optical disk, etc., various storage media that can store program codes.

[0153] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions may occur depending on design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A method for generating a digital grid map based on deep learning, characterized in that: include: The warehouse is divided into areas and cameras are installed and fixed, and the original ground image dataset H before and after the small flag is placed by each camera is obtained; Based on the improved YOLOv8 algorithm model, identify the grids in the ground image dataset F, obtain the pixel area of ​​each grid in the image and generate a unique serial number, and mark the grid as a lane grid or a storage location grid based on the grid color; Differences are made between the original ground image dataset H and the ground image dataset F corresponding to each camera before and after the small flag is placed, to determine the pixel area and coordinate data of the small flag; Record the grid number set, adjacent grid number set and adjacent relationship set of the images in the ground image dataset F; The pixel area of ​​each grid is summed up, and the difference is compared with the pixel area of ​​the image in the original ground image data set H, to obtain at least one blank pixel area, define the blank pixel area as a new grid and generate a unique serial number, determine the adjacent relationship of the new grid, obtain the small flag coordinate data of the small flag pixel area contained in the blank pixel area, use the small flag coordinate data as the actual position coordinate data of the grid corresponding to the blank pixel area, and assign all the small flag coordinate data to the corresponding grid, so as to obtain all the grids, grid serial numbers, grid adjacent relationships and actual position coordinate data of some grids of the original ground data set H captured by each camera; Obtain a grid data set M1 with actual position coordinate data of the original ground data set H corresponding to each camera, assign coordinates to the remaining grids without actual position coordinate data to obtain the original grid data set U photographed by the camera, aggregate the grid coordinates and grid labels of the original grid data set U photographed by the camera, deduplicate the grid coordinate data and grid labels of the aggregated data to establish a grid coordinate data set K, and establish an association relationship between the grid coordinate data set K and the original grid data set U photographed by the camera; Generate a static grid map of the warehouse based on the grid coordinates and grid type of the grid coordinate data set K, and generate a static grid map of each camera based on the association between the grid coordinate data set K and the original grid data set U captured by each camera; Based on the improved YOLOv8 algorithm model, the original grid data set U captured by the camera and the static grid map of the warehouse, the real-time operating status information of all forklifts in the warehouse is obtained, and a dynamic grid map is generated based on the real-time operating status information.

2. The method for generating a digital grid map based on deep learning according to claim 1, characterized in that: The improved YOLOv8 algorithm model includes an input end, a backbone network, a Neck end and an output end; A channel prior convolutional attention module is added to the backbone network, wherein the channel prior convolutional attention module includes a channel attention mechanism and a spatial attention mechanism connected in sequence, a channel attention map is generated through the channel attention mechanism, and a spatial attention map is generated through the spatial attention mechanism; The Neck end includes a bottleneck module, and the convolution module in the bottleneck module is replaced by DCNv3; The spatial attention map is received by DCNv3 for channel mixing to obtain the channel mixing result, and the channel mixing structure is element-wise multiplied with the channel prior to obtain the feature representation of the image content.

3. The method for generating a digital grid map based on deep learning according to claim 2, characterized in that: The method of generating a channel attention map by using a channel attention mechanism and generating a spatial attention map by using spatial attention includes: Calculate channel attention using formula (1); A c (F)=σ(mlp(avgpool(F))+mlp(max pool(F))) )(1); In the formula, A c (F) is channel attention, F is a given feature map F, avgpool(F) is the average pooling operation on the feature map F, maxpool(F) is the maximum pooling operation on the feature map F, mlp is a multi-layer perceptron, and σ is the activation function; Calculate spatial attention using formula (2); In the formula, A t (F) is spatial attention, dwconv is depthwise separable convolution; branch i is the i-th branch.

4. The method for generating a digital grid map based on deep learning according to claim 2, characterized in that: The DCNv3 receives the spatial attention map to perform channel mixing, obtains the channel mixing result, multiplies the channel mixing structure by the channel prior element by element, and obtains the feature representation of the image content, including: The feature representation of the image content is calculated by formula (3); In the formula, y(p0) is the feature representation of the image content, M is the number of clusters, N is the dimension, and for the mth group, w m ∈R is the mth group of position-independent projection weights, h mn ∈R is the modulation scalar of the nth sampling point of the mth group, x m ∈R is the slice input feature map, P0 is the current pixel, p n is the grid sampling position, is the mth group of grid sampling positions p n The corresponding offset.

5. The method for generating a digital grid map based on deep learning according to claim 1, characterized in that: The method of obtaining a grid data set M1 having actual position coordinate data of the original ground data set H corresponding to each camera, and assigning coordinates to the remaining grids without actual position coordinate data to obtain the original grid data set U captured by the camera, includes: For coordinates (X M11 , Y M11 ) of the grid M11, obtain the left adjacent grid of the grid M11, and determine whether the left adjacent grid has coordinates. If yes, no processing is performed; if no, define (X M11 -1, Y M11 ) is the coordinate of the left adjacent grid, and the left adjacent grid is recorded in the data set M2; Similarly, the right adjacent grid, the upper adjacent grid and the lower adjacent grid of the grid M11 are obtained, and the assignment of the actual position coordinate data of the grid in the grid data set M1 to the adjacent grid is completed; The actual position coordinate data of the grids in the newly generated data set M2 are assigned to adjacent grids, and the assigned grids are recorded in the data set M3 until the coordinate assignment of the grids in the original ground image of each camera is completed. All grid numbers, grid coordinates, grid labels, pixel areas corresponding to the grids, and camera numbers in the shooting area of ​​each camera are obtained to obtain the original grid data set U shot by the camera.

6. The method for generating a digital grid map based on deep learning according to claim 1, characterized in that: The method of acquiring the real-time operation status information of all forklifts in the warehouse based on the improved YOLOv8 algorithm model, the original grid data set U captured by the camera, and the static grid map of the warehouse, and generating a dynamic grid map based on the real-time operation status information includes: The pixel area of ​​each forklift in the current camera image and the marking number of the forklift head direction are obtained by improving the YOLOv8 algorithm model; Compare the pixel area of ​​the forklift in the current camera image with the pixel area corresponding to the grid in the original grid data set U captured by the current camera, match the corresponding grids, and if the match is a grid, record the pixel area value of the matching grid to obtain the pixel area to be compared; If multiple grids are matched, the pixel areas of the multiple grids are merged to obtain the pixel area to be compared; Obtaining coordinate data and head direction data of each forklift based on the pixel area to be compared; Generate a dynamic grid map based on the current warehouse static grid map and the coordinate data and head direction data of each forklift.

7. The method for generating a digital grid map based on deep learning according to claim 6, characterized in that: The obtaining of coordinate data of each forklift based on the pixel area to be compared includes: Calculate the forklift horizontal coordinate by formula (4); Where, X n is the horizontal coordinate of the forklift, X is the horizontal coordinate of the pixel area to be compared, X1 is the distance ratio between the overlap area and the left side, and X2 is the distance ratio between the overlap area and the right side; Calculate the forklift ordinate by formula (5); Where Y n is the ordinate of the forklift, Y is the ordinate of the pixel area to be compared, Y1 is the ratio of the distance between the overlapping area and the upper edge, and Y2 is the ratio of the distance between the overlapping area and the lower edge.

8. A system for generating digital grid maps based on deep learning, characterized in that: include: The data acquisition module is used to divide the warehouse into areas and install and fix cameras, and obtain the original ground image dataset H of each camera before placing the small flag and the ground image dataset F after placing the small flag; The grid recognition module is used to identify the grids in the images of the ground image dataset F based on the improved YOLOv8 algorithm model, obtain the pixel area of ​​each grid in the image and generate a unique serial number, and mark the grid as a lane grid or a storage location grid based on the grid color; The difference calculation module is used to make a difference between the original ground image data set H and the ground image data set F before and after the small flag is placed corresponding to each camera, and determine the pixel area and coordinate data of the small flag; A recording module, used to record the grid sequence number set, adjacent grid sequence number set and adjacent relationship set of the images in the ground image dataset F; The actual position coordinate data acquisition module is used to sum up the pixel area of ​​each grid, perform difference comparison with the pixel area of ​​the image in the original ground image data set H, obtain at least one blank pixel area, define the blank pixel area as a new grid and generate a unique serial number, determine the adjacent relationship of the new grid, obtain the small flag coordinate data of the small flag pixel area contained in the blank pixel area, use the small flag coordinate data as the actual position coordinate data of the grid corresponding to the blank pixel area, and assign all the small flag coordinate data to the corresponding grid, so as to obtain the actual position coordinate data of all grids, grid serial numbers, grid adjacent relationships and part of the grids captured by each camera in the original ground data set H; A coordinate assignment module is used to obtain a grid data set M1 with actual position coordinate data of the original ground data set H corresponding to each camera, and to assign coordinates to the remaining grids without actual position coordinate data to obtain an original grid data set U captured by the camera, aggregate the grid coordinates and grid labels of the original grid data set U captured by the camera, perform deduplication processing on the grid coordinate data and grid labels of the aggregated data to establish a grid coordinate data set K, and establish an association relationship between the grid coordinate data set K and the original grid data set U captured by the camera; A static map generation module, used to generate a static grid map of the warehouse based on the grid coordinates and grid type of the grid coordinate data set K, and to generate a static grid map of each camera based on the association between the grid coordinate data set K and the original grid data set U captured by each camera; The dynamic map generation module is used to obtain the real-time operation status information of all forklifts in the warehouse based on the improved YOLOv8 algorithm model, the original grid data set U captured by the camera and the static grid map of the warehouse, and generate a dynamic grid map based on the real-time operation status information.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A non-transitory computer readable medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images