Image recognition method, agv material sorting method and system

Abstract

The present application belongs to the field of AGV, and particularly relates to an image recognition method, an AGV material sorting method and system, comprising: S1, collecting a target image to form a self-built dataset; S2, segmenting the image and then extracting a dataset with material characteristics and tray information therefrom; S3, labeling and enhancing the image; S4, constructing an improved YOLO-v5 model; S5, training the improved YOLO-v5 model to obtain an optimal training model; and S6, classifying and identifying materials by using the trained YOLO-v5 model, judging tray load, and outputting the type of the material and the load condition of the tray. The present application uses MobileNet-v3 as the Backbone part in the improved YOLO-v5 model, and better solves the problems of large Backbone structure parameter quantity and slow detection speed in the original YOLO-v5 model.

Description

Technical Field

[0001] This invention relates to the field of AGVs, specifically to an image recognition method, an AGV material sorting method and system. Background Technology

[0002] AGV is an abbreviation for Automated Guided Vehicle, which refers to a transport vehicle equipped with electromagnetic or optical automatic guidance devices. It can travel along a prescribed guidance path and has safety protection and various transfer functions. AGV belongs to the category of wheeled mobile robots.

[0003] AGVs (Automated Guided Vehicles) offer advantages such as automated material handling and high efficiency, effectively addressing the significant human resource costs associated with product transportation during production. Consequently, they have found widespread application in industries such as automotive, home appliance manufacturing, tobacco, logistics, 3C electronics, pharmaceuticals, and food. With the high-speed operation of factory production lines, users are placing higher demands on the real-time operational status and safety of AGV systems. AGV management, monitoring, and scheduling systems, capable of real-time status monitoring and task information dissemination and reception, are a crucial core technology of AGV systems. With the advent of Industry 4.0, the era of smart factories is gradually unfolding, and the demand for the perfect integration of intelligent equipment and information technology, along with highly flexible manufacturing, is increasing daily.

[0004] For managing multiple AGVs, an AGV scheduling system is required, including task path planning, task allocation, traffic control, vehicle management, and interface control with the customer's ERP and PDM systems. While existing AGV central scheduling systems can perform basic task scheduling, traffic control, and vehicle status monitoring, they lack flexibility in material handling and are overly reliant on fixed routes. For example, the Chinese invention patent CN106856493, a web-based AGV cluster central management system, still requires specifying the starting position and various parameters when issuing handling instructions, meaning AGVs cannot automatically handle materials in some situations. In specific cases requiring material classification, ordinary classification methods rely on path planning for fixed zoning, still leaving many scenarios requiring human intervention. Traditional AGVs still lack cluster management capabilities; when AGV handling malfunctions, neither AGV manufacturers nor customers can achieve on-site monitoring and online diagnostics, making error diagnosis difficult. Summary of the Invention

[0005] In order to overcome the shortcomings and deficiencies of the existing technology, the present invention provides an image recognition method, an AGV material sorting method and system.

[0006] This invention provides an image recognition method, comprising:

[0007] S1, Collect target images to form a self-built dataset;

[0008] S2, segment the image and then extract the dataset containing material features and pallet information;

[0009] S3, perform image annotation enhancement, and divide the dataset into training set, validation set and test set;

[0010] S4, Build an improved YOLO-v5 model;

[0011] S5, train the improved YOLO-v5 model to obtain the optimal trained model;

[0012] S6 uses the trained YOLO-v5 model to classify and identify materials, determine the load on the pallet, and output the type of material and the load status of the pallet.

[0013] Further, S3 includes:

[0014] The Labelimg software was used to label the categories and locations of items in the images, ensuring that the labeling information corresponded to the labeled images.

[0015] Data augmentation of the labeled images is performed by rotating the images clockwise by several times, then randomly adjusting the brightness and chroma of each image, adding Gaussian noise, and randomly superimposing the images. The corresponding annotation files are generated simultaneously with the image augmentation.

[0016] Further, S4 includes:

[0017] S4-1, an improved MobileNet-v3 network structure is used as the Backbone part in the improved YOLO-v5 model. The improved MobileNet-v3 network structure replaces the SE attention mechanism in the MobileNet-v3 base network Bneck with the pixel-level and channel-level attention mechanism CANet. The Backbone network structure in the improved YOLO-v5 model consists of one 3x3 convolution and a 14-layer bneck structure. The input image first passes through a 3x3 convolution to reduce the feature size, and then enters the 14-layer bneck structure for feature extraction. The bneck structure adopts the form of an inverse residual structure. First, a 1x1 pointwise convolution is used to perform dimensionality increase, expand the feature map channels, and enrich the number of features. Then, a 3x3 convolution operation is performed, and the CANet attention mechanism is selectively added. Finally, a 1x1 convolution is performed to obtain the final feature map.

[0018] S4-2 uses the FPN+PAN network structure design from the original YOLO-v5 model as the neck part of the improved YOLO-v5 model. The feature map obtained through S4-1 conveys feature information from bottom to top through the FPN layer, fusing high-level features with low resolution and high semantic information and low-level features with high resolution and low semantic information from bottom to top, and then enhances the localization information from top to bottom through PAN.

[0019] S4-3, The Transformer detection head is used as the head part in the improved YOLO-v5 model. The Transformer detection head is obtained by improving the head part of the original YOLO-v5 model using Transformer. The feature map obtained in S4-2 is fed into the Transformer detection head. The local information obtained by Transformer is larger than that of CNN to enhance the detection of small objects.

[0020] Further, S5 includes:

[0021] The improved YOLO-v5 model is trained using the training set to update the weight parameters. The optimal weight parameters are then tested using the test set to ensure that the accuracy and real-time performance of the model meet the VGA car recognition standard, ultimately resulting in the best trained model.

[0022] The present invention also provides an AGV material sorting method, comprising:

[0023] Step 1: Log in to the client through the web page login service module, and start the current working mode in the client. At this time, the digital camera starts working and transmits the tray load information captured globally to the backend server through the local area network generated by the wireless AP.

[0024] Step 2: The backend server converts the video frames into images and transmits the converted image data to the image recognition module. The image recognition module includes the improved YOLO-v5 model as described in any of claims 1-4. The image recognition module performs recognition and judgment. If the pallet is found to be loaded, the task management module issues a handling task to the AGV equipped with an industrial camera.

[0025] Step 3: After receiving the handling task, the AGV equipped with an industrial camera calculates the route from the current position of the AGV to the handling area through the path planning module. When the AGV travels to the handling area according to the calculated route, the industrial camera captures material images and transmits the scanned material video information to the backend server.

[0026] Step 4: The backend server converts the acquired real-time video information stream into image information and sends it to the image recognition module. The image recognition module performs recognition and judgment, outputs the material category, and at the same time, calls the path planning module again to calculate the area where the current material category should be stored and plans a path for the AGV. The AGV completes the handling work according to the planned path.

[0027] The present invention also provides an AGV material sorting system for implementing the AGV material sorting method described above, comprising a client, a back-end server, a wireless AP, a digital camera, and an AGV. The client, the back-end server, and the wireless AP are sequentially connected in communication. The wireless AP is connected in communication with the digital camera and the AGV, respectively. The AGV is equipped with an automatically rotating industrial camera.

[0028] Furthermore, the backend server includes a web login service module, an image recognition module, a path planning module, a task management module, and a status monitoring module. The web login service module provides a browser-based login control interface, enabling online control on a PC or on-site control on a mobile device. The image recognition module classifies and identifies materials, determines pallet load, and outputs the type of material and the pallet load status. The task management module assigns and manages tasks for the AGV. The path planning module controls the AGV's operating path and trajectory. The status monitoring module monitors the AGV's operating status.

[0029] The beneficial effects of this invention are as follows:

[0030] 1. Traditional AGV sorting systems and methods still rely on manually setting the starting point of the work, and the automation level of AGV work is not high. However, this invention adopts an image recognition module and monitors the load of the pallet through a digital camera, thereby realizing a global monitoring mode, which greatly saves manpower and material costs. As long as the working mode is turned on, the digital camera will transmit the load of the pallet to the image recognition module of the back-end server for judgment. When the pallet load reaches a certain level, the back-end server issues instructions to the AGV through the task management module, thereby realizing unmanned intelligent logistics.

[0031] 2. This invention uses MobileNet-v3 as the Backbone part in the improved YOLO-v5 model, which better solves the problems of large number of Backbone structure parameters and slow detection speed in the original YOLO-v5 model. Especially on mobile and embedded devices, the complex model in the original YOLO-v5 is difficult to utilize.

[0032] 3. The SE part of the lightweight network MobileNet-v3 is replaced with the CANet attention mechanism, so that each bneck network structure in MobileNet-v3 uses pixel-level and channel-level attention mechanisms, which enhances the extraction of image feature information and can better ensure that the AGV can maintain a high material recognition rate even in some factories with poor environments.

[0033] 4. Using Transformer as the head part in the improved YOLO-v5 model increases the sensitivity for small object detection. Attached Figure Description

[0034] Figure 1 This is a schematic diagram of the structure of an AGV material sorting system according to the present invention;

[0035] Figure 2 This is a schematic diagram of the AGV structure of an AGV material sorting system according to the present invention;

[0036] Figure 3 This is a partial structural diagram of an AGV material sorting system according to the present invention;

[0037] Figure 4 This is a flowchart of an AGV material sorting method according to the present invention;

[0038] Figure 5 This is a flowchart of an image recognition method according to the present invention;

[0039] Figure 6 To improve the Bneck network structure in the Backbone of YOLO-v5;

[0040] Figure 7 This is a diagram of the CANet neural network.

[0041] Figure 8 To improve the neural network graph of YOLO-v5.

[0042] Figure 9 The Transformer network structure diagram in the Neck of YOLO-v5 is used to improve the system.

[0043] In the diagram, 1-client, 2-backend server, 3-wireless AP, 4-digital camera, 5-materials, 6-AGV, 7-U-shaped fixed frame, 8-bearing bracket, 9-industrial camera, 10-servo motor. Detailed Implementation

[0044] The technical solution of the present invention will be further described below with reference to the accompanying drawings and specific embodiments. Obviously, the described embodiments are only some embodiments of the invention, but are not intended to limit the present invention.

[0045] Please see Figures 1-3 An AGV material sorting system includes a client 1, a back-end server 2, a wireless AP 3, a digital camera 4, and an AGV 6. The client, the back-end server, and the wireless AP are connected in sequence. The wireless AP is connected in communication with the digital camera and the AGV respectively. The AGV is equipped with an automatically rotating industrial camera 9.

[0046] The backend server 2 includes a web login service module, an image recognition module, a path planning module, a task management module, and a status monitoring module. The web login service module provides a browser-based login control interface, allowing online control from a PC or on-site control from a mobile device. The image recognition module classifies and identifies materials, determines pallet load, and outputs the type of material and the pallet load status. The task management module assigns and manages tasks for the AGV. The path planning module controls the AGV's operating path and trajectory. The status monitoring module monitors the AGV's operating status.

[0047] The client 1 is divided into a working mode and an observer mode. When the client 1 is set to working mode, the digital camera 4 starts to monitor the loading status of the pallet and transmits the video data to the backend server 2. The image recognition module in the backend server 2, which is used to determine whether the pallet being transported is full, determines whether it needs to be transported. If it is, the task management module in the backend server issues a transport command. The AGV 6 with an industrial camera receives the transport command through the wireless AP 3 and travels to the transport area through the path planning module. Then, the industrial camera 9 transmits the video data of the material to be transported to the image recognition module based on the improved YOLO-v5 for recognition and classification to obtain the basic type of the material being transported. The path planning module is called again to calculate the specific path of the material being transported, and the transport work is completed.

[0048] The AGV 6 equipped with an industrial camera 9 consists of a U-shaped mounting frame 7, a servo motor 10, the industrial camera 9, and a bearing bracket 8. The U-shaped mounting frame 7 is connected to the tail of the AGV 6 and is used to fix the industrial camera 9. The industrial camera is placed inside the U-shaped mounting frame 7, at a 90° angle and flush with the AGV 6. The servo motor 10 is connected to the industrial camera 9 through a bearing in the fixed arm of the bearing bracket 8, allowing it to rotate upwards at a 45° angle, providing sufficient line of sight when scanning large materials. The industrial camera 9 transmits the scanned material information to the image classifier in the backend server 2. If the current material type is found, the next handling operation is performed; otherwise, the servo motor 10 is rotated upwards to adjust the angle until the material type is found.

[0049] Please see Figure 4 An AGV material sorting method includes the following steps:

[0050] Step 1: Log in to client 1 through the web page login service module and start the current working mode in client 1. At this time, digital camera 4 starts working and transmits the tray load status captured globally to the backend server 2 through the local area network generated by wireless AP3.

[0051] Step 2: The backend server 2 acquires the real-time video stream from the video capture section and uses OpenCV to convert the video frames into images, setting the frame interval to convert one image every 60 frames. The converted image data is then passed to the image recognition module for recognition and judgment. If the pallet is detected to be loaded, a handling task is issued to the AGV6 equipped with an industrial camera through the task management module.

[0052] Step 3: After receiving the handling task, the AGV6 equipped with an industrial camera calculates the route from the current position of the AGV6 to the handling area through the path planning module. When the AGV6 travels to the handling area, the servo motor 10 starts to adjust the angle of the industrial camera 9 and transmits the scanned material video information to the backend server 2.

[0053] Step 4: The backend server 2 converts the real-time video information stream from the video acquisition section into image information and sends it to the image recognition module mentioned above. It outputs the material category and calls the path planning module again to calculate the area where the current material category should be stored to complete the handling work.

[0054] Specifically, when an AGV6 equipped with an industrial camera malfunctions, the monitoring video during the malfunction period can be retrieved through the status monitoring module in the backend server 2 to reconstruct the cause of the malfunction and better resolve the problem.

[0055] Please see Figures 5-9 The image recognition module described above employs the following image recognition method, which includes the following steps:

[0056] S1, Target Image Acquisition: This invention uses a self-built dataset containing 5220 images, including common material categories and pallets used in logistics handling. The self-built dataset utilizes the OpenCV framework to convert real-time video streams collected by the backend server into images.

[0057] S2, Image Segmentation: Images converted using the OpenCV framework vary in size. Directly feeding them into the YOLO-v5 model for detection causes the images to adaptively scale to 640x640. Large-scale image scaling leads to the loss of small-sized features. Therefore, to ensure the accuracy and reliability of the dataset, the images need to be segmented into 640x640 pixels, and then the dataset containing material features and pallet information is extracted from them.

[0058] S3, image annotation enhancement, divides the dataset into training, validation, and test sets, including:

[0059] In material classification, to reduce the impact of the testing environment on the results, materials are categorized into general boxes, fragile boxes, food, and fresh produce. Labelimg software is used to label the item category and location information in the images, ensuring that the labeled information corresponds to the labeled images.

[0060] Data augmentation is required for the labeled images. Methods employed include randomly adjusting brightness and chroma, rotating the images 90° clockwise three times, then randomly adjusting the brightness and chroma of each image, adding Gaussian noise, and randomly stacking multiple images. Simultaneously, corresponding annotation files are generated. The final dataset will have a training, validation, and test set ratio of 8:1:1.

[0061] S4, construct the improved YOLO-v5 model, following the steps as follows: Figure 7 As shown, it includes:

[0062] S4-1, according to Figure 8 As shown, an improved MobileNet-v3 is used as the backbone in the improved YOLO-v5 model. The improved MobileNet-v3 network structure replaces the SE attention mechanism in the base network Bneck with the pixel-level and channel-level attention mechanism CANet, further enhancing the extraction of more information from the image and improving the AGV's material recognition capabilities in harsh environments. The backbone network structure in the improved YOLO-v5 model consists of one 3x3 convolution and 14 bneck structures. The input image first undergoes a 3x3 convolution to reduce the feature size, and then enters the 14-layer bneck structure for feature extraction. The bneck structure follows... Figure 5 As shown, combining the characteristics of previous MobileNet structures, the overall structure still adopts the inverted residual structure. First, a 1x1 pointwise convolution is used to perform dimensionality increase operation, expand the feature map channels, and enrich the number of features. Then, a 3x3 convolution operation is performed to selectively add the CANet attention mechanism. Finally, a 1x1 convolution is performed to obtain the final feature map.

[0063] S4-2, according to Figure 8 As shown, the neck part of the improved YOLO-v5 model adopts the FPN+PAN network structure design from the original YOLO-v5 model. The feature map obtained after s4-1 is constructed from the FPN layer as follows: Figure 7 As shown, feature information is conveyed from bottom to top. High-level features with low resolution and high semantic information and low-level features with high resolution and low semantic information are fused from bottom to top, so that features at different scales all have rich semantic information. Then, the localization information is enhanced from top to bottom through PAN.

[0064] S4-3, according to Figure 8 As shown, a new detection head was selected for the head part of the improved YOLO-v5 model, and the Transformer was used to improve the original head part, as shown below. Figure 9 As shown, the feature map obtained in S4-2 is fed into the Transformer detection head. The local information obtained by the Transformer is larger than that of the CNN, which can enhance the detection of small objects.

[0065] The Transformer detection head consists of two sub-layers: the MHD layer and the MLP layer. Residual connections exist between each sub-layer. LayerNorm and Dropout layers between each layer help the network converge better and prevent overfitting. The MHD layer helps to capture semantics while focusing on pixels; the MLP layer slows down the convergence process.

[0066] S5, training model: The improved YOLO-v5 model is trained using a training set of common material classifications and pallet loads. The weight parameters are updated, and the optimal weight parameters are tested using a test set to ensure that the accuracy and real-time performance of the recognition meet the VGA car recognition standard, and finally the best training model is obtained.

[0067] S6 uses the trained YOLO-v5 model to classify and identify materials, determine the load on the pallet, and output the type of material and the load status of the pallet.

[0068] The above embodiments are merely preferred embodiments of the present invention and are not intended to limit the technical solutions of the present invention. Any technical solution that can be implemented based on the above embodiments without creative effort should be considered to fall within the scope of protection of the patent of the present invention.

Claims

1. An AGV material sorting method, characterized in that, Comprise: Step 1, log in to the client through the web page login service module, start the current working mode in the client, at this time the digital camera starts to work, the tray load condition captured globally is transmitted to the background server through the local area network generated by the wireless AP; Step 2, the background server converts the video frame into an image, and transmits the converted image data into the image recognition module, which performs identification and judgment, and if the tray has a load, the task management module is used to issue a carrying task to the AGV with an industrial camera; The image recognition module comprises an improved YOLO-v5 model, wherein the improved YOLO-v5 model is obtained by an image recognition method, and the image recognition method comprises: S1, collecting target images to form a self-built data set; S2, segmenting the image, and then extracting a data set with material features and tray information therefrom; S3, image annotation enhancement, dividing the data set into a training set, a validation set and a test set; S4, constructing an improved YOLO-v5 model, comprising: S4-1, using an improved MobileNet-v3 network structure as the Backbone part of the improved YOLO-v5 model, wherein the improved MobileNet-v3 network structure uses a pixel-level and channel-level attention mechanism CANet to replace the SE attention mechanism in the MobileNet-v3 basic network Bneck; the network structure of the Backbone of the improved YOLO-v5 model is composed of 1 3x3 convolution and 14 layers of bneck structure; the input image is first reduced in feature size by a 3x3 convolution, and then enters the 14-layer bneck structure for feature extraction; wherein the bneck structure adopts an inverted residual structure, first uses a 1x1 pointwise convolution for dimension increasing operation, expands the feature map channel, and enriches the feature quantity, then performs a 3x3 convolution operation to selectively add a CANet attention mechanism, and finally obtains the final feature map through a 1x1 convolution; S4-2, using the network structure design of FPN+PAN in the original YOLO-v5 model as the neck part of the improved YOLO-v5 model, the feature map obtained by S4-1 is transmitted from bottom to top by the FPN layer to convey feature information, and the high-layer features with low resolution and high semantic information and the bottom-layer features with high resolution and low semantic information are fused from bottom to top, and then the positioning information is strengthened from top to bottom by the PAN; S4-3, using a Transformer detection head as the head part of the improved YOLO-v5 model, wherein the Transformer detection head is obtained by improving the Head part of the original YOLO-v5 model using a Transformer, and the feature map obtained by S4-2 is transmitted into the Transformer detection head, and the local information obtained by the Transformer is used to strengthen the detection of small objects due to the larger local information obtained by the Transformer than the CNN; S5, training the improved YOLO-v5 model to obtain the best training model; S6, the trained YOLO-v5 model is used for classifying and identifying the materials, judging the load of the tray, and outputting the type of the materials and the load of the tray; Step 3, after the AGV with an industrial camera receives the carrying task, the path planning module is used to calculate the route from the current position of the AGV to the carrying area, and when the AGV travels to the carrying area according to the calculated route, the industrial camera collects the material image and transmits the scanned material video information to the background server; Step 4, the background server converts the obtained real-time video information into image information and transmits it to the image recognition module, the image recognition module performs identification and judgment, outputs the material category, and again calls the path planning module to calculate the area where the current material category is to be stored, plans a path for the AGV, and the AGV completes the carrying work according to the planned path.

2. The AGV material sorting method according to claim 1, wherein, The S3 includes: Labelimg software is used to mark the category and position information of the goods in the image, so as to ensure that the marked information corresponds to the marked picture; The marked image is subjected to data enhancement, the method adopted is to rotate the image by several angles clockwise for several times, then randomly adjust the brightness and chroma of each image, add Gaussian noise, and randomly stack pictures, and generate corresponding mark files while enhancing the image.

3. The AGV material sorting method according to claim 1, wherein, The S5 includes: The training set is used to train and learn the improved YOLO-v5 model, update the weight parameters, and test the best weight parameters of the training set to ensure that the accuracy and real-time performance of the identification meet the identification standard of the VGA trolley, and finally obtain the best training model.

4. An AGV goods sorting system for implementing the AGV goods sorting method according to any one of claims 1 to 3, characterized by, The system comprises a client, a background server, a wireless AP, a digital camera and an AGV, the client, the background server and the wireless AP are sequentially connected in communication, the wireless AP is connected in communication with the digital camera and the AGV, and the AGV is provided with an industrial camera which can rotate automatically.

5. The AGV material sorting system according to claim 4, wherein, The background server comprises a web page login service module, an image recognition module, a path planning module, a task management module and a state monitoring module; the web page login service module is used to provide a login control interface based on a browser end, and can be controlled online on a PC end or controlled on site on a mobile end; the image recognition module is used to classify and identify materials, judge the load of a tray, output the type of the materials and the load of the tray; the task management module is used to assign and manage tasks for the AGV; the path planning module is used to control the running path and the advancing track of the AGV; and the state monitoring module is used to monitor the running state of the AGV.

Images