A foreign object detection method for compact shelving aisles based on YOLO-FOD
By adopting the YOLO-FOD network model in dense rack channels, combined with key point detection and classification algorithms, the problems of low accuracy and insufficient adaptability of foreign matter detection in dense rack channels are solved, and accurate judgment and cost reduction in complex environments are achieved , improving the intelligence of dense racks.
Patent Information
- Application Number
- CN202411088692.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-09
- Publication Date
- 2024-11-26
- Estimated Expiration
- 2044-08-09
AI Technical Summary
The existing dense rack channel foreign matter detection methods are not accurate when detecting foreign matter at the edges, and are difficult to adapt to changes in the channel environment, and are expensive or complex to maintain.
The dense rack channel foreign matter detection method based on YOLO-FOD is adopted, and the YOLO-FOD network model is constructed by integrating key point detection algorithms and classification algorithms, including backbone networks, feature fusion structures and double-headed structures, and the rectangular frame loss, category loss, Key point losses and FOD losses are modeled to achieve accurate positioning of the foreign body state of the channel.
It improves the accuracy and adaptability of foreign object detection in dense rack channels, reduces costs, realizes accurate judgments under channel area changes and complex background conditions, and improves the intelligence of dense rack products.
Smart Images

Figure CN118628839B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the fields of intelligent image processing and machine vision, and in particular to a method for detecting foreign objects in a dense shelving channel based on YOLO-FOD. Background Art
[0002] Compact shelving is a type of intensive equipment for centralized management of items. Electric compact shelving is still insufficient to meet the needs of the intelligent era, especially when the compact shelving is closed. If there are people or items in the passage area, it may cause damage to the shelving at best and cause casualties at worst. This problem needs to be solved urgently.
[0003] In response to the above problems, the solutions adopted by enterprises mainly include installing sensors and using track robots. However, these solutions have obvious limitations. The sensor solution is limited by the detection range and accuracy, and it is difficult to cover all possible foreign body situations; and the track robot solution, although it has high flexibility, is expensive, complex to maintain, and lacks real-time performance. Therefore, the market still lacks a high-precision, stable and economical solution.
[0004] At present, although the method of detecting foreign objects in the channel of compact shelving based on deep learning classification has achieved good detection results, its adaptability is still insufficient when dealing with significant changes in the channel environment, such as changes in the storage items on the shelf. In addition, this method has the problem of low accuracy when detecting foreign objects at the edge. Therefore, it is of great significance to study a method for detecting foreign objects in the channel of compact shelving with high accuracy, strong adaptability and low cost. Summary of the invention
[0005] The purpose of the present invention is to provide a method for detecting foreign objects in dense shelving channels based on YOLO-FOD. By fusing the key point detection algorithm and the classification algorithm, the method can realize the precise positioning of the constantly changing channel area and the position of foreign objects, and judge the state of foreign objects in the channel by the position relationship, thereby solving the problem of low accuracy of the prior art when the detection boundary is unclear.
[0006] The technical solution adopted by the present invention is: a method for detecting foreign objects in a dense shelving channel based on YOLO-FOD, comprising the following steps:
[0007] S1: Collect images of foreign objects in the dense shelving channel, annotate the key point information of the dense shelving channel area, the location information of the dense shelving channel area, the category information of the foreign objects, the location information of the foreign objects and the foreign object status of the channel in the image, generate labels corresponding to the image slices, and use the annotated images to form the YOLO-FOD dataset;
[0008] S2: Divide the YOLO-FOD dataset into a training set and a validation set, and enhance the images in the training set;
[0009] S3: Construct a YOLO-FOD network model, which includes a backbone network, a feature fusion structure and a dual-head structure; the backbone network extracts the underlying texture information and deep abstract semantics of the input image to generate multi-scale features; the feature fusion structure samples and fuses the multi-scale features to generate multi-scale fusion semantic features; the dual-head structure predicts the multi-scale fusion semantic features, determines the channel foreign body state and generates a foreign body detection probability vector, and obtains the foreign body detection result through the foreign body detection probability vector;
[0010] S4: Use the training set to train the YOLO-FOD network model, using the rectangular box loss loss box , category loss loss cls , key point loss loss kpt and FOD losses loss FOD Constructing the total loss function loss total , guiding model optimization;
[0011] S5: Use the trained YOLO-FOD network model to detect foreign objects in the channel.
[0012] Furthermore, the images collected in step S1 include two types of images: images with foreign matter and images without foreign matter. An image with foreign matter means that the foreign matter is located inside the channel in the image, and an image without foreign matter means that the foreign matter is located outside the channel in the image or there is no foreign matter in the channel.
[0013] Furthermore, the label in step S1 includes category information cls, rectangular box information box, key point information kpt and FOD state flag flag, wherein the category information cls is used to identify the category to which the target belongs, and its label value adopts a natural number sequence starting from 0, i.e. j The label value of each category is j -1; the rectangular box information box is used to describe the position of the target in the image, including the horizontal coordinate x of the upper left corner of the rectangular box, the vertical coordinate y of the upper left corner of the rectangular box, the width W of the rectangular box and the height H of the rectangular box; the key point information kpt represents the key point position of the target, which contains a total of N×3 values, where N represents the number of key points, and each key point includes a visible state v, a key point horizontal coordinate x′ and a key point vertical coordinate y′. If the key point is visible in the image, the value of the visible state v is 2; if the key point is blocked in the image, the value of the visible state v is 1; if the key point does not appear in the image, the value of the visible state v is 0; the FOD state flag flag represents the foreign body state of the image, where 0 represents no foreign body and 1 represents foreign body.
[0014] Furthermore, the enhancement processing in step S2 is specifically: normalizing the input image, and enhancing the images in the training set by using random grayscale, random brightness and contrast, random perspective transformation, and horizontal and vertical random flipping methods.
[0015] Furthermore, the backbone network includes a CBS module, an ECSP module and a SimSPPF module; the input image is passed through the CBS module to capture the underlying texture information of the image; then down-sampling is performed through M layers of interlaced CBS modules and ECSP modules to gradually extract deep abstract semantics, where M≧2; finally, multi-scale feature extraction is performed through the SimSPPF module; the backbone network finally outputs L different scale feature maps C1, C2, ..., CL, which are used to input feature fusion structure for multi-scale feature fusion, where L≧3;
[0016] The CBS module includes a two-dimensional convolutional layer Conv2d, a batch normalization layer and an activation function SiLu connected in sequence;
[0017] The ECSP module includes a three-layer network structure, wherein the first layer and the third layer are both CBS modules, and the second layer includes three Ghost modules. The features output by the first layer CBS module are split into two paths, one path of features is sequentially transformed by three Ghost modules for deep feature transformation, and the other path of features is kept as a supplementary feature. The outputs of the three Ghost modules are spliced with the supplementary features in the channel dimension to achieve feature fusion and complementarity, and the spliced features are input into the third layer CBS module, and the third layer CBS module outputs the output features of the ECSP module; the Ghost module includes a CBR module and a DWCBR module, the features input into the Ghost module are first processed by the CBR module, and then deep feature extraction is performed by the DWCBR module, and the output of the DWCBR module is spliced with the output of the CBR module in the channel dimension as the output features of the Ghost module; the CBR module includes a two-dimensional convolution layer Conv2d, a batch normalization layer and an activation function ReLU; the DWCBR module includes a depth-separable convolution, a batch normalization layer and an activation function ReLU, and introduces a grouping number g;
[0018] The SimSPPF module includes a three-layer network structure, wherein the first and third layers are both CBR modules, and the second layer includes three maximum pooling layers; the features output by the first-layer CBR module are split into two paths, one feature path is sequentially subjected to three maximum pooling layers for multi-scale feature extraction, and the outputs of the three maximum pooling layers are spliced with the other feature path in the channel dimension to achieve full fusion of multi-scale information; the spliced features are input into the third-layer CBR module, and the output of the third-layer CBR module is used as the output feature of the SimSPPF module.
[0019] Furthermore, the feature fusion structure adopts an FPN-PAN dual-tower structure, in which upsampling uses a deconvolution operation, and downsampling is achieved through a dual-branch structure, that is, the input features are processed by the CBS module and the maximum pooling layer respectively, and the processing results are spliced and input into the CBS module for processing to obtain the final output of the downsampling;
[0020] The FPN-PAN dual-tower structure achieves efficient detection of multi-scale targets by combining the top-down feature fusion mechanism of the feature pyramid network FPN and the bottom-up path enhancement mechanism of the path aggregation network PAN; the FPN-PAN dual-tower structure upsamples the feature maps C1, C2, ..., CL of various scales generated by the backbone network from small to large, and splices them with the large-scale feature maps. At the same time, the C2f module is used to extract features from the splicing results, and the extracted features are downsampled, spliced and extracted by the C2f module again, thereby achieving full integration of information at each scale; finally, the FPN-PAN dual-tower structure will output features of L scales and input them into the corresponding detection heads respectively.
[0021] Furthermore, the dual-head structure includes a detection head structure and a classification head structure; the number of the detection heads is L, corresponding one-to-one to the features output by the feature fusion structure; the detection head structure is a decoupling head, which includes three branches, which are used to predict category information cls, rectangular box information box and key point information kpt respectively; each branch is composed of two CBS modules and convolutional layers in sequence; the output information of the detection head can obtain the position information of the foreign object and the channel after the non-maximum suppression NMS operation;
[0022] The classification head includes a CBR1D module, a BNeck module, an SBottle module, a global average pooling layer and a fully connected layer FC, which is used to receive the output information of the detection head and judge the channel foreign body state according to the output information of the detection head;
[0023] The CBR1D module includes a one-dimensional convolution layer Conv, a batch normalization layer and an activation function ReLU; the BNeck module is a residual module, including two CBS1D modules, the features input to the BNeck module are divided into two paths, one feature is processed by two CBS modules and added to the other feature to obtain the output features of the BNeck module; the CBS1D module includes a one-dimensional convolution layer Conv, a batch normalization layer and an activation function SiLu connected in sequence;
[0024] The SBottle module adds the attention mechanism SE on the basis of the BNeck module to enhance the feature extraction effect; the input features of the SBottle module are divided into two paths, one feature is processed by two CBS1D modules and then added to the other feature to obtain the added features, and the added features are also divided into two paths, one feature is added to the other feature after passing through the attention mechanism SE to obtain the output features of the SBottle module; the attention mechanism SE is composed of a global average pooling layer, a fully connected layer FC, a ReLU activation function, a fully connected layer FC and a Sigmoid activation function in sequence, and the input features are processed by the attention mechanism SE to obtain a weight vector representing the importance of each channel, thereby learning the key information in the input image;
[0025] The classification head extracts the detailed features of the detection head output information through the CBR1D module, and then downsamples through m layers of interlaced BNeck modules, SBottle modules and CBR1D modules to extract global information, and aggregates features through the global average pooling layer. Finally, the features are mapped into foreign body detection probability vectors through the fully connected layer, and the foreign body detection results are obtained by the index corresponding to the maximum value of the foreign body detection probability vector; where m≥3.
[0026] Furthermore, the total loss function in step S4 is loss total The expression is:
[0027] loss total = loss box + loss cls + loss kpt + loss FOD ;
[0028] Among them, the rectangular box loss loss box Including CIOU loss and DFL loss, category loss loss cls is the cross entropy function, key point loss loss kpt It is the OKS function;
[0029] FOD losses loss FOD The expression is:
[0030] loss FOD = - flag (log( p ( flag ))) + (1- flag) log(1- p ( flag ));
[0031] in, flag is the FOD status indicator, p ( flag ) is the softmax processing value of the classification head output vector.
[0032] Furthermore, in step S4, the optimizer is used to perform network iteration, and the learning rate is modified by a linear descent strategy, which is decreased once after each training cycle. n Learning rate of the round lr ( n )for:
[0033] lr ( n ) = a +( b – a )÷ epochs × epoch ;
[0034] in, a is the initial learning rate, b is a tiny constant less than 1e-8, epochs is the total number of training times, epoch is the current number of training times; during the training process, verification is performed every m′ training times to test the verification set accuracy of the model; during the training, the set of model weights with the highest verification set accuracy is saved as the final YOLO-FOD network model weights.
[0035] The beneficial effects of the present invention are:
[0036] (1) The present invention uses the YOLO-FOD network model to accurately judge the foreign matter status inside the channel when the channel area changes and the background is complex, thereby improving the intelligence level and technical added value of the compact shelving products;
[0037] (2) The present invention combines key point detection and classification algorithms to enable the network to effectively utilize location information and category information, thereby improving the accuracy and adaptability of the network. Compared with foreign object detection methods that only rely on key point detection or classification algorithms, the present invention has more outstanding stability and accuracy, and the application scope of the present invention is wide, not limited to the foreign object detection scenario in the dense shelving channel, but can be extended to target detection tasks in variable areas. BRIEF DESCRIPTION OF THE DRAWINGS
[0038] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0039] Figure 1 A diagram showing the specific composition of a data set label in an embodiment of the present invention;
[0040] Figure 2 A network model diagram of an embodiment of the present invention;
[0041] Figure 3 A schematic diagram of the network structure of a backbone network in an embodiment of the present invention;
[0042] Figure 4 A schematic diagram of a network structure of a feature fusion structure in an embodiment of the present invention;
[0043] Figure 5 This is a schematic diagram of a network structure of a double-head structure in an embodiment of the present invention;
[0044] Figure 6 Schematic diagram of the network structure of the classification head in an embodiment of the present invention;
[0045] Figure 7 This is a diagram of the actual detection effect of an embodiment of the present invention. DETAILED DESCRIPTION
[0046] In order to more clearly understand the above-mentioned purpose, features and advantages of the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. In the following description, many specific details are set forth to facilitate a full understanding of the present invention, but the present invention can also be implemented in other ways different from those described herein, and therefore, the present invention is not limited to the limitations of the specific embodiments disclosed below.
[0047] The embodiment of the present invention provides a method for detecting foreign objects in a dense shelving channel based on YOLO-FOD, comprising the following steps:
[0048] S1: Collect images of foreign objects in the dense shelving channel, annotate the key point information of the dense shelving channel area, the location information of the dense shelving channel area, the category information of the foreign objects, the location information of the foreign objects and the foreign object status of the channel in the image, generate labels corresponding to the image slices, and use the annotated images to form the YOLO-FOD dataset.
[0049] In an embodiment of the present invention, the collected images include two types of images, one with foreign matter and one without foreign matter, wherein the image with foreign matter refers to the image with foreign matter located inside the channel, and the image without foreign matter refers to the image with foreign matter located outside the channel or without any foreign matter in the channel. In order to improve the generalization of the model, the embodiment of the present invention sets a variety of collection conditions: for example, different channel widths, different types and positions of foreign matter, and different lighting conditions. The embodiment of the present invention sets four types of foreign matter and two types of lighting conditions, wherein the lighting conditions include turning on and off the ambient light, and the channel width is set to a continuous value. When collecting images with foreign matter, first turn on the ambient light, then put the foreign matter into the channel in turn according to the type and change the position of the foreign matter in the channel at the same time, and take the image. Subsequently, change the channel width and repeat the above shooting process. Then turn off the ambient light, repeat the above process again, and complete the collection of images with foreign matter. The collection process of images without foreign matter is similar to that of images with foreign matter. When shooting, the foreign matter needs to be placed outside the channel in turn according to the type, and additional scene images without any foreign matter under various channel widths are taken. Finally, the ratio of the two types of images is about 1:1.
[0050] The label is Figure 1 As shown, it includes category information cls, rectangular box information box, key point information kpt and FOD state flag flag. The category information cls is used to identify the category of the target, and its label value adopts a natural number sequence starting from 0, that is, j The label value of each category is j -1; the rectangular frame information box is used to describe the position of the target in the image, including the horizontal coordinate x of the upper left corner of the rectangular frame, the vertical coordinate y of the upper left corner of the rectangular frame, the width W of the rectangular frame, and the height H of the rectangular frame; in the embodiment of the present invention, the target includes the channel area of the compact shelving and foreign objects. The key point information kpt represents the key point position of the target, which contains a total of N×3 values, where N represents the number of key points in the channel, and each key point includes a visible state v, a key point horizontal coordinate x′, and a key point vertical coordinate y′. Figure 1 Medium i ' indicates the i The horizontal coordinate of the key point, y i ' indicates the i The vertical coordinate of the key point, v i Indicates i The visible state of the key point; in the embodiment of the present invention, the value of N is 4. If the key point is visible in the image, the value of the visible state v is 2; if the key point is blocked in the image, the value of the visible state v is 1; if the key point does not appear in the image, the value of the visible state v is 0; the FOD state flag indicates the foreign object state of the image, where 0 indicates no foreign object and 1 indicates foreign object.
[0051] S2: Divide the YOLO-FOD dataset into a training set and a validation set, and perform enhancement processing on the images in the training set. In an embodiment of the present invention, the enhancement processing specifically includes: scaling the input image to 640 pixels × 640 pixels and dividing each pixel value by 255 for normalization, and performing enhancement processing on the images in the training set using random grayscale, random brightness and contrast, random perspective transformation, and horizontal and vertical random flipping methods.
[0052] S3: Build the YOLO-FOD network model, such as Figure 2 As shown, the YOLO-FOD network model includes a backbone network, a feature fusion structure and a dual-head structure; the backbone network extracts the underlying texture information and deep abstract semantics of the input image to generate multi-scale features; the feature fusion structure samples and fuses the multi-scale features to generate multi-scale fusion semantic features; the dual-head structure predicts the multi-scale fusion semantic features, determines the channel foreign body state and generates a foreign body detection probability vector, and obtains the foreign body detection result through the foreign body detection probability vector.
[0053] like Figure 3 As shown, the backbone network includes a CBS module, an ECSP module and a SimSPPF module. Figure 3 Here, h represents the height of the feature map, w represents the width of the feature map, and c represents the number of channels of the feature map. The input image is passed through the CBS module to capture the underlying texture information of the image; then it is downsampled through M layers of interlaced CBS modules and ECSP modules to gradually extract deep abstract semantics, where M≧2; finally, multi-scale feature extraction is performed through the SimSPPF module; the backbone network finally outputs L different scale feature maps C1, C2, ..., CL, which are used to input the feature fusion structure for multi-scale feature fusion, where L≧3. In the embodiment of the present invention, the value of M is 4 and the value of L is 3.
[0054] The CBS module includes a two-dimensional convolutional layer Conv2d, a batch normalization layer and an activation function SiLu connected in sequence.
[0055] The ECSP module includes a three-layer network structure, in which the first and third layers are CBS modules, and the second layer includes three Ghost modules. The features output by the first layer CBS module are split into two paths, one path of features is sequentially transformed through three Ghost modules for deep feature transformation, and the other path of features is kept as a supplementary feature. The outputs of the three Ghost modules are spliced with the supplementary features in the channel dimension to achieve feature fusion and complementarity, and the spliced features are input into the third layer CBS module, and the third layer CBS module outputs the output features of the ECSP module. The Ghost module includes a CBR module and a DWCBR module. The features input into the Ghost module are first processed by the CBR module, and then deep feature extraction is performed through the DWCBR module. The output of the DWCBR module is spliced with the output of the CBR module in the channel dimension as the output features of the Ghost module. The CBR module includes a two-dimensional convolution layer Conv2d, a batch normalization layer, and an activation function ReLU; the DWCBR module includes a depth-separable convolution, a batch normalization layer, and an activation function ReLU, and introduces the number of groups g. In the embodiment of the present invention, the value of the number of groups g is equal to the number of channels c of the feature map.
[0056] The SimSPPF module includes a three-layer network structure, wherein the first and third layers are both CBR modules, and the second layer includes three maximum pooling layers; the features output by the first-layer CBR module are split into two paths, one feature path is sequentially subjected to three maximum pooling layers for multi-scale feature extraction, and the outputs of the three maximum pooling layers are spliced with the other feature path in the channel dimension to achieve full fusion of multi-scale information; the spliced features are input into the third-layer CBR module, and the output of the third-layer CBR module is used as the output feature of the SimSPPF module.
[0057] like Figure 4As shown, the feature fusion structure adopts the FPN-PAN dual-tower structure. The FPN-PAN dual-tower structure realizes efficient detection of multi-scale targets by combining the top-down feature fusion mechanism of the feature pyramid network FPN and the bottom-up path enhancement mechanism of the path aggregation network PAN. The FPN-PAN dual-tower structure upsamples the feature maps C1, C2, ..., CL of various scales generated by the backbone network from small to large in size, and splices them with the large-size feature maps. At the same time, the C2f module is used to extract features from the splicing results, and the extracted features are downsampled, spliced, and extracted by the C2f module again, so as to achieve full integration of information at each scale; finally, the FPN-PAN dual-tower structure will output features of L scales and input them into the corresponding detection heads respectively. In the above process, upsampling uses deconvolution operation, and downsampling is realized through a dual-branch structure, that is, the input features are processed by the CBS module and the maximum pooling layer respectively, and the processing results are spliced and input into the CBS module for processing to obtain the final output of downsampling.
[0058] The dual-head structure includes a detection head and a classification head. The number of the detection heads is L, and the detection heads correspond to the features output by the feature fusion structure one by one. Figure 5 As shown in the figure, the detection head is a decoupled head, which includes three branches, which are used to predict category information cls, rectangular box information box and key point information kpt respectively; each branch is composed of two CBS modules and convolutional layers in sequence; the output information of the detection head can obtain the location information of foreign objects and channels after the non-maximum suppression NMS operation.
[0059] like Figure 6 As shown, the classification head includes a CBR1D module, a BNeck module, an SBottle module, a global average pooling layer and a fully connected layer FC, which are used to receive the output information of the detection head and judge the channel foreign body state according to the output information of the detection head. Figure 6 Here h represents the height of the feature map, w represents the width of the feature map, and c represents the number of channels of the feature map.
[0060] The CBR1D module includes a one-dimensional convolution layer Conv, a batch normalization layer and an activation function ReLU, which is essentially a one-dimensional form of the CBR module; the BNeck module is a residual module, including two CBS1D modules, and the features input to the BNeck module are divided into two paths. One path of features is processed by two CBS modules and added to the other path of features to obtain the output features of the BNeck module; the CBS1D module includes a one-dimensional convolution layer Conv, a batch normalization layer and an activation function SiLu connected in sequence.
[0061] The SBottle module adds the attention mechanism SE on the basis of the BNeck module to enhance the feature extraction effect; the features input to the SBottle module are divided into two paths, one feature is processed by two CBS1D modules and then added to the other feature to obtain the added features, and the added features are also divided into two paths, one feature is added to the other feature after passing through the attention mechanism SE to obtain the output features of the SBottle module; the attention mechanism SE is composed of a global average pooling layer, a fully connected layer FC, a ReLU activation function, a fully connected layer FC and a Sigmoid activation function in sequence, and the input features are processed by the attention mechanism SE to obtain a weight vector representing the importance of each channel, thereby learning the key information in the input image. Figure 6 The r in it represents the channel compression ratio. When the input channel number is c, the fully connected layer FC will output a feature map with a channel number of c / r.
[0062] The classification head extracts the detailed features of the output information of the detection head through the CBR1D module, and then downsamples through the m-layer interlaced BNeck module, SBottle module and CBR1D module to extract global information, and aggregates features through the global average pooling layer, and finally maps the features into a foreign body detection probability vector through the fully connected layer, and obtains the foreign body detection result with the index corresponding to the maximum value of the foreign body detection probability vector; wherein m≥3. In the embodiment of the present invention, the value of m is 3.
[0063] In the embodiment of the present invention, each type of convolution layer and each type of pooling layer includes three parameters that need to be set: convolution kernel size k, convolution step size s, and padding size p. Figures 3 - 6 The various convolutional layer parameters in each module are labeled in the figure. The convolution kernel size k has 4 values, namely k1=1, k2=2, k3=3 and k5=5; the convolution step size s has 2 values, namely s1=1 and s2=2; the padding size p has 3 values, namely p0=0, p1=1 and p2=2; the number of groups g is generally equal to the number of channels c of the feature map.
[0064] S4: Use the training set to train the YOLO-FOD network model, using the rectangular box loss loss box , category loss loss cls , key point loss loss kpt and FOD losses loss FOD Constructing the total loss function loss total , guiding model optimization.
[0065] The total loss function losstotal The expression is:
[0066] loss total = loss box + loss cls + loss kpt + loss FOD ;
[0067] Among them, the rectangular box loss loss box Including CIOU loss and DFL loss, category loss loss cls is the cross entropy function, key point loss loss kpt It is the OKS function.
[0068] FOD losses loss FOD The expression is:
[0069] loss FOD = - flag (log( p ( flag ))) + (1- flag ) log(1- p ( flag ));
[0070] in, flag is the FOD state flag, and p(flag) is the softmax processing value of the classification head output vector.
[0071] During the training process, the optimizer is used to iterate the network, and the linear descent strategy is used to modify the learning rate, which is reduced once after each training cycle. n Learning rate of the round lr ( n )for:
[0072] lr ( n ) = a +( b – a )÷ epochs × epoch ;
[0073] in, a is the initial learning rate, b is a tiny constant less than 1e-8, epochs is the total number of training times, epochis the current number of training times; during the training process, verification is performed every m′ training times to test the verification set accuracy of the model; during the training, the set of model weights with the highest verification set accuracy is saved as the final YOLO-FOD network model weights.
[0074] S5: Use the trained YOLO-FOD network model to detect foreign objects in the channel, export the trained model weights as an onnx file, and call the file to infer the channel video frames to obtain the foreign object status in the channel.
[0075] The parameters of the backbone network in the embodiment of the present invention are shown in Table 1, and the parameters of the classification head are shown in Table 2. The size of the feature map C1 generated by the backbone network is 80×80×256, the size of the feature map C2 is 40×40×512, and the size of the feature map C3 is 20×20×512, which are used to input into the feature extraction structure for multi-scale feature fusion to enhance the robustness of the network.
[0076] Table 1 Backbone network parameters
[0077]
[0078] Table 2 Classification header parameters
[0079]
[0080] The actual detection effect of the embodiment of the present invention under different channel widths, different foreign matter conditions and different lighting conditions is as follows: Figure 7 As shown, Figure 7 In the above, human indicates that the foreign object type is a human, box indicates that the foreign object type is a box, and corridor indicates that the foreign object type is a corridor. In order to test the detection effect of the method described in the embodiment of the present invention, a detection experiment in a real environment was carried out, and the test method is as follows:
[0081] There is a foreign body test method: the width of the aisle of the compact shelving is continuously adjusted through the joystick, different types of foreign bodies are placed in aisles of various widths in turn, and the positions of foreign bodies in the aisles are changed to simulate various scenarios;
[0082] Foreign matter-free test method: Automatically open and close the compact shelving to adjust the channel width. During the opening and closing process, place various foreign objects outside the channel in turn and change the position of the external foreign objects;
[0083] According to the above operation method, 3600×3 frames are continuously tested to fully evaluate the performance and stability of the system. The test data is shown in Table 3:
[0084] Table 3 Experimental test results
[0085]
[0086] It can be seen from the data in Table 3 that the embodiment of the present invention can accurately detect the state of foreign objects under different channel widths, different types and positions of foreign objects, and different lighting conditions, and is well adapted to the complex environment of the compact shelving warehouse. The embodiment of the present invention uses the YOLO-FOD network model to accurately judge the foreign object status inside the channel when the channel area changes and the background is complex, thereby improving the intelligence level and technical added value of the compact shelving products. In addition, the embodiment of the present invention combines key point detection and classification algorithms to enable the network to effectively utilize location information and category information, thereby improving the accuracy and adaptability of the network; compared with foreign object detection methods that only rely on key point detection or classification algorithms, the embodiment of the present invention has more outstanding stability and accuracy, and the application range of the embodiment of the present invention is wide, not limited to the compact shelving channel foreign object detection scenario, and can be extended to target detection tasks in variable areas.
[0087] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.
Claims
1. A method for detecting foreign objects in a dense shelving channel based on YOLO-FOD, characterized in that: The steps include: S1: Collect images of foreign objects in the dense shelving channel, annotate the key point information of the dense shelving channel area, the location information of the dense shelving channel area, the category information of the foreign objects, the location information of the foreign objects and the foreign object status of the channel in the image, generate labels corresponding to the image slices, and use the annotated images to form the YOLO-FOD dataset; S2: Divide the YOLO-FOD dataset into a training set and a validation set, and enhance the images in the training set; S3: Construct a YOLO-FOD network model, which includes a backbone network, a feature fusion structure and a dual-head structure; the backbone network extracts the underlying texture information and deep abstract semantics of the input image to generate multi-scale features; the feature fusion structure samples and fuses the multi-scale features to generate multi-scale fusion semantic features; the dual-head structure predicts the multi-scale fusion semantic features, determines the channel foreign body state and generates a foreign body detection probability vector, and obtains the foreign body detection result through the foreign body detection probability vector; the dual-head structure includes a detection head and a classification head, the number of the detection heads is L, which corresponds to the features output by the feature fusion structure one by one, and the detection head structure is a decoupling head, which includes three branches, which are used to predict the category information cls, the rectangular box information box and Key point information kpt; the output information of the detection head is subjected to non-maximum suppression NMS operation to obtain the position information of the foreign body and the channel; the classification head includes a CBR1D module, a BNeck module, an SBottle module, a global average pooling layer and a fully connected layer FC, which is used to receive the output information of the detection head and judge the foreign body state of the channel according to the output information of the detection head; the classification head extracts the detailed features of the output information of the detection head through the CBR1D module, and then downsamples through m layers of interlaced BNeck modules, SBottle modules and CBR1D modules to extract global information, and aggregates features through a global average pooling layer, and finally maps the features to a foreign body detection probability vector through a fully connected layer, and obtains the foreign body detection result with the index corresponding to the maximum value of the foreign body detection probability vector; wherein, m≥3; S4: Use the training set to train the YOLO-FOD network model, using the rectangular box loss , category loss , key point loss and FOD losses Constructing the total loss function , guiding model optimization; the FOD loss The expression is: ; in, is the FOD status indicator, The softmax processing value of the classification head output vector; S5: Use the trained YOLO-FOD network model to detect foreign objects in the channel.
2. According to a YOLO-FOD-based method for detecting foreign objects in a densely packed channel according to claim 1, it is characterized in that: The images collected in step S1 include two types of images, one with foreign matter and one without foreign matter. An image with foreign matter means that the foreign matter is located inside the channel in the image, and an image without foreign matter means that the foreign matter is located outside the channel in the image or there is no foreign matter in the channel.
3. According to a YOLO-FOD-based method for detecting foreign objects in a densely packed channel according to claim 1, it is characterized in that: The label in step S1 includes category information cls, rectangular box information box, key point information kpt and FOD state flag flag. The category information cls is used to identify the category to which the target belongs. Its label value adopts a natural number sequence starting from 0, that is, j The label value of each category is j -1; the rectangular box information box is used to describe the position of the target in the image, including the horizontal coordinate x of the upper left corner of the rectangular box, the vertical coordinate y of the upper left corner of the rectangular box, the width W of the rectangular box and the height H of the rectangular box; the key point information kpt represents the key point position of the target, which contains a total of N×3 values, where N represents the number of key points, and each key point includes a visible state v, a key point horizontal coordinate x′ and a key point vertical coordinate y′. If the key point is visible in the image, the value of the visible state v is 2; if the key point is blocked in the image, the value of the visible state v is 1; if the key point does not appear in the image, the value of the visible state v is 0; the FOD state flag flag represents the foreign body state of the image, where 0 represents no foreign body and 1 represents foreign body.
4. According to a YOLO-FOD-based method for detecting foreign objects in a densely packed channel according to claim 1, it is characterized in that: The enhancement processing in step S2 is specifically: normalizing the input image, and enhancing the image in the training set by using random grayscale, random brightness and contrast, random perspective transformation, and horizontal and vertical random flipping methods.
5. According to a YOLO-FOD-based method for detecting foreign objects in a dense shelving channel according to claim 1, it is characterized in that: The backbone network includes a CBS module, an ECSP module and a SimSPPF module; the input image is passed through the CBS module to capture the underlying texture information of the image; then down-sampling is performed through M layers of interlaced CBS modules and ECSP modules to gradually extract deep abstract semantics, where M≥2; finally, multi-scale feature extraction is performed through the SimSPPF module; the backbone network finally outputs L different scale feature maps C1, C2, ..., CL, which are used to input feature fusion structure for multi-scale feature fusion, where L≥3; The CBS module includes a two-dimensional convolutional layer Conv2d, a batch normalization layer and an activation function SiLu connected in sequence; The ECSP module includes a three-layer network structure, wherein the first layer and the third layer are both CBS modules, and the second layer includes three Ghost modules. The features output by the first layer CBS module are split into two paths, one path of features is sequentially transformed by three Ghost modules for deep feature transformation, and the other path of features is kept as a supplementary feature. The outputs of the three Ghost modules are spliced with the supplementary features in the channel dimension to achieve feature fusion and complementarity, and the spliced features are input into the third layer CBS module, and the third layer CBS module outputs the output features of the ECSP module; the Ghost module includes a CBR module and a DWCBR module, the features input into the Ghost module are first processed by the CBR module, and then deep feature extraction is performed by the DWCBR module, and the output of the DWCBR module is spliced with the output of the CBR module in the channel dimension as the output features of the Ghost module; the CBR module includes a two-dimensional convolution layer Conv2d, a batch normalization layer and an activation function ReLU; the DWCBR module includes a depth-separable convolution, a batch normalization layer and an activation function ReLU, and introduces a grouping number g; The SimSPPF module includes a three-layer network structure, wherein the first and third layers are both CBR modules, and the second layer includes three maximum pooling layers; the features output by the first-layer CBR module are split into two paths, one feature path is sequentially subjected to three maximum pooling layers for multi-scale feature extraction, and the outputs of the three maximum pooling layers are spliced with the other feature path in the channel dimension to achieve full fusion of multi-scale information; the spliced features are input into the third-layer CBR module, and the output of the third-layer CBR module is used as the output feature of the SimSPPF module.
6. According to a YOLO-FOD-based method for detecting foreign objects in a densely packed channel according to claim 1, it is characterized in that: The feature fusion structure adopts the FPN-PAN dual-tower structure, in which upsampling uses a deconvolution operation, and downsampling is achieved through a dual-branch structure, that is, the input features are processed by the CBS module and the maximum pooling layer respectively, and the processing results are spliced and input into the CBS module for processing to obtain the final output of the downsampling; The FPN-PAN dual-tower structure achieves efficient detection of multi-scale targets by combining the top-down feature fusion mechanism of the feature pyramid network FPN and the bottom-up path enhancement mechanism of the path aggregation network PAN; the FPN-PAN dual-tower structure upsamples the feature maps C1, C2, ..., CL of various scales generated by the backbone network from small to large, and splices them with the large-scale feature maps. At the same time, the C2f module is used to extract features from the splicing results, and the extracted features are downsampled, spliced and extracted by the C2f module again, thereby achieving full integration of information at each scale; finally, the FPN-PAN dual-tower structure will output features of L scales and input them into the corresponding detection heads respectively.
7. The method for detecting foreign objects in a dense shelving channel based on YOLO-FOD according to claim 1, characterized in that: The CBR1D module includes a one-dimensional convolution layer Conv, a batch normalization layer and an activation function ReLU; the BNeck module is a residual module, including two CBS1D modules, the features input to the BNeck module are divided into two paths, one feature is processed by two CBS modules and added to the other feature to obtain the output features of the BNeck module; the CBS1D module includes a one-dimensional convolution layer Conv, a batch normalization layer and an activation function SiLu connected in sequence; The SBottle module adds an attention mechanism SE on the basis of the BNeck module to enhance the feature extraction effect; the features input to the SBottle module are divided into two paths, one feature is processed by two CBS1D modules and then added to the other feature to obtain the added features, and the added features are also divided into two paths, one feature is processed by the attention mechanism SE and then added to the other feature to obtain the output features of the SBottle module; The attention mechanism SE is composed of a global average pooling layer, a fully connected layer FC, a ReLU activation function, a fully connected layer FC and a Sigmoid activation function in sequence. After the input features are processed by the attention mechanism SE, a weight vector representing the importance of each channel is obtained, thereby learning the key information in the input image.
8. The method for detecting foreign objects in a compact shelving channel based on YOLO-FOD according to claim 1, characterized in that: The total loss function in step S4 The expression is: ; Among them, the rectangular box loss Including CIOU loss and DFL loss, category loss is the cross entropy function, key point loss It is the OKS function.
9. The method for detecting foreign objects in a dense shelving channel based on YOLO-FOD according to claim 1, characterized in that: In step S4, the optimizer is used to iterate the network, and the learning rate is modified by a linear descent strategy, which is decreased once after each training cycle. Learning rate of the round for: ; in, is the initial learning rate, is a tiny constant less than 1e-8, is the total number of training times, is the current number of training times; during the training process, verification is performed every m′ training times to test the verification set accuracy of the model; during the training, the set of model weights with the highest verification set accuracy is saved as the final YOLO-FOD network model weights.
