Foreign matter detection model training method and foreign matter detection method

Abstract

The invention provides a foreign matter detection model training method and a foreign matter detection method, and belongs to the technical field of computer vision. The method comprises the following steps: extracting respective first feature maps of a training image under different channels, respectively extracting a first attention map and a second attention map of each first feature map based on a space attention mechanism, and obtaining a second feature map corresponding to each channel based on the first attention map, the second attention map and the first feature map, calculating a channel attention weight of each feature in each second feature map based on a channel attention mechanism, performing weighted average on the features in the second feature maps based on the channel attention weights, and constructing a third feature map under each channel according to a weighted average result, and training the foreign matter detection model based on a fusion feature map constructed by the first feature map and the third feature map, the marking result of the training image and a preset loss function.

Description

Technical Field

[0001] This application belongs to the field of computer vision technology, and particularly relates to a method for training a foreign object detection model, a method for detecting foreign objects, an electronic device, and a readable storage medium. Background Art

[0002] The conveyor belt is one of the important devices in many production links. During the transportation of the conveyor belt, if foreign objects such as steel bars, concrete blocks, and woven fabrics are mixed in, it will cause problems such as tearing and entanglement of the conveyor belt, resulting in equipment damage and production stagnation, and even potentially threatening the personal safety of relevant staff.

[0003] Related technologies judge whether there are foreign objects on the conveyor belt by manually observing the conveyor belt.

[0004] However, continuously observing the conveyor belt manually requires a high level of attention from personnel. Long-term monitoring will cause visual fatigue, resulting in a problem of low accuracy in foreign object detection. Summary of the Invention

[0005] The purpose of the embodiments of this application is to provide a method for training a foreign object detection model, a method for detecting foreign objects, an electronic device, and a readable storage medium, which can solve the problem that the existing solution requires a high level of attention from personnel, and long-term monitoring will cause visual fatigue, resulting in a problem of low accuracy in foreign object detection.

[0006] To solve the above technical problems, this application is implemented as follows: In a first aspect, the embodiments of this application provide a method for training a foreign object detection model, and the method includes: Obtain training images, and label the object objects in the training images to obtain the labeling results of the training images; the training images include: images of conveyor belts with object objects; the object objects include foreign objects; Extract the respective first feature maps of the training images in different channels through a convolutional layer and a cross-stage fusion layer; Based on the spatial attention mechanism, extract the first attention map in the horizontal coordinate direction and the second attention map in the vertical coordinate direction of each first feature map respectively, and based on the first attention map, the second attention map, and the first feature map, obtain a second feature map corresponding to each channel; Based on the channel attention mechanism, calculate the channel attention weights of each feature in each second feature map, and based on the channel attention weights, perform weighted averaging on the features in the second feature map, and construct a third feature map for each channel according to the weighted average result; Fuse the first feature map and the third feature map to obtain a fused feature map, and train the foreign object detection model based on the fused feature map, the annotation result of the training image, and a preset loss function.

[0007] In a second aspect, an embodiment of the present application provides a method for detecting foreign objects, the method including: Obtain a video to be detected; Input video frames in the video to be detected into a foreign object detection model to detect the foreign object; the foreign object detection model is trained by the training method of the foreign object detection model as described in the first aspect; Wherein, the video frame includes: an image of a conveyor belt with an object; the object includes a foreign object.

[0008] In a third aspect, an embodiment of the present application discloses a training device for a foreign object detection model, the device including: An acquisition module, configured to acquire a training image and annotate an object in the training image to obtain an annotation result of the training image; the training image includes: an image of a conveyor belt with an object; the object includes a foreign object; A first extraction module, configured to extract respective first feature maps of the training image in different channels through a convolutional layer and a cross-stage fusion layer; A determination module, configured to respectively extract a first attention map in the horizontal coordinate direction and a second attention map in the vertical coordinate direction of each first feature map based on a spatial attention mechanism, and obtain a second feature map corresponding to each channel based on the first attention map, the second attention map, and the first feature map; A construction module, configured to calculate channel attention weights of each feature in each second feature map based on a channel attention mechanism, perform weighted averaging on the features in the second feature map based on the channel attention weights, and construct a third feature map for each channel according to the weighted averaging result; A training module, configured to fuse the first feature map and the third feature map to obtain a fused feature map, and train the foreign object detection model based on the fused feature map, the annotation result of the training image, and a preset loss function.

[0009] In a fourth aspect, an embodiment of the present application discloses a detection device for foreign objects, the device including: An acquisition module, configured to acquire a video to be detected; A detection module, configured to input video frames in the video to be detected into a foreign object detection model to detect the foreign object; the foreign object detection model is trained by the training method of the foreign object detection model as described in the first aspect; Among them, the video frame includes: an image of a conveyor belt with an object; the object includes a foreign object.

[0010] In a fifth aspect, an embodiment of the present application discloses an electronic device, including a processor and a memory. The memory stores a program or instruction that can run on the processor. When the program or instruction is executed by the processor, the steps of the method described in the first aspect or the second aspect are implemented.

[0011] In an embodiment of the present application, after extracting the respective first feature maps of the training image in different channels, based on the spatial attention mechanism, the first attention map in the horizontal coordinate direction and the second attention map in the vertical coordinate direction of each first feature map are respectively extracted, and based on the first attention map, the second attention map, and the first feature map, a second feature map corresponding to each channel is obtained. Since the first attention map and the second attention map represent the position information in the horizontal coordinate direction and the vertical coordinate direction, the second feature map incorporates the global information of the first feature maps in different directions on the basis of the first feature map, and can better capture the relationship between different positions in the input training image. Based on the channel attention mechanism, the channel attention weight of each feature in each second feature map is calculated, and based on the channel attention weight, the features in the second feature map are weighted and averaged, and a third feature map under each channel is constructed according to the weighted average result. The third feature map models the importance of the channels on the basis of the second feature map, dynamically adjusts the channel weights, highlights the key features, and suppresses the unimportant features, further improving the feature representation ability. The foreign object detection model trained with the training image can more accurately learn the features of the foreign object, thereby improving the foreign object detection accuracy.

[0012] The above description is only an overview of the technical solution of the present application. In order to be able to understand the technical means of the present application more clearly, it can be implemented according to the content of the specification. And in order to make the above and other purposes, features, and advantages of the present application more obvious and understandable, the specific embodiments of the present application are specifically given below. Description of the Drawings

[0013] The above and / or additional aspects and advantages of the present application will become obvious and easy to understand from the description of the embodiments in conjunction with the following drawings, where: Figure 1 is a method for training a foreign object detection model provided by an embodiment of the present application; Figure 2 is a computational graph of a spatial attention mechanism provided by an embodiment of the present application; Figure 3 is another method for training a foreign object detection model provided by an embodiment of the present application; Figure 4It is a computational graph of a channel attention mechanism provided by an embodiment of the present application; Figure 5 It is an architecture diagram of a foreign object detection model provided by an embodiment of the present application; Figure 6 It is a conveyor belt foreign object detection and warning system based on the Jetson Orin NX platform provided by an embodiment of the present application; Figure 7 It is a detailed step flow chart of conveyor belt foreign object detection based on the Jetson Orin NX platform provided by an embodiment of the present application; Figure 8 It is a schematic diagram of the deployment principle of the Jetson Orin NX platform provided by an embodiment of the present application; Figure 9 It is a method for detecting foreign objects provided by an embodiment of the present application; Figure 10 It is a schematic structural diagram of a training device for a foreign object detection model provided by an embodiment of the present application; Figure 11 It is a schematic structural diagram of a detection device for foreign objects provided by an embodiment of the present application; Figure 12 It is a structural block diagram of an electronic device provided by an embodiment of the present application. Detailed implementation manners

[0014] Next, the technical solutions in the embodiments of the present application will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present application.

[0015] The terms "first", "second", etc. in the specification of the present application are used to distinguish similar objects, rather than to describe a specific order or sequence. It should be understood that such data can be interchanged under appropriate circumstances so that the embodiments of the present application can be implemented in an order other than those illustrated or described herein, and the objects distinguished by "first", "second", etc. are generally of the same category, and the number of objects is not limited. For example, the first object can be one or multiple. In addition, "and / or" in the specification means at least one of the connected objects, and the character " / " generally indicates an "or" relationship between the associated objects before and after.

[0016] Next, in conjunction with the accompanying drawings, the training method of the foreign object detection model and the method for detecting foreign objects provided by the embodiments of the present application will be described in detail through specific embodiments and their application scenarios.

[0017] Figure 1 This is a method for training a foreign object detection model provided by an embodiment of the present application. Refer to Figure 1 , the method includes: Step 101: Obtain training images, and label the object objects in the training images to obtain the labeling results of the training images; the training images include: images of a conveyor belt with object objects; the object objects include foreign objects.

[0018] Exemplarily, first, a captured image of a conveyor belt with foreign objects can be taken through a camera, or a network image of a conveyor belt with foreign objects on the network can be directly obtained, and training images are constructed based on the captured image and the network image. Specifically, the captured image and the network image are used as the original images, and data augmentation operations are performed on the original images to expand the original images to obtain expanded images. The original images and the expanded images are used as training images. Expanding the original images can increase the amount of data for identifying foreign objects on the conveyor belt, thereby increasing the probability of obtaining ideal weights during the training of the foreign object detection model.

[0019] Exemplarily, the data augmentation operations can be rotation, flipping, contrast adjustment, adding noise, etc. For example, by rotating the original image, the image obtained after rotation is used as the augmented data of the original image. Another example is by flipping the original image, and the image obtained after flipping is used as the augmented data of the original image.

[0020] Exemplarily, labeling is a process of artificially recording feature information of object objects on training images. The labeling results can be divided into 4 categories: large pieces (large stones or concrete blocks), metals (steel bars, broken metal blocks, iron sheets, etc.), fibers (plastics, gunny bags, ropes, etc.), and humans (used to identify personnel approaching the conveyor belt). After completing the labeling operation, the standard results obtained from the training images and the training images can be stored.

[0021] Step 102: Extract the respective first feature maps of the training images in different channels through a convolutional layer and a cross-stage fusion layer.

[0022] Exemplarily, the convolutional layer is a core component in a deep learning convolutional neural network. Through the sliding of a convolutional kernel on the input image for convolutional operations, local features in the image are extracted. Convolutional operations can automatically capture basic features such as edges and textures in the image, and different types of features can be extracted through different convolutional kernels.

[0023] Exemplarily, the role of the cross-stage fusion layer is to fuse the features extracted in different stages, comprehensively utilize the detailed features extracted by the shallow network and the semantic features extracted by the deep network, so that the model can learn richer and more expressive feature representations. For example, the cross-stage fusion layer can be a C2F layer. First, the basic features of the input image are extracted through the initial convolutional block. Then, with the help of multiple Bottleneck blocks, the features are refined and enhanced to capture complex patterns and details, and the Concat operation is used to fuse different feature maps, integrating multi-scale and multi-level information, and finally outputting an informative feature map to provide a good feature representation for subsequent detection and classification tasks.

[0024] The convolutional layer can extract the basic features of the image, and the cross-stage fusion layer can integrate the features of different levels, making the first feature map contain various feature information from simple to complex, from details to semantics, and enhancing the model's ability to understand the image. For example, in the image classification task, it can enable the model to more accurately distinguish the feature differences of different category images.

[0025] Step 103: Based on the spatial attention mechanism, the first attention map in the horizontal coordinate direction and the second attention map in the vertical coordinate direction of each first feature map are respectively extracted, and based on the first attention map, the second attention map, and the first feature map, a second feature map corresponding to each channel is obtained.

[0026] Exemplarily, the spatial attention mechanism (Coordinate Attention) encodes the spatial information. By respectively performing pooling operations on the input feature map in the horizontal and vertical directions, two feature vectors with different spatial information are obtained. For example, the strategy of performing global average pooling in the horizontal and vertical directions respectively. Taking the dimension of the input feature map as C×H×W, where C is the number of channels, H is the height, and W is the width as an example, see Figure 2 , by using a pooling kernel with dimension (H, 1) in the horizontal coordinate direction, global average pooling is performed on the first feature map of each channel to obtain a first feature map with dimension C×1×W. In the vertical coordinate direction, a pooling kernel with dimension (1, W) is used to perform global average pooling to obtain a second feature map with dimension C×H×1. This step can effectively capture the global information of the feature map in different directions, thereby integrating the spatial coordinate information into the generated attention map, enabling the model to more accurately locate and identify the target.

[0027] Exemplarily, based on the first attention map, the second attention map, and the first feature map, a second feature map corresponding to each channel is obtained. Since the first attention map and the second attention map have the global information of the first feature map in different directions, the coordinate attention map can be generated by fusing and transforming these two attention maps. The coordinate attention map can simultaneously consider the position information of the feature in the space and the correlation between channels, so as to more accurately locate the important regions in the image.

[0028] Step 104: Based on the channel attention mechanism, calculate the channel attention weights of each feature in each second feature map, and based on the channel attention weights, perform weighted averaging on the features in the second feature map, and construct a third feature map for each channel according to the weighted averaging result.

[0029] Exemplarily, in a convolutional neural network, the output of each convolutional layer is usually a multi-channel feature map, and the feature maps of different channels often represent different semantic information. The channel attention mechanism will perform weighted processing on the features of each channel, and by learning the importance of each channel, different weights are assigned to different channels, so as to highlight the features of important channels and suppress the features of unimportant channels.

[0030] Exemplarily, when operating on each second feature map based on the channel attention mechanism, generally, global pooling is first performed on the feature map in the spatial dimensions (height and width) to aggregate the features of each channel into a value to obtain the statistical information of the channel. For example, through global average pooling. Then, after passing through a fully connected layer and an activation function, values with the same number as the number of channels are output, and these values are mapped to between 0 and 1 through the Sigmoid activation function. Each value is the channel attention weight of the corresponding channel, measuring the importance of the features of this channel.

[0031] After obtaining the channel attention weights, multiply all the feature values of each channel in the second feature map by the corresponding channel attention weights respectively to achieve weighted averaging of the features of this channel. After completing the above weighted averaging operation for each channel of the second feature map, combine the weighted averaging results of each obtained channel to construct the third feature map for each channel.

[0032] Step 105: Perform feature fusion on the first feature map and the third feature map to obtain a fused feature map, and train the foreign object detection model based on the fused feature map, the annotation result of the training image, and a preset loss function.

[0033] Exemplarily, the first feature map and the third feature map represent features at different levels or scales. For example, the first feature map may contain more underlying detailed features, such as edge, texture and other information. The third feature map may contain higher-level semantic features, such as the overall shape of the object, category features, etc. After fusing them, the model can utilize the feature information at these different scales simultaneously. Fusing different feature maps can increase the diversity and richness of features, enabling the model to learn more discriminative feature representations. Thus, it can more comprehensively describe the foreign objects in the image and improve the recognition and localization ability of foreign objects.

[0034] In the embodiment of the present application, after extracting the respective first feature maps of the training image in different channels, based on the spatial attention mechanism, the first attention map in the horizontal coordinate direction and the second attention map in the vertical coordinate direction of each first feature map are respectively extracted, and based on the first attention map, the second attention map, and the first feature map, a second feature map corresponding to each channel is obtained. Since the first attention map and the second attention map represent the position information in the horizontal coordinate direction and the vertical coordinate direction, therefore, on the basis of the first feature map, the second feature map incorporates the global information of the first feature maps in different directions and can better capture the relationship between different positions in the input training image. Based on the channel attention mechanism, the channel attention weight of each feature in each second feature map is calculated, and based on the channel attention weight, the features in the second feature map are weighted and averaged, and a third feature map under each channel is constructed according to the weighted average result. The third feature map models the importance of channels on the basis of the second feature map, dynamically adjusts the channel weights, highlights the key features, and suppresses the unimportant features, further enhancing the feature representation ability. The foreign object detection model trained with the training image can more accurately learn the features of the foreign object, thereby improving the foreign object detection accuracy.

[0035] Figure 3 This is another method for training a foreign object detection model provided by the embodiment of the present application. Refer to Figure 3 , the method includes: Step 201, obtain a training image, and label the object in the training image to obtain the labeling result of the training image; the training image includes: an image of a conveyor belt with an object; the object includes a foreign object.

[0036] This step can refer to the above step 101 and will not be elaborated here.

[0037] Step 202, extract the respective first feature maps of the training image in different channels through a convolutional layer and a cross-stage fusion layer.

[0038] This step can refer to the above step 102 and will not be elaborated here.

[0039] Step 203: Based on the spatial attention mechanism, extract the first attention map in the horizontal coordinate direction and the second attention map in the vertical coordinate direction of each first feature map respectively, and obtain the second feature map corresponding to each channel based on the first attention map, the second attention map, and the first feature map.

[0040] This step can refer to the above step 103 and will not be elaborated here.

[0041] Optionally, step 203 may specifically include: Sub-step 2031: Concatenate the first attention map and the second attention map to obtain the concatenated attention map; Sub-step 2032: Perform a normalization operation on the concatenated attention map to obtain the normalized attention map; Sub-step 2033: Determine the second feature map according to the normalized attention map and the first feature map.

[0042] For sub-steps 2031 - 2033, the normalization operation includes: Batch Normalization (BatchNorm), Non-linear Normalization, and Sigmoid Normalization. Concatenate the first attention map and the second attention map, and through a convolutional layer, obtain the concatenated attention map with a dimension of C / r×1×(W + H). Then perform normalization on the concatenated attention map, that is, Batch Normalization, Non-linear Normalization, and Sigmoid Normalization, to obtain the normalized attention map. The following will be described in detail in combination with Figure 2 for elaboration.

[0043] Concatenation and Convolution (Concat+Conv2d): Concatenate the first attention map and the second attention map along the channel dimension, and then perform a convolution operation to integrate the spatial information of different dimensions. The output dimension becomes C / r×1×(W + H), where r is a scaling factor. First, use pooling kernels with dimensions (H,1) and (1,W) to encode each channel along the horizontal coordinate direction and the vertical coordinate direction. The specific calculation formula for the output of the c-th channel with a height of h is as follows:

[0044] where, represents the feature value at the position with a height coordinate of h and a width coordinate of i on the c-th channel, W represents the width value of the feature map, represents the output value of the c-th channel with a height of h after the pooling operation.

[0045] Similarly, the output expression of the c-th channel with a width of w is as follows:

[0046] Among them, represents the eigenvalue at the position with height coordinate j and width coordinate w on the c-th channel, and H represents the height value of the feature map. represents the output value of the c-th channel with width w after the pooling operation.

[0047] Normalization and Non-linear Transformation (BatchNorm + Non-linear): The convolution result is processed by batch normalization and non-linear activation functions. Batch normalization stabilizes the training process, accelerates convergence, and makes the model more effective in learning spatial position features. Non-linear transformation (such as ReLU, etc.) increases the non-linear expression ability of the model, enabling the model to learn more complex spatial attention weight mapping relationships.

[0048] Convolution and Activation (Conv2d + Sigmoid): Then, through two parallel convolutional layers, the feature dimensions are further adjusted to match the first attention map and the second attention map in the spatial dimension. Then, through the Sigmoid activation function, the output value is mapped between 0 and 1 to generate a normalized attention map. The Sigmoid output value represents the importance degree of the corresponding spatial position. The closer the value is to 1, the more important this position is in feature extraction.

[0049] Optionally, the normalized attention map has weights corresponding one by one to the features in the first feature map; Sub-step 2033 may specifically include: Sub-step 20331: Based on the weights of the normalized attention map, perform weighted averaging on the features in the first feature map, and construct the second feature map according to the weighted average result.

[0050] Exemplarily, referring to Figure 2 , after obtaining the weights of the normalized attention map, multiply the normalized attention map by the corresponding elements of the first feature map output by the residual module, assign weight values to each pixel point on each channel, and realize the weighted adjustment of each spatial position of the first feature map. The important position features are enhanced, and the unimportant position features are suppressed, obtaining the final output feature map, and completing the processing of the feature map by the spatial attention mechanism.

[0051] Step 204: Based on the channel attention mechanism, calculate the channel attention weights of each feature in each second feature map, and based on the channel attention weights, perform weighted averaging on the features in the second feature map, and construct the third feature map under each channel according to the weighted average result.

[0052] This step can refer to the above Step 104 and will not be elaborated here.

[0053] Optionally, step 204 may specifically include: Sub-step 2041: Perform global average pooling on the features in the second feature map to obtain a three-dimensional tensor, and perform non-linear mapping on the three-dimensional tensor through a fully connected network and a non-linear activation function to obtain the channel attention weights of each feature in the second feature map.

[0054] Exemplarily, referring to Figure 4 , X represents the input feature map of the channel attention mechanism, with a feature channel number of C’, width of W’, and height of H’. After a series of general transformations such as convolution, a second feature map U with a feature channel number of C, width of W, and height of H is obtained. Global average pooling is performed on the input second feature map U through a compression (Squeeze) operation to compress the channel dimension. The local description operator of the feature information distribution obtained by performing global average pooling on the second feature map U through the Squeeze operation can be understood as the numerical distribution of the C feature maps obtained at this layer. The input second feature map is U, with a shape of C×H×W, and a tensor of size C×1×1 will be obtained after global average pooling. The specific calculation formula is as follows:

[0055] where, represents the feature value at the position with width coordinate j and height coordinate i on the c-th channel, W represents the width value of the feature map, H represents the height value of the feature map, represents the compression operation on the feature map, represents the feature map obtained after the compression operation.

[0056] After obtaining the three-dimensional tensor z, through the excitation stage, that is, a two-layer fully connected network, the importance weights of the channels are learned. This fully connected network consists of a compression operation and an excitation operation. The compression operation inputs the C×1×1 tensor obtained in the previous step into a fully connected layer to compress the channel number to a smaller value. The excitation operation inputs the compressed feature map into a ReLU activation function to obtain an excitation vector. The specific calculation formula is as follows:

[0057] where, δ represents the ReLU activation function, σ represents the Sigmoid activation function, W1 represents the dimensionality reduction parameter, W2 represents the dimensionality increase parameter, z represents the feature map obtained after the compression operation, W includes the W1 parameter and the W2 parameter, s represents the excitation vector, and Fex represents the excitation operation.

[0058] Expand the excitation vector obtained in the previous step into a C×1×1 tensor, and multiply it element-wise with the features in the second feature map U to obtain a weighted feature map for each channel, which is the corresponding third feature map. The specific calculation formula is as follows:

[0059] where Fscale represents the recalibration operation, s c represents the excitation vector of the c-th channel, u c represents the second feature map of the c-th channel, represents the third feature map of the c-th channel.

[0060] Step 205: Perform feature fusion on the first feature map and the third feature map to obtain a fused feature map, and based on the fused feature map, obtain the foreign object recognition result of the foreign object detection model for the training image.

[0061] Exemplarily, performing feature fusion on the first feature map and the third feature map means integrating the different feature information contained in these two feature maps to generate a new fused feature map. The first feature map and the third feature map may be outputs from different layers or different branches of the foreign object detection model, and they each contain features regarding different aspects of the training image. The feature fusion method can be a concatenation operation. For example, the two feature maps are connected in the channel dimension. The feature fusion method can also be a weighted sum, which is not specifically limited here.

[0062] Using the fused feature map as the input, perform subsequent classification, regression, etc. operations through the foreign object detection model to obtain the foreign object recognition result. Fusing multiple feature maps can enable the model to learn more robust feature representations, reduce the dependence on a single feature map, and thus maintain good detection performance in actual application scenarios.

[0063] Optionally, the foreign object detection model includes: a feature extraction module; the feature extraction module includes: a convolutional layer and a cross-stage fusion layer; After step 202, the method further includes: Step A1: Perform a convolution operation on the second feature map through the convolutional layer, and perform feature extraction on the convolution result through the cross-stage fusion layer to obtain a fourth feature map; Step 205 can specifically include: Sub-step 2051: Perform an upsampling operation on the third feature map, and splice the feature map obtained by the upsampling operation with the fourth feature map to obtain a spliced feature map; Sub-step 2052: Perform feature extraction on the spliced feature map through the cross-stage fusion layer, and perform the upsampling operation on the feature extraction result to obtain a fifth feature map; Sub-step 2053: Splice the fifth feature map with the first feature map to obtain the fused feature map.

[0064] For step A1 and sub-steps 2051 - 2053, see Figure 5 , the feature map finally output by the feature extraction module is the third feature map, with a dimension of 20×20×1024. The feature map output by the first spatial attention mechanism is the second feature map, with a dimension of 80×80×256. Perform a convolution operation on the second feature map through a convolutional layer, and then perform feature extraction on the convolution result through the cross-stage fusion layer to obtain a fourth feature map, with a dimension of 40×40×512. Perform an upsampling operation on the third feature map, and splice the feature map obtained by the upsampling operation with the fourth feature map to obtain a spliced feature map, with a dimension of 40×40×1536. Perform feature extraction on the spliced feature map through the cross-stage fusion layer, and perform the upsampling operation on the feature extraction result to obtain a fifth feature map, with a dimension of 80×80×512. Splice the fifth feature map with the first feature map to obtain a fused feature map, with a dimension of 80×80×756.

[0065] Step 206: Train the foreign object detection model based on the foreign object recognition result, the annotation result of the training image, and the preset loss function to obtain a trained foreign object detection model.

[0066] Exemplarily, the foreign object recognition result is the output obtained by the foreign object detection model after processing the training image based on the fused feature map, including information such as the category, location (such as bounding box coordinates), and confidence of the foreign object. The annotation result of the training image is the true information pre-annotated for the training image by humans, that is, the accurate category and location of the foreign object actually existing in the image. The annotation result is used as the target for the model to learn and is used to measure the accuracy of the model prediction. The preset loss function is a function that measures the difference between the model prediction result and the true annotation result. By calculating the value of the loss function, the current prediction performance of the model can be known.

[0067] Based on the foreign object recognition result, the annotation result of the training image, and the preset loss function, the specific steps for training the foreign object detection model are as follows: Calculate the loss: Input the foreign object recognition result and the annotation result of the training image into the preset loss function to calculate a loss value. This loss value reflects the degree of difference between the model prediction and the actual situation. The smaller the loss value, the more accurate the model prediction.

[0068] Backpropagation: According to the calculated loss value, use the backpropagation algorithm to calculate the gradients of the loss function with respect to each parameter in the model. The gradient represents the rate of change of the loss function with respect to each parameter. By the gradient, we can know in which direction each parameter should be adjusted to reduce the loss value.

[0069] Parameter update: Use an optimization algorithm (such as stochastic gradient descent) to update the parameters of the model according to the calculated gradients. The optimization algorithm will adjust the parameters of the model at a certain learning rate based on the magnitude and direction of the gradients, so that the model can be closer to the true annotation results in subsequent predictions.

[0070] Repeat iteration: Repeat the above steps to continuously train the model until the performance of the model reaches a satisfactory level or the preset number of training epochs is reached.

[0071] Optionally, the preset loss function includes: overlap loss, center distance loss, and width-height loss; the foreign object recognition result includes: the predicted bounding box of the foreign object; the annotation result of the training image includes: the true bounding box of the foreign object; the overlap loss reflects the degree of overlap between the predicted bounding box and the true bounding box; the center distance loss reflects the degree of offset of the center point of the predicted bounding box relative to the center point of the true bounding box in terms of position; the width-height loss reflects the degree of offset of the predicted bounding box relative to the true bounding box in terms of width and height; Step 206 may specifically include: Sub-step 2061: Calculate the first ratio of the intersection area and the union area of the predicted bounding box and the true bounding box to obtain the overlap loss; Sub-step 2062: Calculate the square of the coordinate difference of the center points of the predicted bounding box and the true bounding box, and calculate the second ratio of the square of the difference to the square of the diagonal length to obtain the center distance loss; the diagonal is the minimum bounding rectangle of the predicted bounding box and the true bounding box; Sub-step 2063: Calculate the square of the width difference between the predicted bounding box and the true bounding box, and the square of the width of the minimum bounding rectangle that simultaneously contains the predicted bounding box and the true bounding box, and calculate the third ratio of the square of the width difference to the square of the width; Sub-step 2064: Calculate the square of the height difference between the predicted bounding box and the true bounding box, and the square of the height of the minimum bounding rectangle that simultaneously contains the predicted bounding box and the true bounding box, and calculate the fourth ratio of the square of the height difference to the square of the height; Sub-step 2065: Calculate the sum result of the third ratio and the fourth ratio to obtain the width-height loss, and train the parameters of the foreign object detection model according to the overlap loss, the center distance loss, and the width-height loss to obtain a trained foreign object detection model.

[0072] For sub-steps 2061 - 2065, the preset loss function can be the EIOU loss, which is mainly divided into three parts. The overlap loss is used to evaluate the coordinate difference between the predicted bounding box and the ground truth bounding box, and the center distance loss provides direct feedback on the localization error of the predicted bounding box. The loss is adjusted according to the direction and orientation of the object, enhancing the model's ability to recognize objects in different directions, especially when dealing with non-standard objects. The width-height loss is used to consider the aspect ratio and position of the bounding box, improving the ability of the framework to match the shape of the ground truth bounding box and making the convergence speed faster.

[0073] The specific calculation formula of the preset loss function is as follows:

[0074] Among them, the IOU is calculated by dividing the intersection of the predicted bounding box and the ground truth bounding box by the union of the two. b, b gt represent the center points of the predicted bounding box and the ground truth bounding box respectively. w, w gt represent the widths of the predicted bounding box and the ground truth bounding box respectively. h, h gt represent the heights of the predicted bounding box and the ground truth bounding box respectively. ρ represents the Euclidean distance between the center points of the predicted bounding box and the ground truth bounding box. w c and h c are the width and height of the smallest bounding rectangle covering the predicted bounding box and the ground truth bounding box.

[0075] Optionally, the method further includes: Step 207: Convert the format of the trained foreign object detection model to the target format; Step 208: Configure the trained foreign object detection model to generate a configuration file; the configuration file includes: input dimension, output dimension, and the number of categories; Step 209: Package the trained foreign object detection model according to the target format and the configuration file to obtain a deployable model file.

[0076] Regarding steps 207 - 209, the Jetson Orin NX computing platform refers to deploying the trained model on the Jetson Orin NX device, and accelerating and optimizing the model inference by combining TensorRT and tkDNN, and performing real-time processing of video streams on the deployed model by combining DeepStream to achieve high-performance deep learning inference and real-time video analysis, improving the real-time detection performance of the model. DeepStream is an official deployment application installed on the deployment device Jetson Orin NX.

[0077] TensorRT is a deep learning inference optimizer by NVIDIA, which is used to efficiently run trained deep learning models on GPUs. It can convert models trained by common deep learning frameworks (such as TensorFlow, PyTorch, etc.) into high-performance inference engines, thus accelerating the inference process of the models. It can be understood that the same model is expressed in different ways for convenient deployment on a specified platform. The main role of TensorRT is to improve the inference speed and efficiency of deep learning models. By optimizing the computational graph and network structure of the model, methods such as reducing redundant calculations, merging layers, and streamlining precision are used to improve the performance of the model.

[0078] tkDNN parses darknet layer by layer, then uses TensorRT to build a network layer by layer, generates the engine file of TensorRT, and then the inference file format is yolov8.rt.

[0079] The specific process of model deployment and inference on the Jetson Orin NX computing platform is as follows: 1. Output the trained and optimized model in the required format for deployment and import it for inference acceleration; 2. Model inference acceleration stage: Optimize the exported model with TensorRT and tkDNN, improve the model parameter information, and improve the inference performance. Convert the model into an inference engine file, which is the optimized deployment file; 3. Apply the inference file deployed on the Jetson Orin NX computing platform, combine it with the DeepStream application configuration file in the platform, and input the specified source, inference configuration, and output parameters; The input source is the video RTSP stream, MP4 file, or camera for foreign object recognition. The inference configuration is the improved YOLOV8 model accelerated by TensorRT. The output is the video with foreign objects marked with anchor boxes after recognition; 4. Run the DeepStream application to complete the model deployment work combining TensorRT, tkDNN, and DeepStream, and achieve the deployment and real-time detection of the model.

[0080] The Jetson Orin NX computing platform realizes the inference acceleration of the detection system and the real-time processing of video streams, making the model run more efficiently on the GPU, improving the model detection speed, and meeting the scenario requirements of foreign object recognition on the conveyor belt.

[0081] In the embodiment of the present application, after extracting the respective first feature maps of the training image in different channels, based on the spatial attention mechanism, the first attention map in the horizontal coordinate direction and the second attention map in the vertical coordinate direction of each first feature map are respectively extracted, and based on the first attention map, the second attention map, and the first feature map, the second feature map corresponding to each channel is obtained. Since the first attention map and the second attention map represent the position information in the horizontal coordinate direction and the vertical coordinate direction, therefore, on the basis of the first feature map, the second feature map incorporates the global information of the first feature maps in different directions and can better capture the relationship between different positions in the input training image. Based on the channel attention mechanism, the channel attention weights of each feature in each second feature map are calculated, and based on the channel attention weights, the features in the second feature map are weighted and averaged, and the third feature map under each channel is constructed according to the weighted average result. On the basis of the second feature map, the third feature map models the importance of the channels, dynamically adjusts the channel weights, highlights the key features, and suppresses the unimportant features, further improving the representation ability of the features. The foreign object detection model trained with the training image can more accurately learn the features of the foreign object, thereby improving the accuracy of foreign object detection.

[0082] Figure 6 This is a conveyor belt foreign object detection and warning system based on the Jetson Orin NX platform provided by the embodiment of the present application. Refer to Figure 6 , for the item situation in the conveyor belt scenario, after the camera captures and collects the images, the image data collected by the camera is transmitted to the deep learning processing module in the data processing sub-module. Based on the Jetson Orin NX platform, this module uses deep learning algorithms to analyze and process the image data to identify whether there are foreign objects. The data analyzed by the deep learning processing module is transmitted to the foreign object monitoring center, which determines whether there are foreign objects. If it is determined that there are foreign objects, the foreign object monitoring center will transmit the information to the foreign object warning processing module for subsequent processing work.

[0083] Figure 7 This is a detailed step flowchart of conveyor belt foreign object detection based on the Jetson Orin NX platform provided by the embodiment of the present application. Refer to Figure 7, this step may specifically include: Obtaining data: Obtaining conveyor belt foreign object data and establishing a foreign object data set. Data preprocessing: Labeling and classifying the pictures in the foreign object data set. Data partitioning: Partitioning the preprocessed data set into a training set, a test set, and a validation set. Model construction: Constructing a foreign object detection model that adds channel and spatial attention mechanisms and an improved loss function. Training the model: Inputting the training set into the improved detection model to train the detection model; updating the model weight parameters through the loss function and the optimizer, and obtaining a trained conveyor belt foreign object detection model after several rounds of training. Model evaluation: Using the validation set to validate the trained conveyor belt foreign object detection model; using the test set to test and evaluate the trained model weights; obtaining the final conveyor belt foreign object detection model. Model deployment: Converting the final conveyor belt foreign object detection model into a quantization model and deploying the model to an embedded platform through a combined tool.

[0084] Figure 8 is a schematic diagram of the deployment of the Jetson Orin NX platform provided by the embodiments of the present application. Refer to Figure 8 , which may specifically include: The data set provides samples for model learning and is the training basis. Using the data set to train the foreign object detection model to obtain a trained foreign object detection model. During the model inference acceleration process, performing inference acceleration processing on the trained foreign object detection model, aiming to improve the model inference speed, reduce latency, and make it more suitable for actual application scenarios. After inference acceleration, converting the model into a model deployment file, which contains information such as the model structure and parameters for model deployment. Finally, applying the deployed model to the Jetson Orin NX embedded platform, which has a certain computing power and can support the model to run in real time in actual scenarios (such as edge devices) to complete tasks such as object detection.

[0085] In an embodiment of the present application, after extracting the respective first feature maps of the training image in different channels, based on the spatial attention mechanism, the first attention map in the horizontal coordinate direction and the second attention map in the vertical coordinate direction of each first feature map are respectively extracted, and based on the first attention map, the second attention map, and the first feature map, a second feature map corresponding to each channel is obtained. Since the first attention map and the second attention map represent the position information in the horizontal coordinate direction and the vertical coordinate direction, therefore, on the basis of the first feature map, the second feature map incorporates the global information of the first feature maps in different directions and can better capture the relationships between different positions in the input training image. Based on the channel attention mechanism, the channel attention weights of each feature in each second feature map are calculated, and based on the channel attention weights, the features in the second feature map are weighted and averaged, and a third feature map under each channel is constructed according to the weighted average result. On the basis of the second feature map, the third feature map models the importance of channels, dynamically adjusts the channel weights, highlights key features, and suppresses unimportant features, further improving the representation ability of features. The foreign object detection model trained with the training image can more accurately learn the features of foreign objects, thereby improving the accuracy of foreign object detection.

[0086] Figure 9 This is a foreign object detection method provided by an embodiment of the present application. Refer to Figure 9 , the method includes: Step 301, obtain a video to be detected; Step 302, input the video frames in the video to be detected into a foreign object detection model to detect the foreign object; the video frames include: images of a conveyor belt with object objects; the object objects include foreign objects.

[0087] Exemplarily, the foreign object detection model is trained by the above-mentioned training method of the foreign object detection model.

[0088] Optionally, the method further includes: Step 303, classify the foreign object, and divide the foreign object into a first category, a second category, and a third category; the first category is a harmful foreign object; the second category is a foreign object that affects the product quality; the third category is a foreign object other than those belonging to the first category and the second category; Step 304, if the foreign object belongs to the first category, issue a warning to stop immediately; Step 305, if the foreign object belongs to the second category, trigger an audible and visual alarm and generate a maintenance work order; Step 306, if the foreign object belongs to the third category, record a log and prompt the existence of a foreign object.

[0089] For steps 303 - 306, during the preliminary training process, possible foreign objects were classified into 4 major categories: large blocks (large stones or concrete blocks), metals (steel bars, metal fragments, iron sheets, etc.), fibers (plastics, gunny bags, ropes, etc.), and humans (used to identify personnel approaching the conveyor belt). They were divided according to risk and foreign object size. Red: Foreign objects such as large stones, concrete, metal fragments, and steel bars that can cause equipment damage, and immediate shutdown for handling is required. Orange: Foreign objects such as plastics and ropes that can affect product quality, triggering an audible and visual alarm and generating a maintenance work order. Yellow: Personnel, small fibers, etc., recording logs and indicating their presence. The video stream is processed in real time by Jetson Orin NX to perform primary classification, and the results are transmitted to the central server for data fusion analysis to optimize the classification decision.

[0090] In the embodiment of the present application, after extracting the respective first feature maps of the training image in different channels, based on the spatial attention mechanism, the first attention map in the horizontal coordinate direction and the second attention map in the vertical coordinate direction of each first feature map are respectively extracted, and based on the first attention map, the second attention map, and the first feature map, the second feature map corresponding to each channel is obtained. Since the first attention map and the second attention map represent the position information in the horizontal coordinate direction and the vertical coordinate direction, therefore, on the basis of the first feature map, the second feature map incorporates the global information of the first feature maps in different directions and can better capture the relationship between different positions in the input training image. Based on the channel attention mechanism, the channel attention weights of each feature in each second feature map are calculated, and based on the channel attention weights, the features in the second feature map are weighted and averaged, and the third feature map under each channel is constructed according to the weighted average result. The third feature map models the importance of channels on the basis of the second feature map, dynamically adjusts the channel weights, highlights the key features, and suppresses the unimportant features, further improving the feature representation ability. The foreign object detection model trained with the training image can more accurately learn the features of foreign objects, thereby improving the foreign object detection accuracy.

[0091] Figure 10 It is a schematic structural diagram of a training device for a foreign object detection model provided by an embodiment of the present application. The training device 400 may include the following modules: An acquisition module 401, configured to acquire a training image and label the object in the training image to obtain a labeling result of the training image; the training image includes: an image of a conveyor belt with an object; the object includes a foreign object; A first extraction module 402, configured to extract the respective first feature maps of the training image in different channels through a convolutional layer and a cross-stage fusion layer; A determination module 403, configured to respectively extract a first attention map in the horizontal coordinate direction and a second attention map in the vertical coordinate direction of each first feature map based on a spatial attention mechanism, and obtain a second feature map corresponding to each channel based on the first attention map, the second attention map, and the first feature map; A construction module 404, configured to calculate the channel attention weights of each feature in each second feature map based on a channel attention mechanism, perform weighted averaging on the features in the second feature map based on the channel attention weights, and construct a third feature map for each channel according to the weighted averaging result; A training module 405, configured to perform feature fusion on the first feature map and the third feature map to obtain a fused feature map, and train the foreign object detection model based on the fused feature map, the annotation result of the training image, and a preset loss function.

[0092] Optionally, the determination module includes: A first splicing sub-module, configured to splice the first attention map and the second attention map to obtain a spliced attention map; A normalization sub-module, configured to perform a normalization operation on the spliced attention map to obtain a normalized attention map; A determination sub-module, configured to determine the second feature map according to the normalized attention map and the first feature map.

[0093] Optionally, the normalized attention map has weights corresponding one by one to the features in the first feature map; the determination sub-module includes: A weighted averaging unit, configured to perform weighted averaging on the features in the first feature map based on the weights of the normalized attention map, and construct the second feature map according to the weighted averaging result.

[0094] Optionally, the construction module includes: A mapping sub-module, configured to perform global average pooling on the features in the second feature map to obtain a three-dimensional tensor, and perform a non-linear mapping on the three-dimensional tensor through a fully connected network and a non-linear activation function to obtain the channel attention weights of each feature in the second feature map.

[0095] Optionally, the foreign object detection model includes: a feature extraction module; the feature extraction module includes: a convolutional layer and a cross-stage fusion layer; the apparatus further includes: A second extraction module, configured to perform a convolution operation on the second feature map through the convolutional layer, and perform feature extraction on the convolution result through the cross-stage fusion layer to obtain a fourth feature map; The training module includes: The first extraction sub-module is used to perform an upsampling operation on the third feature map, and splice the feature map obtained by the upsampling operation with the fourth feature map to obtain a spliced feature map; The second extraction sub-module is used to perform feature extraction on the spliced feature map through the cross-stage fusion layer, and perform the upsampling operation on the feature extraction result to obtain a fifth feature map; The second splicing sub-module is used to splice the fifth feature map with the first feature map to obtain the fusion feature map.

[0096] Optionally, the training module includes: An acquisition sub-module is used to obtain the foreign object recognition result of the foreign object detection model for the training image based on the fusion feature map; A training sub-module is used to train the foreign object detection model based on the foreign object recognition result, the annotation result of the training image, and the preset loss function to obtain a trained foreign object detection model.

[0097] Optionally, the preset loss function includes: overlap loss, center distance loss, and width-height loss; the foreign object recognition result includes: the predicted bounding box of the foreign object; the annotation result of the training image includes: the ground truth bounding box of the foreign object; the overlap loss reflects the overlap degree between the predicted bounding box and the ground truth bounding box; the center distance loss reflects the offset degree of the center point of the predicted bounding box relative to the center point of the ground truth bounding box in position; the width-height loss reflects the offset degree of the predicted bounding box relative to the ground truth bounding box in width and height; The training sub-module includes: The first calculation unit is used to calculate the first ratio of the intersection area to the union area of the predicted bounding box and the ground truth bounding box to obtain the overlap loss; The second calculation unit is used to calculate the square of the coordinate difference of the center points of the predicted bounding box and the ground truth bounding box, and calculate the second ratio of the square of the difference to the square of the diagonal length to obtain the center distance loss; the diagonal is the minimum bounding rectangle of the predicted bounding box and the ground truth bounding box; The third calculation unit is used to calculate the square of the width difference between the predicted bounding box and the ground truth bounding box, and the square of the width of the minimum bounding rectangle that simultaneously contains the predicted bounding box and the ground truth bounding box, and calculate the third ratio of the square of the width difference to the square of the width; The fourth calculation unit is used to calculate the square of the height difference between the predicted bounding box and the ground truth bounding box, and the square of the height of the minimum bounding rectangle that simultaneously contains the predicted bounding box and the ground truth bounding box, and calculate the fourth ratio of the square of the height difference to the square of the height; A fifth computing unit, configured to calculate the sum result of the third ratio and the fourth ratio to obtain the width-height loss, and train the parameters of the foreign object detection model according to the overlap loss, the center distance loss, and the width-height loss, so as to obtain a trained foreign object detection model.

[0098] Figure 11 FIG. 4 is a schematic structural diagram of a foreign object detection device provided by an embodiment of the present application. The detection device 500 may include the following modules: An acquisition module 501, configured to acquire a video to be detected; A detection module 502, configured to input a video frame in the video to be detected into a foreign object detection model to detect the foreign object; Wherein, the video frame includes: an image of a conveyor belt with an object; the object includes a foreign object.

[0099] In the embodiment of the present application, after the respective first feature maps of the training image in different channels are extracted, based on the spatial attention mechanism, the first attention map in the horizontal coordinate direction and the second attention map in the vertical coordinate direction of each first feature map are respectively extracted, and based on the first attention map, the second attention map, and the first feature map, a second feature map corresponding to each channel is obtained. Since the first attention map and the second attention map represent the position information in the horizontal coordinate direction and the vertical coordinate direction, therefore, on the basis of the first feature map, the second feature map incorporates the global information of the first feature maps in different directions and can better capture the relationship between different positions in the input training image. Based on the channel attention mechanism, the channel attention weight of each feature in each second feature map is calculated, and based on the channel attention weight, the features in the second feature map are weighted and averaged, and a third feature map under each channel is constructed according to the weighted average result. The third feature map models the importance of channels on the basis of the second feature map, dynamically adjusts the channel weights, highlights the key features, and suppresses the unimportant features, further improving the feature representation ability. The foreign object detection model trained by the training image can more accurately learn the features of the foreign object, thereby improving the foreign object detection accuracy.

[0100] An embodiment of the present application discloses an electronic device, as Figure 12 shown, including: a processor 601 and a memory 602. The memory 602 stores a program or instruction that can run on the processor 601. When the program or instruction is executed by the processor 601, the method described in any one of the above is implemented.

[0101] An embodiment of the present application discloses a computer-readable storage medium, and the computer program stored therein enables a processor to execute the method described in any one of the above.

[0102] It should be noted that in this text, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements not only includes those elements, but also includes other elements not explicitly listed, or further includes elements inherent in such a process, method, article or device. Without further limitation, an element defined by the statement "including one..." does not exclude the existence of additional identical elements in the process, method, article or device including such an element.

[0103] The embodiments of the present application have been described above in conjunction with the accompanying drawings. However, the present application is not limited to the above specific embodiments. The above specific embodiments are merely illustrative and not restrictive. Under the inspiration of the present application, those of ordinary skill in the art can also make many forms without departing from the scope protected by the purpose of the present application, and all of them fall within the protection scope of the present application.

Claims

1. A training method for a foreign object detection model, characterized in that The method includes: Obtaining a training image, and annotating the object in the training image to obtain an annotation result of the training image; the training image includes: an image of a conveyor belt with an object; the object includes a foreign object; Extracting respective first feature maps of the training image in different channels through a convolutional layer and a cross-stage fusion layer; Based on a spatial attention mechanism, respectively extracting a first attention map in the horizontal coordinate direction and a second attention map in the vertical coordinate direction of each first feature map, and obtaining a second feature map corresponding to each channel based on the first attention map, the second attention map, and the first feature map; Based on a channel attention mechanism, calculating a channel attention weight for each feature in each second feature map, and performing weighted averaging on the features in the second feature map based on the channel attention weight, and constructing a third feature map for each channel according to the weighted averaging result; Performing feature fusion on the first feature map and the third feature map to obtain a fused feature map, and training the foreign object detection model based on the fused feature map, the annotation result of the training image, and a preset loss function.

2. The method according to claim 1, wherein The obtaining a second feature map corresponding to each channel based on the first attention map, the second attention map, and the first feature map includes: Concatenating the first attention map and the second attention map to obtain a concatenated attention map; Performing a normalization operation on the concatenated attention map to obtain a normalized attention map; Determining the second feature map according to the normalized attention map and the first feature map.

3. The method according to claim 2, wherein The normalized attention map has weights corresponding one by one to the features in the first feature map; The determining the second feature map according to the normalized attention map and the first feature map includes: Performing weighted averaging on the features in the first feature map based on the weights of the normalized attention map, and constructing the second feature map according to the weighted averaging result.

4. The method according to claim 1, characterized in that The calculating a channel attention weight for each feature in each second feature map based on the channel attention mechanism includes: Performing global average pooling on the features in the second feature map to obtain a three-dimensional tensor, and performing a non-linear mapping on the three-dimensional tensor through a fully connected network and a non-linear activation function to obtain the channel attention weight for each feature in the second feature map.

5. The method according to claim 1, wherein The foreign object detection model includes: a feature extraction module; the feature extraction module includes: a convolutional layer and a cross-stage fusion layer; After respectively extracting a first attention map in the horizontal coordinate direction and a second attention map in the vertical coordinate direction of each first feature map based on the spatial attention mechanism, and obtaining a second feature map corresponding to each channel based on the first attention map, the second attention map, and the first feature map, the method further includes: Performing a convolution operation on the second feature map through the convolutional layer, and performing feature extraction on the convolution result through the cross-stage fusion layer to obtain a fourth feature map; Performing feature fusion on the first feature map and the third feature map to obtain a fused feature map includes: Performing an upsampling operation on the third feature map, and splicing the feature map obtained by the upsampling operation with the fourth feature map to obtain a spliced feature map; Performing feature extraction on the spliced feature map through the cross-stage fusion layer, and performing the upsampling operation on the feature extraction result to obtain a fifth feature map; Splicing the fifth feature map with the first feature map to obtain the fused feature map.

6. The method according to claim 1, characterized in that, Training the foreign object detection model based on the fused feature map, the annotation result of the training image, and a preset loss function includes: Based on the fused feature map, obtaining the foreign object recognition result of the foreign object detection model for the training image; Training the foreign object detection model based on the foreign object recognition result, the annotation result of the training image, and the preset loss function to obtain a trained foreign object detection model.

7. The method according to claim 6, characterized in that, The preset loss function includes: overlap loss, center distance loss, and width-height loss; the foreign object recognition result includes: the predicted bounding box of the foreign object; the annotation result of the training image includes: the ground truth bounding box of the foreign object; the overlap loss reflects the overlapping degree between the predicted bounding box and the ground truth bounding box; the center distance loss reflects the offset degree of the center point of the predicted bounding box relative to the center point of the ground truth bounding box in terms of position; the width-height loss reflects the offset degree of the predicted bounding box relative to the ground truth bounding box in terms of width and height; Training the foreign object detection model based on the foreign object recognition result, the annotation result of the training image, and the preset loss function to obtain a trained foreign object detection model includes: Calculating the first ratio of the intersection area to the union area of the predicted bounding box and the ground truth bounding box to obtain the overlap loss; Calculating the square of the coordinate difference of the center points of the predicted bounding box and the ground truth bounding box, and calculating the second ratio of the square of the difference to the square of the diagonal length to obtain the center distance loss; the diagonal is the minimum bounding rectangle of the predicted bounding box and the ground truth bounding box; Calculating the square of the width difference between the predicted bounding box and the ground truth bounding box, and the square of the width of the minimum bounding rectangle that simultaneously contains the predicted bounding box and the ground truth bounding box, and calculating the third ratio of the square of the width difference to the square of the width; Calculating the square of the height difference between the predicted bounding box and the ground truth bounding box, and the square of the height of the minimum bounding rectangle that simultaneously contains the predicted bounding box and the ground truth bounding box, and calculating the fourth ratio of the square of the height difference to the square of the height; Calculating the sum result of the third ratio and the fourth ratio to obtain the width-height loss, and training the parameters of the foreign object detection model according to the overlap loss, the center distance loss, and the width-height loss to obtain a trained foreign object detection model.

8. A method for detecting foreign objects, characterized in that, Includes: Obtaining a video to be detected; Inputting the video frames in the video to be detected into the foreign object detection model to detect the foreign object; The foreign object detection model is trained by the training method of the foreign object detection model according to any one of claims 1-7; Among them, the video frame includes: an image of a conveyor belt with an object; the object includes a foreign object.

9. An electronic device, characterized in that, It includes a processor and a memory, and the memory stores programs or instructions that can run on the processor. When the programs or instructions are executed by the processor, the method described in any one of claims 1-8 is implemented.

10. A readable storage medium, characterized in that, When the instructions in the readable storage medium are executed by the processor of the electronic device, the electronic device is enabled to execute the method described in any one of method claims 1-8.

Images