Training method and device for foreign matter classification model of coal conveying belt

Abstract

The invention provides a training method and device for a foreign matter classification model of a coal conveying belt, and relates to the technical field of image processing of intelligent mines. The training method comprises the following steps: acquiring a training image and label information of the training image, inputting the training image into a foreign matter classification model, carrying out key point detection on the training image to obtain a key point set of the training image, determining a key local image set of the training image according to the key point set, and coding the key local image to obtain a foreign matter classification result. Carrying out self-attention perception on the coding sequence to obtain texture feature information of the key local images, carrying out cross attention perception on the texture feature information to obtain correlation information among the key local images, and determining a foreign matter classification prediction result of the training image according to the texture feature information and the correlation information. And according to the label information and the foreign matter classification prediction result, adjusting the foreign matter classification model until the training is finished to obtain a target foreign matter classification model.

Description

Technical Field

[0001] The present application relates to the field of natural language processing technology, and in particular to a training method and device for a foreign body classification model for a coal conveyor belt. Background Art

[0002] When foreign objects appear on the coal conveyor belt, they may cause damage to the coal conveyor belt. For example, when the foreign objects on the coal conveyor belt are large foreign objects such as gangue, anchor rods, etc., they often cause scratches and tears on the belt, further leading to problems such as coal accumulation and coal blockage at the coal discharge point. Summary of the Invention

[0003] The purpose of this application is to solve one of the technical problems in the related art at least to a certain extent.

[0004] To this end, the first purpose of this application is to propose a training method for a foreign body classification model for a coal conveyor belt, which can train a foreign body classification model to classify and identify foreign bodies appearing on the coal conveyor belt, thereby enabling timely early warning and effectively ensuring safe production in coal mines.

[0005] The second purpose of this application is to propose a method for classifying foreign matter on a coal conveyor belt.

[0006] The third object of this application is to provide a training device for a foreign matter classification model of a coal conveyor belt.

[0007] The fourth object of this application is to provide a foreign matter classification device for a coal conveyor belt.

[0008] The fifth objective of this application is to provide an electronic device.

[0009] A sixth object of this application is to provide a computer-readable storage medium.

[0010] The seventh object of this application is to provide a computer program product.

[0011] To achieve the above objectives, the first embodiment of the present application proposes a method for training a foreign matter classification model for a coal conveyor belt, comprising:

[0012] Obtaining a training image and label information of the training image;

[0013] Inputting the training image into a foreign body classification model, performing key point detection on the training image to obtain a key point set of the training image, and determining a key local image set of the training image based on the key point set of the training image;

[0014] Encoding the key local images in the key local image set to obtain a coding sequence of the training images;

[0015] Performing self-attention perception on the coding sequence to obtain texture feature information of the key local image;

[0016] Performing cross-attention perception on texture feature information of the key local images to obtain correlation information between the key local images;

[0017] Determining a foreign body classification prediction result of the training image based on the texture feature information and the correlation information;

[0018] According to the label information and the foreign body classification prediction result, the model parameters of the foreign body classification model are adjusted until the training is completed to obtain the target foreign body classification model.

[0019] To achieve the above-mentioned purpose, the second embodiment of the present application proposes a method for classifying foreign matter on a coal conveyor belt, comprising:

[0020] Acquire coal flow images of coal conveyor belts;

[0021] Inputting the coal flow image into a target foreign matter classification model to perform foreign matter classification, thereby obtaining a foreign matter classification result of the coal image;

[0022] Among them, the target foreign body classification model is a model trained using the training method described in the first aspect.

[0023] To achieve the above-mentioned purpose, the third embodiment of the present application proposes a training device for a foreign body classification model of a coal conveyor belt, comprising:

[0024] A sample acquisition unit, configured to acquire a training image and label information of the training image;

[0025] a local area acquisition unit, configured to input the training image into a foreign body classification model, perform key point detection on the training image to obtain a key point set of the training image, and generate a key local image set of the training image based on the key point set of the training image;

[0026] an encoding unit, configured to encode the key local images in the key local image set and determine an encoding sequence of the training images;

[0027] A first perception unit is configured to perform self-attention perception on the coding sequence to obtain texture feature information of the key local image;

[0028] a second perception unit, configured to perform cross-attention perception on the texture feature information of the key local images to obtain correlation information between the key local images;

[0029] A prediction unit, configured to determine a foreign body classification prediction result of the training image based on the texture feature information and the correlation information;

[0030] An adjustment unit is used to adjust the model parameters of the foreign body classification model according to the label information of the training image and the foreign body classification prediction result until the training is completed to obtain the target foreign body classification model.

[0031] To achieve the above-mentioned purpose, the fourth embodiment of the present application proposes a foreign matter sorting device for a coal conveyor belt, comprising:

[0032] An image acquisition unit, used for acquiring coal flow images of the coal conveyor belt;

[0033] An input unit, configured to input the coal flow image into a target foreign matter classification model to perform foreign matter classification, and obtain a foreign matter classification result of the coal image;

[0034] The target foreign body classification model is a model trained using the training method described in any one of claims 1 to 7.

[0035] To achieve the above-mentioned purpose, the fifth aspect embodiment of the present application proposes an electronic device, comprising: a processor; and a memory communicatively connected to the processor; the memory stores computer-executable instructions; the processor executes the computer-executable instructions stored in the memory, so that the processor can execute the method described in the above-mentioned first aspect embodiment.

[0036] To achieve the above-mentioned purpose, the sixth embodiment of the present application proposes a computer-readable storage medium on which a computer program is stored, and the computer instructions are used to enable the computer to execute the method described in the above-mentioned embodiment.

[0037] To achieve the above-mentioned purpose, the seventh embodiment of the present application proposes a computer program product, including a computer program, which implements the method described in the above-mentioned embodiment when executed by a processor.

[0038] The present application provides a training method and device for a foreign matter classification model for a coal conveyor belt, which obtains a training image and label information of the training image, performs key point detection on the training image to obtain a key point set, and determines a key local image set of the training image based on the key point set of the training image. Furthermore, the key local image is encoded to obtain a coding sequence, self-attention perception is performed on the coding sequence to obtain texture feature information of the key local image, and cross-attention perception is performed on the texture feature information of the key local image to obtain correlation information between the key local images. Based on the texture feature information and the correlation information, the foreign matter classification prediction result of the training image is determined, and the foreign matter classification model is adjusted based on the label information and the foreign matter classification prediction result until the training is completed and the target foreign matter classification model is obtained. Key local features are extracted through the self-attention mechanism, and further global attention is paid to the key local features through the cross-attention mechanism to obtain global features. The key local features and global features can be better utilized in the foreign body classification model, thereby solving the problems of difficulty in focusing key information for foreign body classification on coal conveyor belts and difficulty in perceiving correlation information between local information. This is conducive to improving the recognition accuracy of foreign body classification tasks on coal conveyor belts. According to the foreign body recognition results, the foreign body handling process of the coal conveyor belt is adapted to reduce the impact of foreign bodies on coal transmission and increase the service life of the coal conveyor belt.

[0039] Additional aspects and advantages of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0041] Figure 1 A flowchart of a method for training a foreign body classification model for a coal conveyor belt provided in an embodiment of the present application;

[0042] Figure 2 A schematic structural diagram of a foreign matter classification model for a coal conveyor belt provided in an embodiment of the present application;

[0043] Figure 3 A flowchart of another method for training a foreign body classification model for a coal conveyor belt provided in an embodiment of the present application;

[0044] Figure 4 A flowchart of another method for training a foreign body classification model for a coal conveyor belt provided in an embodiment of the present application;

[0045] Figure 5A schematic flow chart of a method for classifying foreign matter on a coal conveyor belt provided in an embodiment of the present application;

[0046] Figure 6 A schematic structural diagram of a training device for a foreign body classification model for a coal conveyor belt provided in an embodiment of the present application;

[0047] Figure 7 This is a schematic diagram of the results of foreign matter classification on a coal conveyor belt provided in an embodiment of the present application. DETAILED DESCRIPTION

[0048] The following describes in detail embodiments of the present application, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present application, and should not be construed as limiting the present application.

[0049] The following is an explanation of the training method and device of the foreign matter classification model for the coal conveyor belt, and the foreign matter classification method and device for the coal conveyor belt, with reference to the accompanying drawings.

[0050] Figure 1 This is a flow chart of a method for training a foreign body classification model for a coal conveyor belt provided in an embodiment of the present application. The execution subject of the method for training a foreign body classification model for a coal conveyor belt can be an electronic device or a server, which is not limited here.

[0051] like Figure 1 As shown, the training method of the foreign matter classification model for the coal conveyor belt may include but is not limited to the following steps:

[0052] S101: Obtain training images and label information of the training images.

[0053] In some embodiments, a historical coal flow video of a coal conveyor belt under a specified mine is obtained, wherein the historical coal flow video monitors the presence of foreign matter on the coal conveyor belt.

[0054] In some embodiments, the historical coal flow video is extracted according to set rules to obtain multiple image frames, wherein the image frames extracted from the historical coal flow video can be used as training images for training the model.

[0055] In some embodiments, multiple training images can be obtained by extracting from historical coal flow videos according to set frame intervals or time intervals.

[0056] In some embodiments, image frames with image quality that meets requirements can be extracted from historical coal flow videos to obtain multiple training images.

[0057] In some embodiments, the training image is an image containing a single object.

[0058] In some embodiments, the collected training images may be annotated to obtain label information of the training images.

[0059] In some embodiments, the training images may be annotated using an annotation tool or manually.

[0060] In some embodiments, distributed batch annotation can be performed on multiple training images. For example, the multiple training images can be divided into batches, and annotation tasks can be generated for each batch of training images. These tasks are then sent to different annotation platforms, where annotation tools within the annotation platforms annotate the training images in the annotation tasks. For another example, user input can be received on the annotation platform, and label information for the training images can be determined based on the input.

[0061] Furthermore, after obtaining the label information of the training images, a training image sequence or a training data set can be generated based on multiple training images and the label information of the training images. The foreign body classification model is trained based on the obtained training image sequence or training data set. Optionally, the foreign body classification model can be trained in batches based on the training images in the training image sequence or training data set. Each batch includes multiple training images. For example, multiple training images containing a single target can constitute a training image sequence X = {x i} i＝1,...,N , where x i represents the i-th training image.

[0062] In some embodiments, after acquiring the training images, a preset sample enhancement strategy can be used to perform data enhancement on the training images without affecting the semantic information to enrich the training samples. In this application, the preset sample enhancement strategy can generate multi-view training images, which can supplement the training samples and thus improve the generalization ability of the model.

[0063] S102: Input the training image into the foreign body classification model, perform key point detection on the training image to obtain a key point set of the training image, and generate a key local image set of the training image based on the key point set of the training image.

[0064] In some embodiments, an initial foreign object classification model may be pre-constructed. In order to enable the foreign object classification model to classify and identify foreign objects on the coal conveyor belt, the foreign object classification model may be trained based on the training image sequence obtained in step S101.

[0065] In some embodiments, the training images in the training image sequence are input into the foreign object classification model in batches, and the foreign object classification model is used to perform key point detection on the training images to obtain a key point set of the training images.

[0066] In some embodiments, after key point detection is performed on a training image, the obtained key points need to be screened or filtered. Optionally, abnormal key points are identified and removed, and a key point set of the training image is constructed based on the retained key points.

[0067] In some embodiments, for each key point in the key point set, a key local image corresponding to the key point is generated. For example, a fixed size is expanded outward from the key point to obtain the key local image of the key point. That is, the key local image of the key point can be the image area including the key point.

[0068] Furthermore, a key local image set of the training image can be obtained based on the key local images of the key points. It can be understood that the key local image set includes the retained key local image of each key point.

[0069] S103 , encoding the key local images in the key local image set to determine an encoding sequence of the training images.

[0070] In some embodiments, the key local images in the key local image set can be input into an encoder in the foreign object classification model, and the encoder encodes the key local images to obtain a coding sequence of training images. It should be noted that the coding sequence of training images includes the coding vector of each key local image.

[0071] S104, performing self-attention perception on the coding sequence to obtain texture feature information of the key local images, and performing cross-attention perception on the texture feature information of the key local images to obtain correlation information between the key local images.

[0072] In some embodiments, a foreign object classification model performs self-attention on the encoded sequence to obtain a local feature vector of the key local image, wherein the local feature vector may include key feature information of the key local image and can be used to represent texture feature information of the key local image. Optionally, a predetermined convolutional layer performs self-attention on the encoded sequence to obtain texture feature information of the key local image.

[0073] In some embodiments, the foreign object classification model can perceive the texture feature information of different key local images based on a cross-attention perception mechanism to capture the interrelationships and dependencies between different local regions, thereby obtaining correlation information between key local images. This correlation information can reflect the interrelationships and dependencies between key local images, as well as the degree of intimacy between key local images. For example, by calculating the attention weight of one key local image on another key local image, the importance of the key local image in the overall understanding of the image can be determined.

[0074] S105, determining the foreign body classification prediction result of the training image based on the texture feature information and the correlation information, and adjusting the model parameters of the foreign body classification model based on the label information of the training image and the foreign body classification prediction result until the training is completed to obtain the target foreign body classification model.

[0075] In some embodiments, after obtaining the texture feature information of the key local images and the correlation information between the key local images, the texture feature information and the correlation information between the key local images can be fused to obtain the image features of the training image. Alternatively, a multi-layer linear perception mechanism can be used to fuse the texture feature information of the key local images and the correlation information between the key local images to generate the image features of the training image.

[0076] In some embodiments, after acquiring the image features, the image features are decided by an abnormal classification model, and the foreign body classification prediction result corresponding to the training image is output.

[0077] In some embodiments, after obtaining the foreign object classification prediction result corresponding to the training image, the loss value can be calculated based on the label information of the training image and the foreign object classification prediction result to obtain the loss value of the foreign object classification model. Furthermore, the model parameters of the foreign object classification model are adjusted based on the loss value until the training is completed and the target foreign object classification model is obtained.

[0078] In some embodiments, based on the cross entropy loss function, loss calculation is performed on the label information of the training image and the foreign object classification prediction result to obtain the loss value of the foreign object classification model.

[0079] In some embodiments, model parameters of the foreign object classification model are adjusted based on the loss value and the stochastic gradient descent algorithm.

[0080] In some embodiments, after adjusting the model parameters of the foreign object classification model, the adjusted foreign object classification model can be continued to be trained using the next training image. In response to the convergence of the recalculated loss value and the determination that the training end condition is met, the training is terminated to obtain the target foreign object classification model.

[0081] In some embodiments, in response to the foreign object classification model being trained a set number of times and determining that a training end condition is met, the training is ended to obtain a target foreign object classification model.

[0082] In some embodiments, in response to the training time of the foreign object classification model reaching a set time, it is determined that the training end condition is met, and the training is ended to obtain the target foreign object classification model.

[0083] In the embodiment of the present application, key local features are extracted through the self-attention mechanism, and further global attention is paid to the key local features through the cross-attention mechanism to obtain global features. The key local features and global features can be better utilized in the foreign matter classification model, thereby solving the problems of difficulty in focusing key information for foreign matter classification on the coal conveyor belt and difficulty in perceiving correlation information between local information. This is beneficial to improving the recognition accuracy of the foreign matter classification task of the coal conveyor belt, and then, based on the foreign matter recognition results, an adaptive foreign matter handling process is performed on the coal conveyor belt to reduce the impact of foreign matter on coal transmission and to increase the service life of the coal conveyor belt.

[0084] Based on the above embodiments, Figure 2 This is a schematic diagram of the structure of a foreign body classification model for a coal conveyor belt provided in an embodiment of the present application. Figure 2 As shown, the foreign object classification model includes a key point detector 201, a key area mining module 202, an encoder 203, a self-attention perception module 204, a cross-attention perception module 205 and a decision module 206.

[0085] In some embodiments, the foreign object classification model library may be a vision transformer network (ViT), which may be a deep neural network.

[0086] Based on the above embodiment, step S102 of "performing key point detection on the training image to obtain a key point set of the training image, and generating a key local image set of the training image based on the key point set of the training image" is explained as follows:

[0087] S301: Input a training image into a key point detector, and perform key point detection on the training image to obtain an initial key point set of the training image.

[0088] That is to say, key point detection is performed on the training image through the key point detector in the foreign object classification model.

[0089] In some embodiments, the key point detector can adopt the Features From Accelerated Segment Test (FAST) algorithm, the Scale-Invariant Feature Transform (SIFT) algorithm, the Speeded-Up Robust Features (SURF) algorithm, the Oriented FAST and Rotated BRIEF (ORB) algorithm, etc.

[0090] In some embodiments, the key point detector may use a FAST algorithm to quickly detect corner points of the training image to obtain an initial key point set of the training image.

[0091] S302 , obtaining a Haar response value of each key point in the initial key point set, and filtering the initial key point set according to the Haar response value of the key point and a set threshold to obtain a key point set of the training image.

[0092] In some embodiments, after obtaining an initial set of key points, a Harris response value may be calculated for each key point in the set to obtain a Harris response value for each key point. Furthermore, the Harris response value of each key point is compared with a set threshold. If the Harris response value of each key point is greater than or equal to the set threshold, the key point may be retained; if the Harris response value of each key point is less than the set threshold, the key point may be deleted.

[0093] For example, after the training image sequence X is acquired, the training image x in the training image sequence X can be i Send it to FAST’s key point detector to generate a training image x i The corresponding initial key point set is further filtered by calculating the Haar response value of each key point and combining the set threshold to obtain the training image x i The final set of keypoints.

[0094] In this application, the initial key point set is filtered in the above manner to obtain the key point set of the training image, thereby controlling the quantity and quality of the key points. S303: The key point set of the training image is input into the key area mining module to generate a key local image of each key point in the key point set.

[0095] In some embodiments, after obtaining the keypoint set of the training image, the key region mining module in the foreign object classification model can be used to generate a key local image for each keypoint in the keypoint set. In other words, the keypoint set of the training image is input into the key region mining module to generate a key local image for each keypoint in the keypoint set.

[0096] In some embodiments, the key region mining module may construct an image region of the key point with the key point as the center point as the key local image of the key point.

[0097] S303, for key point p k , get the key point p k The key local image and key point p j The intersection-over-union (IOU) between key local images.

[0098] Wherein, 1≤k, j≤K, K represents the number of key points in the key point set.

[0099] In some embodiments, for any key point p in the key point set k , get the key point p k The key local image and any other key point p in the key point set j The Intersection over Union (IoU) between the key local images.

[0100] S304, according to the key point p k The key local image and key point p j The IOU between the key local images and the set IOU threshold are used to filter the key local images to obtain the key local image set of the training image.

[0101] It should be noted that the value range of IoU is (0, 1). The larger the value of IoU, the closer the key point p is. k The key local image and key point p j The closer the key local images are, the larger the overlapping area is. The smaller the IOU value is, the more important the key point p is. k The key local image and key point p j The greater the difference between the key local images, the smaller the overlapping area.

[0102] In some embodiments, some key points in the initial key point set may be clustered together, and it is necessary to filter the key local images corresponding to the key points to reduce the local redundancy of the key local images. Optionally, an IOU threshold for filtering is set, and the key local images are filtered by the IOU threshold. k The key local image and key point pj The IOU between the key local images is compared with the set IOU threshold. In response to the IOU being less than the set IOU threshold, the key point p is k The key local image is retained, and in response to the IOU being greater than or equal to the set IOU threshold, the key point p is k The key local images are deleted.

[0103] Furthermore, based on the key local images of the retained key points, a set of key local images of the training image is generated. In this application, non-maximum suppression technology is used to mine key local images with key information to facilitate improving the accuracy of subsequent classification and recognition.

[0104] Optionally, the set IOU threshold may be 0.5.

[0105] In some embodiments, the key local image may be filtered using the following formula (1):

[0106]

[0107] Where, L(p j ) represents the key point p j The N'×N' image neighborhood of the center point, that is, the key point p j The key local image of L(p k ) represents the key point p k The N'×N' image neighborhood of the center point, that is, the key point p k The key local image.

[0108] IOU(a,b)=(a∩b) / (a∪b) represents the IOU between the key local image a and the key local b, where a can represent the key point p j The key local image, b can represent the key point p k The key local image.

[0109] μ is an IOU hyperparameter used to select key local images, which is the IOU threshold set above.

[0110] L k represents the key local image corresponding to the kth key point, and L represents the key local image set of the training image.

[0111] In the example of this application, filtering of key points can remove interfering key points or abnormal key points, improve the quality of key points, and reduce the number of key points. Furthermore, the non-maximum suppression technology filters the key local images of key points, which can eliminate redundant local areas and then mine key local images with key information for retention. The above two filtering methods are helpful in saving the computational cost of subsequent classification and recognition, and can also improve the accuracy and timeliness of subsequent classification and recognition.

[0112] Based on the above embodiment, step S103 of "performing key point detection on the training image to obtain a key point set of the training image, and generating a key local image set of the training image based on the key point set of the training image" is explained as follows:

[0113] S401: Input the key local image into an encoder in the foreign object classification model, and the encoder performs convolution processing on the key local image to generate a first coding vector of the key local image.

[0114] S402, determining a position coding vector of a key local image.

[0115] S403 , performing an addition operation on the first coding vector and the position coding vector to obtain a second coding vector of the key local image.

[0116] S404 : Determine a coding sequence of the training image based on the second coding vector of the key local image.

[0117] In some embodiments, each key local image in the key local image set is input into the encoder in the foreign object classification model respectively, and the encoder uses a 1×1×C convolution layer to increase the number of channels in the image. Further, an N'×N'×C convolution layer is used to generate a first encoding vector for each key local image.

[0118] In some embodiments, the process of determining the first encoding vector can be represented by the following formula (2):

[0119] E=[e0,e1,...,e |L| ](2)

[0120] where e0∈R C is a global embedding vector; optionally, the global image is resized to an N'×N' image through a convolutional layer, and further based on the N'×N' image and generates a global embedding vector

[0121] e i ∈R C is the key local image L(p i )∈L corresponding to the first encoding vector.

[0122] Furthermore, an initial position encoding vector can be assigned to each encoding vector in E. The position encoding vector is a learnable position embedding vector, which can be marked as E'=[e0′,e1,′...,e′ |L| ].

[0123] Furthermore, each position coding vector in the position coding vector E' is introduced into the corresponding position in E by addition operation to obtain the second coding vector corresponding to each key local image: i +e i '. That is, the first coding vector and the position coding vector are added to obtain the second coding vector of the key local image,

[0124] Furthermore, based on the second encoding vector of the key local image, an encoding sequence of the training image is obtained.

[0125] Optionally, the encoding sequence of the training image is shown in the following formula (3):

[0126] E+E'=[e0+e0′,e1+e1,′...,e |L| +e′ |L| ] (3)

[0127] In the embodiments of the present application, encoding the key local images through an encoder can reduce the processing of the entire image, thereby reducing the amount of computation and improving the processing speed. Furthermore, the key local images usually contain the most important information in the image. By encoding the key local images, attention can be focused on the key local images, and more accurate and meaningful features can be extracted to better capture the characteristics of the target, reduce the interference of background noise, and provide more accurate input for subsequent tasks, thereby improving the accuracy of foreign object classification.

[0128] Based on the above embodiment, step S104 of "performing self-attention perception on the coding sequence to obtain texture feature information of the key local images, and performing cross-attention perception on the texture feature information of the key local images to obtain correlation information between the key local images" is explained as follows:

[0129] S501: Determine the query code q, key code k, and value code v of the second code vector in the code sequence.

[0130] S502, through the self-attention perception module, multi-head attention operation is performed on the query code q, key code k and value code v to obtain the third coding vector of the key local image.

[0131] S503 : Using a residual strategy to mix the second coding vector and the third coding vector of the key local image to obtain a fourth coding vector of the key local image, wherein the fourth coding vector is used to represent texture feature information of the key local image.

[0132] S504, through the cross attention perception module, a multi-head cross attention operation is performed on the texture feature information, the key code k and the value code v to obtain the fifth coding vector of the key local image.

[0133] S505 , using a residual strategy to mix the second coding vector and the fifth coding vector of the key local image to obtain a sixth coding vector of the key local image, wherein the sixth coding vector is used to represent correlation information corresponding to the key local image.

[0134] In some embodiments, after the encoder outputs the encoding sequence of the training image, the encoding sequence can be input into the self-attention module, and the self-attention module can perceive the texture feature information contained in each key local image.

[0135] Furthermore, the texture feature information perceived by the self-attention module is input into the cross-attention module, which perceives the correlation information between different key local images. It can be understood that the correlation information between different key local images can reflect the mutual relationship and dependency between the key local images.

[0136] In some embodiments, the above Figure 4 The embodiment shown generates a coding sequence E+E' of the position coding vector carrying the key local image and the first coding vector. Further, each coding vector e in E+E' i +e i ′ is mapped to the query code q, key code k and value code v to determine the encoding vector e i +e i ′ corresponds to the query code q, key code k and value code v.

[0137] The encoding vector e i +e i The process of mapping ′ to query code q, key code k and value code v can be represented as follows:

[0138] q=k=v←e i +e i '

[0139] Furthermore, the query code q, key code k and value code v are fed into the multi-head attention, and for each key local image L(p i) is encoded to obtain the third encoding vector of the key local image, which can be expressed as:

[0140] MultiHead(q,k,v)=Concat(h1,...,h n )·W o ,

[0141]

[0142] Among them, MultiHead() represents the multi-head attention operation, n is the number of attention heads; Concat() represents the vector concatenation operation. i represents the output vector of the i-th attention head. and Represents three non-shared learnable weights. Softmax is an activation function used to normalize the feature distribution. k = C / n represents the dimension of the output vector of the i-th attention head. o The vector h used to output each attention head i Perform inner product operation.

[0143] In some embodiments, in order not to reduce the feature information in each key local image, a residual strategy is used to mix the second coding vector and the third coding vector of the key local image to obtain a fourth coding vector of the key local image, where the fourth coding vector is used to represent the texture feature information of the key local image.

[0144] In some embodiments, the second encoding vector e of the key local image is encoded by adopting a residual strategy. i +e i The process of mixing with the third encoding vector MultiHead(q,k,v) can be expressed as:

[0145]

[0146] in, is the fourth encoded vector of the key local image output by the self-attention module. L2Norm() represents the L2 normalization operation. MLP() is the Multilayer Perceptron (MLP).

[0147] It can be understood that the fourth encoding vector of the key local image is used to represent texture feature information of the key local image.

[0148] In some embodiments, after the texture feature information (i.e., the fourth coding vector) of the key local image is perceived by the self-attention module, the texture feature information of the key local image, the corresponding key code k, and the value code v can be input into the cross-attention module, and the cross-attention module performs feature extraction or perception on the above input to obtain a sixth coding vector of the key local image, wherein the sixth coding vector is used to represent the correlation information corresponding to the key local image. It is understood that the correlation information can be used to reflect the mutual relationship and dependency between the current key local image and one or more other key local images.

[0149] Optionally, the process of the cross-attention module perceiving the correlation information corresponding to the key local images can be expressed as:

[0150]

[0151]

[0152] Among them, the cross attention module and Respectively k and v are learnable weight parameters. Includes more aggregation cues at both global and key local levels.

[0153] Furthermore, in the crisscross attention module, the encoder output The information in can be used to enhance the decoder representation and obtain the decoupled encoding

[0154] Optionally, It can be expressed as:

[0155]

[0156] In an embodiment of the present application, during the training process, key local images are mined from training images based on prior information, and the texture feature information of the key local images can be quickly and accurately perceived through the self-attention mechanism. Furthermore, the correlation information between key local images can be accurately perceived through the cross-attention mechanism, and predictions are made based on the texture feature information and the correlation information, thereby improving the accuracy of classification predictions and reducing the training difficulty of the model.

[0157] Based on the above embodiment, step S105 "predicting the foreign object classification prediction result based on the texture feature information and the correlation information, and adjusting the model parameters of the foreign object classification model based on the label information of the training image and the foreign object classification prediction result until the training is completed to obtain the target foreign object classification model" can be explained as follows:

[0158] In some embodiments, after obtaining the correlation information perceived by the cross-attention module, the foreign object classification prediction result can be predicted based on the texture feature information and correlation information of the key local image through the decision module in the foreign object classification model.

[0159] In some embodiments, by introducing self-attention and cross-attention mechanisms into the foreign object classification model, it is possible to perceive the texture feature information of the key local image and the correlation information between the key local images respectively, and splice the texture feature information and the correlation information to obtain the image features of the key local image. Optionally, the dimension of the image features of the key local image can be a 768-dimensional image feature.

[0160] Furthermore, the image features of the extracted key local image are sent to the decision module, and the decision module outputs the foreign body classification of the key local image according to the image features.

[0161] In some embodiments, the decision module may include a self-attention unit and a multi-layer linear perceptron, which is used to predict the category to which the foreign object in the key local image belongs.

[0162] Alternatively, the decision-making process of the decision module can be expressed as:

[0163]

[0164] where z i ∈R |class| represents the probability of the predicted category, and |class| represents the number of foreign object categories.

[0165] Furthermore, through argmaxz i The foreign body classification prediction result that can be determined is to select the category with the largest probability in the probability distribution as the category to which the foreign body in the key local image belongs.

[0166] In an embodiment of the present application, after obtaining the category to which the foreign matter in the key local image belongs, the loss value of the foreign matter classification model based on the key local image is calculated based on the cross entropy loss function, combined with the label information of the training image and the foreign matter classification prediction result.

[0167] Alternatively, the process of calculating the loss value using the cross entropy loss function can be expressed as:

[0168]

[0169] Among them, z i represents the predicted category of the foreign object of the key local image i in the training image; i represents the true value label of the key local image i in the training image, B is the training batch, and |B| represents the training batch size.

[0170] Furthermore, the foreign body classification model based on the key local image is optimized according to the loss value. Optionally, the model parameters of the foreign body classification model are adjusted or optimized according to the stochastic gradient descent algorithm in combination with the loss value.

[0171] Figure 6 The present application provides a flow chart of a method for classifying foreign matter on a coal conveyor belt. Figure 6 As shown, the foreign matter classification method of the coal conveyor belt may include but is not limited to the following steps:

[0172] S601, acquiring a coal flow image of a coal conveyor belt.

[0173] S602: Input the coal flow image into the target foreign matter classification model to perform foreign matter classification, and obtain a foreign matter classification result of the coal image.

[0174] In some embodiments, the target foreign body classification model is a model trained using the training method described in the above embodiments.

[0175] In some embodiments, the coal flow image is input into the target foreign body classification model, and the key point detection of the coal flow image is performed by a key point detector to obtain an initial key point set of the coal flow image. Furthermore, the Haar response value of the key point in the initial key point set is determined, and the initial key point set is filtered according to the Haar response value and the set threshold to obtain the final key point set of the coal flow image.

[0176] Furthermore, a key local image mining module is used to generate a key local image of each key point in the key point set, and the key local image is filtered using a non-maximum suppression technique to obtain a key local image set of the coal image.

[0177] Furthermore, the key local image is encoded by the encoder to obtain a coding sequence of the coal image, and the coding sequence is input into the self-attention perception module. The self-attention perception module performs a multi-head attention operation on the coding vector corresponding to the key local image in the coding sequence to obtain the texture feature information of the key local image.

[0178] Furthermore, the texture feature information of the key local image is input into the cross-attention perception module, and the cross-attention perception module performs a multi-head cross-attention operation on the texture feature information of the key local image, as well as the corresponding key code k and value code v to obtain the correlation information corresponding to the key local image.

[0179] After obtaining the texture feature information and correlation information corresponding to the key local image, the texture feature information and correlation information can be input into the decision module. The decision module predicts the foreign body category of the key local image based on the texture feature information and correlation information, obtains the foreign body classification prediction result, and then determines the foreign body category result of the coal flow image.

[0180] In the embodiment of the present application, through the pre-trained target foreign matter classification model, the real-time coal flow image can be accurately classified as a foreign matter, which solves the problem that the key information for foreign matter classification on the coal conveyor belt is difficult to focus on and the correlation information between local information is difficult to perceive. It is beneficial to improve the recognition accuracy of the foreign matter classification task of the coal conveyor belt, and then according to the foreign matter recognition results, the coal conveyor belt is adapted to the foreign matter processing process, which reduces the impact of foreign matter on coal transmission and can increase the service life of the coal conveyor belt.

[0181] Based on the foreign body classification model for coal conveyor belts provided in the embodiments of this application and other foreign body classification models for coal conveyor belts, foreign body classification processing can be performed on different coal flow image datasets to obtain recognition results for each foreign body classification model. The different coal flow image datasets can include open source coal flow image datasets (for example, the CUMT-BelT dataset) and classic image classification datasets (for example, the Cifar-10 dataset and the Cifar-100 dataset).

[0182] Other foreign body classification models for coal conveyor belts may include but are not limited to: extremely efficient convolutional neural networks for mobile devices (for example, Shufflenet), improved lightweight deep neural networks (for example, MobileNetV2), residual networks (for example, ResNet110), visual transformers (for example, VisionTransformer), and window-based visual transformers (for example, Swin-Transformer).

[0183] Using different coal conveyor belt foreign body classification models, the CUMT-BelT dataset was processed for foreign body classification. Model evaluation indicators corresponding to each foreign body classification model were obtained, as shown in Table 1 below. Model evaluation indicators may include, but are not limited to, the model's classification accuracy (Class Accuary), the model's parameters (Parameters), and the model's computational complexity (Floating Point Operations per Second, or FLOPs). Class Accuary (%) reflects the accuracy of foreign body classification; a larger value indicates higher algorithm performance. Parameters and FLOPs reflect the algorithm's operational efficiency; smaller values ​​indicate higher algorithm efficiency.

[0184] Table 1

[0185]

[0186] Using different coal conveyor belt foreign body classification models, the Cifar-10 dataset is processed for foreign body classification. The model evaluation indicators corresponding to each foreign body classification model can be obtained, as shown in Table 2 below:

[0187] Table 2

[0188]

[0189] Using different coal conveyor belt foreign body classification models, the Cifar-100 dataset is processed for foreign body classification. The model evaluation indicators corresponding to each foreign body classification model can be obtained, as shown in Table 3 below:

[0190]

[0191] As can be seen from Tables 1 to 3, the Class Accuary (%) obtained by the model provided by the embodiment of the present application on the CUMT-BelT dataset, the Cifar-10 dataset, and the Cifar-100 dataset can reach the highest level compared to other models. Therefore, it can be determined that compared to other foreign body classification models, the foreign body classification model provided by the embodiment of the present application, unlike other models that mainly focus on global information and ignore key component-level information, uses key parts with rich appearance clues to improve classification results. First, key point detection is performed on the key points in the image, and then key component-level information is extracted through a key local mining strategy. Then, by introducing an attention module with a cross-attention block, the key part-level information is fully utilized for feature aggregation to perform foreign body classification, which greatly improves the accuracy of foreign body classification.

[0192] like Figure 7 As shown, based on ViT and the foreign body classification model provided by the present application, foreign body classification is performed on the same coal image dataset, and the classification results are as follows: Figure 7 As shown, Figure (a) is the classification result of foreign body classification processing of the coal image dataset based on ViT, and Figure (b) is the classification result of foreign body classification processing of the coal image dataset based on the foreign body classification model provided by the implementation of this application.

[0193] To implement the above-mentioned foreign object classification model training, an embodiment of the present application provides a training device for a foreign object classification model for a coal conveyor belt. The training device for a foreign object classification model for a coal conveyor belt includes: a sample acquisition unit, a local image acquisition unit, an encoding unit, a first perception unit, a second perception unit, a prediction unit, and an adjustment unit.

[0194] A sample acquisition unit, configured to acquire a training image and label information of the training image;

[0195] a local image acquisition unit, configured to input the training image into a foreign body classification model, perform key point detection on the training image to obtain a key point set of the training image, and generate a key local image set of the training image based on the key point set of the training image;

[0196] an encoding unit, configured to encode the key local images in the key local image set and determine an encoding sequence of the training images;

[0197] A first perception unit is configured to perform self-attention perception on the coding sequence to obtain texture feature information of the key local image;

[0198] a second perception unit, configured to perform cross-attention perception on the texture feature information of the key local images to obtain correlation information between the key local images;

[0199] A prediction unit, configured to determine a foreign body classification prediction result of the training image based on the texture feature information and the correlation information;

[0200] An adjustment unit is used to adjust the model parameters of the foreign body classification model according to the label information of the training image and the foreign body classification prediction result until the training is completed to obtain the target foreign body classification model.

[0201] In some embodiments, the local image acquisition unit is further configured to:

[0202] Performing key point detection on the training image using a key point detector in the foreign object classification model to obtain an initial key point set of the training image;

[0203] Obtaining a Haar response value for each key point in the initial key point set;

[0204] The initial key point set is filtered according to the Haar response value of the key point and a set threshold to obtain the key point set of the training image.

[0205] In some embodiments, the local image acquisition unit is further configured to:

[0206] Generate a key local image of each key point in the key point set through a key area mining module in the foreign object classification model;

[0207] For key point p k , get the key point p k The key local image and key point p jThe intersection-over-union (IOU) ratio between the key local images is, where 1≤k, j≤K, K represents the number of key points in the key point set;

[0208] In response to the IOU being less than the set IOU threshold, the key point p k The key local images are retained;

[0209] In response to the IOU being greater than or equal to the IOU threshold, k The key local images are deleted.

[0210] In some embodiments, the encoding unit is further configured to:

[0211] Inputting a key local image in the key local image set into an encoder in the foreign object classification model, and having the encoder perform convolution processing on the key local image to generate a first encoding vector for the key local image;

[0212] Determining a position encoding vector of the key local image;

[0213] performing an addition operation on the first coding vector and the position coding vector to obtain a second coding vector of the key local image;

[0214] Based on the second encoding vector of the key local image, a coding sequence corresponding to the training image is determined.

[0215] In some embodiments, the first sensing unit is further configured to:

[0216] Determining a query code q, a key code k, and a value code v of a second code vector in the code sequence;

[0217] Performing a multi-head attention operation on the query code q, the key code k, and the value code v through the self-attention perception module in the foreign object classification model to obtain a third encoding vector of the key local image;

[0218] The second coding vector and the third coding vector are mixed by adopting a residual strategy to obtain a fourth coding vector of the key local image, where the fourth coding vector is used to represent the texture feature information.

[0219] In some embodiments, the second sensing unit is further configured to:

[0220] Performing a multi-head cross-attention operation on the texture feature information, the key code k, and the value code v through the cross-attention perception module in the foreign object classification model to obtain a fifth coding vector corresponding to the key local image;

[0221] The second coding vector and the fifth coding vector are mixed with each other using a residual strategy to obtain a sixth coding vector of the key local image, where the sixth coding vector is used to represent the correlation information corresponding to the key local image.

[0222] In some embodiments, the prediction unit is further configured to:

[0223] fusing the texture feature information and the correlation information to obtain image features;

[0224] Predicting a foreign body classification prediction result of the training image based on the image features;

[0225] In some embodiments, the adjustment unit is further configured to:

[0226] Based on the cross entropy loss function, performing loss calculation on the label information and the foreign object classification prediction result to obtain a loss value of the foreign object classification model;

[0227] Based on the loss value and the stochastic gradient descent algorithm, the model parameters of the foreign object classification model are adjusted.

[0228] In the embodiment of the present application, key local features are extracted through the self-attention mechanism, and further global attention is paid to the key local features through the cross-attention mechanism to obtain global features. The key local features and global features can be better utilized in the foreign matter classification model, thereby solving the problems of difficulty in focusing key information for foreign matter classification on the coal conveyor belt and difficulty in perceiving correlation information between local information. This is beneficial to improving the recognition accuracy of the foreign matter classification task of the coal conveyor belt, and then, based on the foreign matter recognition results, an adaptive foreign matter handling process is performed on the coal conveyor belt to reduce the impact of foreign matter on coal transmission and to increase the service life of the coal conveyor belt.

[0229] In order to achieve the above-mentioned foreign body classification and identification, the embodiment of the present application provides a foreign body classification device for a coal conveyor belt. The foreign body classification device for a coal conveyor belt includes: an image acquisition unit and an input unit;

[0230] An image acquisition unit, used for acquiring coal flow images of the coal conveyor belt;

[0231] An input unit, configured to input the coal flow image into a target foreign matter classification model to perform foreign matter classification, and obtain a foreign matter classification result of the coal image;

[0232] The target foreign body classification model is a model trained using the training method provided in the above embodiment.

[0233] In the embodiment of the present application, through the pre-trained target foreign matter classification model, the real-time coal flow image can be accurately classified as a foreign matter, which solves the problem that the key information for foreign matter classification on the coal conveyor belt is difficult to focus on and the correlation information between local information is difficult to perceive. It is beneficial to improve the recognition accuracy of the foreign matter classification task of the coal conveyor belt, and then according to the foreign matter recognition results, the coal conveyor belt is adapted to the foreign matter processing process, which reduces the impact of foreign matter on coal transmission and can increase the service life of the coal conveyor belt.

[0234] Since the device provided in the embodiment of the present application corresponds to the methods provided in the above-mentioned embodiments, the implementation of the method is also applicable to the device provided in this embodiment and will not be described in detail in this embodiment.

[0235] In the embodiments provided above, the methods and devices provided in the embodiments of the present application are introduced. In order to implement the various functions of the methods provided in the embodiments of the present application, the electronic device may include a hardware structure and a software module, and implement the aforementioned functions in the form of a hardware structure, a software module, or a hardware structure plus a software module. One of the aforementioned functions may be executed in the form of a hardware structure, a software module, or a hardware structure plus a software module.

[0236] In order to implement the above embodiments, the present application also proposes an electronic device, comprising: a processor, and a memory communicatively connected to the processor; the memory stores computer-executable instructions; the processor executes the computer-executable instructions stored in the memory to implement the method provided by the above embodiments.

[0237] In order to implement the above embodiments, the present application also proposes a computer-readable storage medium, in which computer-executable instructions are stored. When the computer-executable instructions are executed by a processor, they are used to implement the methods provided by the above embodiments.

[0238] In order to implement the above embodiments, the present application also proposes a computer program product, including a computer program, which implements the methods provided by the above embodiments when executed by a processor.

[0239] The collection, storage, use, processing, transmission, provision and application of user personal information involved in this application are in compliance with relevant laws and regulations and do not violate public order and good morals.

[0240] It is important to note that personal information collected from users should be used for legitimate and reasonable purposes and should not be shared or sold beyond these legitimate uses. Furthermore, such collection / sharing should be conducted only after receiving the user's informed consent, including but not limited to notifying the user to read the user agreement / user notice and sign an agreement / authorization that includes the relevant user information before using the feature. Furthermore, any necessary steps must be taken to safeguard and secure access to such personal information and ensure that others with access to personal information comply with its privacy policy and procedures.

[0241] This application contemplates providing implementation options for users to selectively block the use or access of personal information data. Specifically, this application contemplates providing hardware and / or software to prevent or block access to such personal information data. Risks can be minimized by limiting data collection and deleting data once it is no longer needed. Furthermore, where applicable, such personal information can be de-identified to protect user privacy.

[0242] In the descriptions of the aforementioned embodiments, the reference terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" mean that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic expressions of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, unless they are contradictory.

[0243] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of the technical features being referred to. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of such features. Throughout the description of this application, "plurality" means at least two, for example, two, three, etc., unless otherwise specifically defined.

[0244] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, segment or portion of code comprising one or more executable instructions for implementing the steps of a custom logical function or process, and the scope of the preferred embodiments of the present application includes alternative implementations in which functions may be performed out of the order shown or discussed, including performing functions in a substantially simultaneous manner or in the reverse order depending on the functions involved, which should be understood by those skilled in the art to which the embodiments of the present application belong.

[0245] The logic and / or steps represented in the flowcharts or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing the logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (e.g., a computer-based system, a system including a processor, or other system that can fetch and execute instructions from an instruction execution system, apparatus, or device). For purposes of this specification, a "computer-readable medium" can be any device that can contain, store, communicate, propagate, or transport a program for use by, or in conjunction with, an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of computer-readable media include the following: an electrical connection with one or more wires (electronic devices), a portable computer disk cartridge (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and programmable read-only memory (EPROM or flash memory), fiber optic devices, and a portable compact disc read-only memory (CDROM). Furthermore, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium and then editing, interpreting or processing it in another suitable manner if necessary, and then storing it in a computer memory.

[0246] It should be understood that various parts of the present application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented using software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented using hardware, as in another embodiment, any one of the following technologies known in the art or a combination thereof can be used to implement: a discrete logic circuit having a logic gate circuit for implementing a logic function on a data signal, an application-specific integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0247] Those skilled in the art will understand that all or part of the steps in the method of the above embodiment can be completed by instructing related hardware through a program, and the program can be stored in a computer-readable storage medium. When the program is executed, it includes one or a combination of the steps of the method embodiment.

[0248] In addition, the functional units in the various embodiments of the present application may be integrated into a processing module, or each unit may exist physically separately, or two or more units may be integrated into a module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium.

[0249] The storage medium mentioned above may be a read-only memory, a magnetic disk, or an optical disk, etc. Although the embodiments of the present application have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present application. Persons skilled in the art may make changes, modifications, substitutions, and variations to the above embodiments within the scope of the present application.

Claims

1. A method for training a foreign body classification model for a coal conveyor belt, characterized in that: The method comprises: Obtaining a training image and label information of the training image; Inputting the training image into a foreign body classification model, performing key point detection on the training image to obtain a key point set of the training image, and determining a key local image set of the training image based on the key point set of the training image; Encoding the key local images in the key local image set to obtain a coding sequence of the training images; Performing self-attention perception on the coding sequence to obtain texture feature information of the key local image; Performing cross-attention perception on texture feature information of the key local images to obtain correlation information between the key local images; Determining a foreign body classification prediction result of the training image based on the texture feature information and the correlation information; According to the label information and the foreign body classification prediction result, the model parameters of the foreign body classification model are adjusted until the training is completed to obtain the target foreign body classification model.

2. The method according to claim 1, characterized in that The performing key point detection on the training image to obtain a key point set of the training image includes: Performing key point detection on the training image using a key point detector in the foreign object classification model to obtain an initial key point set of the training image; Obtaining a Haar response value for each key point in the initial key point set; The initial key point set is filtered according to the Haar response value of the key point and a set threshold to obtain the key point set of the training image.

3. The method according to claim 1, characterized in that Generating a key local image of the training image according to the key point set of the training image includes: Generate a key local image of each key point in the key point set through a key area mining module in the foreign object classification model; For key point p k , get the key point p k The key local image and key point p j The intersection-over-union (IOU) ratio between the key local images is, where 1≤k, j≤K, K represents the number of key points in the key point set; In response to the IOU being less than the set IOU threshold, the key point p k The key local images are retained; In response to the IOU being greater than or equal to the IOU threshold, k The key local images are deleted.

4. The method according to claim 1, wherein The step of encoding the key local images in the key local image set and determining the encoding sequence of the training images includes: Inputting a key local image in the key local image set into an encoder in the foreign object classification model, and having the encoder perform convolution processing on the key local image to generate a first encoding vector for the key local image; Determining a position encoding vector of the key local image; performing an addition operation on the first coding vector and the position coding vector to obtain a second coding vector of the key local image; Based on the second encoding vector of the key local image, a coding sequence corresponding to the training image is determined.

5. The method according to claim 4, characterized in that The performing self-attention perception on the coding sequence to obtain texture feature information of the key local image includes: Determining a query code q, a key code k, and a value code v of a second code vector in the code sequence; Performing a multi-head attention operation on the query code q, the key code k, and the value code v through the self-attention perception module in the foreign object classification model to obtain a third encoding vector of the key local image; The second coding vector and the third coding vector are mixed by adopting a residual strategy to obtain a fourth coding vector of the key local image, where the fourth coding vector is used to represent the texture feature information.

6. The method according to claim 5, characterized in that The performing cross-attention perception on the texture feature information of the key local images to obtain correlation information between the key local images includes: Performing a multi-head cross-attention operation on the texture feature information, the key code k, and the value code v through the cross-attention perception module in the foreign object classification model to obtain a fifth coding vector corresponding to the key local image; The second coding vector and the fifth coding vector are mixed with each other using a residual strategy to obtain a sixth coding vector of the key local image, where the sixth coding vector is used to represent the correlation information corresponding to the key local image.

7. The method according to any one of claims 1 to 6, characterized in that The determining, based on the texture feature information and the correlation information, a foreign body classification prediction result of the training image includes: fusing the texture feature information and the correlation information to obtain image features; Predicting a foreign body classification prediction result of the training image based on the image features; The method further comprises adjusting the model parameters of the foreign object classification model according to the label information and the foreign object classification prediction result until the training is completed to obtain the target foreign object classification model, including: Based on the cross entropy loss function, performing loss calculation on the label information and the foreign object classification prediction result to obtain a loss value of the foreign object classification model; Based on the loss value and the stochastic gradient descent algorithm, the model parameters of the foreign object classification model are adjusted.

8. A method for sorting foreign matter on a coal conveyor belt, characterized in that: The method comprises: Acquire coal flow images of coal conveyor belts; Inputting the coal flow image into a target foreign matter classification model to perform foreign matter classification, thereby obtaining a foreign matter classification result of the coal image; The target foreign body classification model is a model trained using the training method described in any one of claims 1 to 7.

9. A training device for a foreign body classification model of a coal conveyor belt, characterized in that: The device comprises: A sample acquisition unit, configured to acquire a training image and label information of the training image; a local image acquisition unit, configured to input the training image into a foreign body classification model, perform key point detection on the training image to obtain a key point set of the training image, and generate a key local image set of the training image based on the key point set of the training image; an encoding unit, configured to encode the key local images in the key local image set and determine an encoding sequence of the training images; A first perception unit is configured to perform self-attention perception on the coding sequence to obtain texture feature information of the key local image; a second perception unit, configured to perform cross-attention perception on the texture feature information of the key local images to obtain correlation information between the key local images; A prediction unit, configured to determine a foreign body classification prediction result of the training image based on the texture feature information and the correlation information; An adjustment unit is used to adjust the model parameters of the foreign body classification model according to the label information of the training image and the foreign body classification prediction result until the training is completed to obtain the target foreign body classification model.

10. A foreign matter sorting device for a coal conveyor belt, characterized in that: The device comprises: An image acquisition unit, used for acquiring coal flow images of the coal conveyor belt; An input unit, configured to input the coal flow image into a target foreign matter classification model to perform foreign matter classification, and obtain a foreign matter classification result of the coal image; The target foreign body classification model is a model trained using the training method described in any one of claims 1 to 7.

Images