Coal piling detection method and device, electronic equipment and storage medium

Abstract

The invention discloses a coal piling detection method and device, electronic equipment and a storage medium. According to the specific scheme, a to-be-processed image including a target object is obtained, prompt word information corresponding to the to-be-processed image is determined, the target object is coal located at the connecting position of a plurality of belt conveyors, and the prompt word information is used for representing the expected coal piling degree; processing the to-be-processed image and the cue word information based on the target coal piling detection model to obtain a target coal piling detection result corresponding to the cue word information; wherein the target coal piling detection model comprises a character coding module and a first model, and the first model is obtained after adjustment based on a trained convolutional neural network model. According to the invention, the efficiency and accuracy of coal piling detection are improved, and the safety of coal transportation is ensured.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to a method, apparatus, electronic device, and storage medium for detecting coal piles. Background Technology

[0002] The belt conveyor head is the connecting part between multiple belt conveyors in a coal mine transportation system, mainly responsible for smoothly transferring coal from one belt conveyor to another. Due to the special structure and operating environment of the belt conveyor head, coal accumulation is highly likely to occur. Once coal accumulation occurs at the belt conveyor head, if the conveyor is not stopped in time, the coal will submerge the head area, causing the motor, gearbox, and other transmission components to become blocked, leading to equipment overload, burnout of the motor and gearbox, and even tearing of the conveyor belt, resulting in serious equipment damage accidents. In severe cases, it may even endanger the lives of underground workers.

[0003] Currently, existing technologies primarily rely on manual monitoring, with inspectors monitoring in real time via surveillance cameras. However, in coal mine transportation scenarios, numerous belt conveyors are used, requiring monitoring of a large number of conveyor heads. Manually monitoring coal stockpiles not only increases the risk of undetected and unaddressed blockages, leading to low efficiency and accuracy in coal stockpiling detection, but also raises labor costs. Summary of the Invention

[0004] This invention provides a coal pile detection method, device, electronic equipment, and storage medium, which improves the efficiency and accuracy of coal pile detection and ensures the safety of coal transportation.

[0005] According to one aspect of the present invention, a method for detecting coal stockpiles is provided, applied in the coal transportation process, the method comprising:

[0006] Acquire an image to be processed that includes the target object, and determine the prompt word information corresponding to the image to be processed. The target object is coal located at the connection point of multiple belt conveyors, and the prompt word information is used to characterize the degree of coal accumulation to be detected.

[0007] Based on the target coal pile detection model, the image to be processed and the prompt word information are processed to obtain the target coal pile detection result corresponding to the prompt word information;

[0008] The target coal pile detection model includes a text encoding module and a first model, which is obtained by adjusting a pre-trained convolutional neural network model.

[0009] According to another aspect of the present invention, a coal pile detection device is provided, the device comprising:

[0010] The image information determination module is used to acquire an image to be processed that includes the target object and to determine the prompt word information corresponding to the image to be processed. The target object is coal located at the connection point of multiple belt conveyors, and the prompt word information is used to characterize the degree of coal accumulation to be detected.

[0011] The detection result determination module is used to process the image to be processed and the prompt word information based on the target coal pile detection model to obtain the target coal pile detection result corresponding to the prompt word information;

[0012] The target coal pile detection model includes a text encoding module and a first model, which is obtained by adjusting a pre-trained convolutional neural network model.

[0013] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising:

[0014] At least one processor; and

[0015] A memory that is communicatively connected to at least one processor; wherein,

[0016] The memory stores a computer program that can be executed by at least one processor, such that the at least one processor is able to perform the coal pile detection method according to any embodiment of the present invention.

[0017] According to another aspect of the present invention, a computer-readable storage medium is provided, the computer-readable storage medium storing computer instructions for causing a processor to execute and implement the coal pile detection method of any embodiment of the present invention.

[0018] According to another aspect of the present invention, a computer program product is provided, comprising a computer program, characterized in that the computer program, when executed by a processor, implements a coal pile detection method as described in any embodiment of the present invention.

[0019] The technical solution of this invention involves acquiring an image to be processed, including the target object, and determining the corresponding prompt word information. A target coal pile detection model processes the image to be processed and the prompt word information to obtain the target coal pile detection result corresponding to the prompt word information. This enables the detection of coal accumulation at the connection points of multiple belt conveyors, providing data support for subsequent early warning. The target coal pile detection model processes the prompt word information and the image to be processed, achieving the processing of image features and semantic features, thereby more clearly defining the severity of coal accumulation on the belt conveyor. This invention solves the problems of low efficiency and accuracy, and high labor costs associated with manual monitoring of coal piles in existing technologies. By processing the prompt word information and the image to be processed using a target coal pile detection model, the efficiency and accuracy of coal pile detection are improved, ensuring the safety of coal transportation.

[0020] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of the present invention, nor is it intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description

[0021] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0022] Figure 1 This is a flowchart of a coal pile detection method provided in an embodiment of the present invention;

[0023] Figure 2 This is a flowchart of a first model determination method provided in an embodiment of the present invention;

[0024] Figure 3 This is a structural example diagram of the preset forward feedback module provided in an embodiment of the present invention;

[0025] Figure 4 This is a schematic diagram of the structure of a coal pile detection device provided in an embodiment of the present invention;

[0026] Figure 5 This is a schematic diagram of the structure of an electronic device that implements the coal pile detection method of this invention. Detailed Implementation

[0027] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0028] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0029] Example 1

[0030] Figure 1 This is a flowchart of a coal pile detection method provided in Embodiment 1 of the present invention. This embodiment is applicable to situations where coal accumulation at the connection points of multiple belt conveyors is detected using a target coal pile detection model. This method can be executed by a coal pile detection device, which can be implemented in hardware and / or software. The coal pile detection device can be configured in electronic devices such as mobile phones, computers, or servers. Figure 1 As shown, the method includes:

[0031] S110. Obtain the image to be processed, which includes the target object, and determine the prompt word information corresponding to the image to be processed. The target object is the coal located at the connection point of multiple belt conveyors, and the prompt word information is used to characterize the degree of coal accumulation to be detected.

[0032] In this embodiment of the invention, to ensure the safety of coal transportation, images of coal located at the connection points of multiple belt conveyors can be acquired to determine the coal accumulation status at the connection points, facilitating timely handling of coal accumulation issues. Belt conveyors are conveying machines used to transport coal. The connection point of multiple belt conveyors, commonly referred to as the conveyor head, is used to transfer coal conveyed from one belt conveyor to another. The target object is the coal located at the connection point (conveyor head) of multiple belt conveyors. The image to be processed can be an image containing the target object acquired by an image acquisition device deployed at a preset position corresponding to the connection point. The prompt information can be used to characterize the expected degree of coal accumulation. The degree of coal accumulation is the degree of coal blockage. Optionally, the degree of coal accumulation can include: no blockage, slight blockage, moderate blockage, and severe blockage. For example, the prompt message could be: "this is {coal blockage level}", where the coal blockage level can be characterized by corresponding fields, such as "mild coal blockage", "moderate coal blockage", "severe coal blockage", and "normal". For instance, if the goal is to detect the target coal blockage in the image under the specified coal blockage level, the prompt message could be set to: "this is {severe coal blockage}".

[0033] Specifically, when multiple belt conveyors are detected to be in operation, image acquisition devices deployed at preset locations acquire images of coal at the connection points of the conveyors. Based on the desired coal accumulation level, prompts corresponding to the images are set to enhance semantic information and improve the model's detection accuracy.

[0034] S120. Based on the target coal pile detection model, process the image to be processed and the prompt word information to obtain the target coal pile detection result corresponding to the prompt word information.

[0035] The target coal blockage detection model is used to determine the target coal blockage detection results at the connection point. The target coal blockage detection results can be used as an evaluation attribute characterizing the degree of coal blockage at the connection point. Optionally, the evaluation attribute can be a probability value. For example, if the prompt message is "thisis{severe coalblockage}", then by processing the prompt message and the image to be processed according to the target coal blockage detection model, the probability value corresponding to severe blockage at the connection point can be determined.

[0036] The target coal pile detection model includes a text encoding module and a first model, which is obtained by adjusting a pre-trained convolutional neural network model. The text encoding module is used to encode prompt word information to extract the semantic information corresponding to the prompt words. The first model is used to process the text-encoded features output by the text encoding module and the image to be processed to obtain the target coal pile detection result corresponding to the prompt word information. Optionally, the pre-trained convolutional neural network model can be a network model trained by adjusting a deep convolutional neural network ResNet18.

[0037] Specifically, the text encoding module in the target coal pile detection model processes the prompt word information to obtain the text encoding features corresponding to the prompt word information. The first model of the target coal pile detection model then processes the text encoding features and the image to be processed to obtain the target coal pile detection result. This result is used to determine whether the coal at the connection point needs to be processed.

[0038] In this embodiment of the invention, the first model includes: a target convolutional neural network module, a feature alignment module, and a target convolutional layer.

[0039] The target convolutional neural network module can be used to perform convolution processing on the image to be processed, obtaining the corresponding image features. Optionally, the target convolutional neural network module can be trained by adjusting a deep convolutional neural network ResNet18. The feature alignment module is used to perform feature alignment processing on the text encoding features output by the text encoding module and the corresponding image features of the target convolutional neural network module. The target convolutional layer can be used to perform convolution processing on the aligned features output by the feature alignment module to obtain the target coal pile detection result.

[0040] The specific method for processing the image and prompt word information using the target coal pile detection model can be as follows: Encode the prompt word information using the text encoding module in the target coal pile detection model to determine the text encoding features corresponding to the prompt word information; perform convolution processing on the image to be processed using the target convolutional neural network module to determine image features; perform feature alignment processing on the text encoding features and image features using the feature alignment module to obtain aligned features; and process the aligned features based on the target convolutional layer to obtain the target coal pile detection result.

[0041] The text encoding feature can be the encoding feature of the text corresponding to the prompt word information. Optionally, the text encoding module can be a pre-trained MobileCLIP text encoder (textEncoder). The prompt word information (text) is encoded by the text encoding module (textEncoder) to obtain the text encoding feature (featureText = textEncoder(text)). Optionally, the dimension of the text encoding feature output by the text encoding module is C*D, where C is the length of the prompt word information. Optionally, the length of the prompt word information can be determined according to the category of the prompt word information. That is, the length of the prompt word information is consistent with the number of categories of the prompt word information. In this embodiment of the invention, the categories of the prompt word information include four types: no congestion, slight congestion, moderate congestion, and severe congestion. Therefore, C is 4, and D is the embedding dimension. For example, the dimension of the text encoding feature can be 4*512. The image feature can be the image feature after convolution processing of the image to be processed. The alignment feature can be the feature after aligning the text encoding feature and the image feature.

[0042] Specifically, the text encoding module in the target coal pile detection model encodes the prompt word information to obtain text encoding features corresponding to the prompt word information. The target convolutional neural network module in the first model performs convolution processing on the image to be processed to obtain image features. The feature alignment module in the first model performs feature alignment processing on the image features and text encoding features to obtain aligned features. The target convolutional layer in the first model processes the aligned features to obtain the target coal pile detection result.

[0043] Optionally, the method further includes: if the target coal pile detection result meets the preset coal pile detection result, and the first coal pile detection result corresponding to the coal images collected within a preset time after the image acquisition time corresponding to the image to be processed all meet the preset coal pile detection result, then an early warning processing is performed based on the target coal pile detection result and the first coal pile detection result.

[0044] The preset coal pile detection result can be a pre-set standard evaluation attribute corresponding to the prompt message. For example, taking the evaluation attribute as a probability value, if the prompt message is "this is {severe coal blockage}", then the preset coal pile detection result can be the standard probability value corresponding to severe blockage. For example, the standard probability value can be 70%. That is, if the probability value of severe blockage determined by the target coal pile detection result is 72%, which exceeds the preset standard probability value of 70% for severe blockage, then the target coal pile detection result is determined to meet the preset coal pile detection result.

[0045] The preset duration can be a pre-set time period following the image acquisition time corresponding to the image to be processed. The coal image can be at least one image containing the target object acquired within the preset duration. The first coal pile detection result can be the result obtained by performing coal pile detection processing on the coal image using a target coal pile detection model. It should be noted that the number of first coal pile detection results is consistent with the number of coal images. The early warning processing method can be at least one of telephone, SMS, email, and / or audible and visual warnings.

[0046] Specifically, if the evaluation attribute of the coal pile degree corresponding to the target coal pile detection result exceeds the standard evaluation attribute of the same coal pile degree corresponding to the preset coal pile detection result, and the evaluation attribute of the coal pile degree corresponding to the first coal pile detection result of at least one coal image acquired within a preset time after the image acquisition time corresponding to the image to be processed all exceed the standard evaluation attribute of the same coal pile degree corresponding to the preset coal pile detection result, it indicates that the coal pile at the connection point of multiple belt conveyors is serious. An early warning message can be generated based on the target coal pile detection result and the first coal pile detection result, and early warning processing can be carried out based on the early warning message to solve the problem of equipment damage and personnel safety issues caused by failure to promptly address the coal pile situation, thus ensuring the safety of coal transportation. It should be noted that the early warning message can be an early warning message corresponding to the coal pile degree corresponding to the warning word information.

[0047] For example, taking the evaluation attribute as a probability value, we can set the level corresponding to each degree of coal accumulation and the theoretical probability value (preset coal accumulation detection result) for each level. For example, mild accumulation is level 1, moderate accumulation is level 2, severe accumulation is level 3, and no accumulation is level 0. For instance, we can set the prompt message to correspond to moderate accumulation, which is level 2, and the theoretical probability value threshold for level 2 is 80%. Accordingly, if the detected target coal accumulation result is moderate accumulation (actual probability value 83% > theoretical probability value threshold 80%), and the duration for which the actual probability value is greater than the theoretical probability value exceeds a preset duration, then an early warning will be issued. For instance, the early warning message could be that moderate accumulation of coal has occurred at the connection point of multiple belt conveyors, and the level of coal accumulation is level 2.

[0048] The technical solution of this embodiment acquires an image to be processed, including the target object, and determines the corresponding prompt word information. A target coal pile detection model processes the image to be processed and the prompt word information to obtain the target coal pile detection result corresponding to the prompt word information. This enables the detection of coal accumulation at the connection points of multiple belt conveyors, providing data support for subsequent early warning. The target coal pile detection model processes the prompt word information and the image to be processed, achieving the processing of image features and semantic features, improving the model's generalization ability, and thus more clearly defining the severity of coal accumulation on the belt conveyor. This invention solves the problems of low efficiency and accuracy, and high labor costs caused by manual monitoring of coal piles in existing technologies. By processing the prompt word information and the image to be processed using a target coal pile detection model, the efficiency and accuracy of coal pile detection are improved, ensuring the safety of coal transportation.

[0049] Example 2

[0050] Figure 2 This is a flowchart of a first model determination method provided in Embodiment 2 of the present invention. This embodiment, based on the above embodiments, allows for the prior acquisition of a first model before processing the image to be processed and the prompt word information using the text encoding module of the target coal pile detection model and the first model. Based on the text encoding module and the first model, a target coal pile detection model is then obtained. Specific implementation details can be found in the technical solution of this embodiment. Technical terms that are the same as or corresponding to those in the above embodiments will not be repeated here. Figure 2 As shown, the method includes:

[0051] S210. Obtain the preset convolutional neural network module, and remove the last average pooling layer and the last fully connected layer of the preset convolutional neural network module to obtain the convolutional neural network module to be trained.

[0052] The preset convolutional neural network module can be a pre-defined neural network module. Optionally, the preset neural network module can be a deep convolutional neural network ResNet18. Average pooling layers can be used to reduce the dimensionality of high-dimensional feature maps to obtain a lower-dimensional representation. Fully connected layers can be used to integrate scattered features into a global representation. The convolutional neural network module to be trained can be a module with the last average pooling layer and the last fully connected layer of the preset convolutional neural network module removed.

[0053] Specifically, a preset convolutional neural network module is obtained, and the last average pooling layer and the last fully connected layer of the preset convolutional neural network module are removed to obtain the convolutional neural network module to be trained.

[0054] For example, the preset neural network module is a deep convolutional neural network ResNet18. The last average pooling layer and the last fully connected layer in the ResNet18 network are removed, and the ResNet18 network after removal is used as the convolutional neural network module to be trained.

[0055] S220. Obtain the preset forward feedback module, and based on the preset forward feedback module, the convolutional neural network module to be trained, the feature alignment module, and the convolutional layer to be adjusted, obtain the convolutional neural network model to be trained.

[0056] The preset forward feedback module, from an input-output perspective, includes at least two linear layers, a feature fusion and concatenation layer, and at least one other linear layer. The at least two linear layers are parallel linear layers used to perform parallel feature extraction on the features input to the preset forward feedback module. The feature fusion and concatenation layer is used to fuse and concatenate at least two features output by the at least two linear layers. The at least one linear layer is used to perform feature extraction again on the fused and concatenated features to determine the final output features of the preset forward feedback module based on the features output by the at least one linear layer and the features of the initial input preset forward feedback module. Optionally, the preset forward feedback module can be a text reparameterization module in a YOLOo model. The preset forward feedback module is a lightweight auxiliary network; introducing it allows for the introduction of only a small number of training parameters to enhance the text encoding features. During the inference phase, the reparameterization method is used to reparameterize these features back into the main network to improve inference efficiency.

[0057] The feature alignment module is used to align the features output by the preset feedforward module with the features output by the convolutional neural network module to be trained. The convolutional layer to be adjusted can be a pre-set 1*1 convolutional layer, used to replace the last average pooling layer and the last fully connected layer that were removed. The modules of the convolutional neural network model to be trained are described in the following order from the perspective of input and output: preset feedforward module, convolutional neural network module to be trained, feature alignment module, and convolutional layer to be adjusted. The preset feedforward module and the convolutional neural network module to be trained can be two parallel modules.

[0058] Specifically, a preset forward feedback module is obtained, and a convolutional neural network model to be trained is constructed based on the preset forward feedback module, the convolutional neural network module to be trained, the feature alignment module, and the convolutional layer to be adjusted. The trained convolutional neural network model is obtained by training the convolutional neural network model to be trained, and then the first model is obtained by adjusting it.

[0059] S230. Based on the training samples, the convolutional neural network model to be trained is trained to obtain a trained convolutional neural network model.

[0060] It should be noted that since the text encoding module is a pre-trained module, it may not be included in the model being built and trained. Correspondingly, each training sample may include: the sample text encoding features corresponding to the sample prompt word information, the sample coal pile image, and the corresponding theoretical detection results. To improve the accuracy of the trained convolutional neural network model, as many and rich training samples as possible can be obtained to process the convolutional neural network model to be trained based on multiple training samples.

[0061] Specifically, the training samples are used to train the convolutional neural network model to be trained. Based on the corresponding loss function and the output of the convolutional neural network model to be trained, the model parameters of the convolutional neural network model to be trained are corrected to obtain a trained convolutional neural network model.

[0062] In this embodiment of the invention, the process of training the convolutional neural network model to be trained may be as follows: acquiring multiple sample coal pile images and sample prompt word information corresponding to the sample coal pile images, and encoding each sample prompt word information based on a pre-trained text encoding module to obtain sample text encoding features corresponding to each sample prompt word information; constructing training samples based on the sample coal pile images, sample text encoding features, and corresponding theoretical detection results; inputting the training samples into the convolutional neural network model to be trained to obtain actual detection results; determining the cross-entropy loss value based on the theoretical detection results and actual detection results, and correcting the model parameters of the convolutional neural network model to be trained based on the cross-entropy loss value to obtain the trained convolutional neural network model.

[0063] The sample prompt text can be used to characterize the degree of coal blockage in the sample coal blockage images. The sample coal blockage images can be obtained from a relevant database, including images of coal located at the connection points of multiple belt conveyors, or images of coal located at the connection points of multiple belt conveyors acquired by an image acquisition device at a preset location. Optionally, each sample coal blockage image can be defined, determining whether it belongs to mild coal blockage, moderate coal blockage, severe coal blockage, or normal. Based on the determined degree of coal blockage, the sample prompt text, i.e., thisis{degree of coal blockage}, is determined.

[0064] The sample text encoding features can be the features obtained after encoding the sample prompt words through a text encoding module. The theoretical detection result can be a predetermined theoretical evaluation attribute (e.g., theoretical probability value) of the sample coal pile image under the coal pile degree represented by the sample prompt word information.

[0065] The actual detection result can be the output of the convolutional neural network model to be trained, representing the actual evaluation attribute (e.g., actual probability value) under the coal accumulation degree characterized by the sample prompt word information. The cross-entropy loss value can be the loss value obtained after processing the theoretical and actual detection results according to a preset cross-entropy loss function. The cross-entropy loss value is used to characterize the degree of difference between the theoretical and actual detection results.

[0066] Specifically, multiple sample coal pile images are acquired, and sample prompt words corresponding to each sample coal pile image are determined. Each sample prompt word is encoded using a pre-trained text encoding module to obtain sample text encoding features corresponding to each sample prompt word. Training samples are constructed based on the sample coal pile images, sample text encoding features, and theoretical detection results corresponding to the sample coal pile images and sample text encoding features. The training samples are input into the convolutional neural network model to be trained to obtain actual detection results. A preset cross-entropy loss function is used to process the theoretical and actual detection results, resulting in a cross-entropy loss value. This cross-entropy loss value is then used to correct the model parameters of the convolutional neural network model to be trained.

[0067] When using cross-entropy loss to correct model parameters in a training convolutional neural network (CNN) model, the convergence of a preset cross-entropy loss function can be used as a training objective. This could be achieved by checking if the training error is less than a preset error, if the error trend stabilizes, or if the current number of iterations equals a preset number. If convergence is detected—for example, if the training error of the preset cross-entropy loss function is less than the preset error, or if the error trend stabilizes—it indicates that the CNN model has been trained successfully, and iterative training can be stopped. If convergence has not yet been achieved, additional training samples can be obtained to continue training the CNN model until the training error of the preset cross-entropy loss function is within a preset range. When the training error of the preset cross-entropy loss function converges, the trained CNN model is obtained.

[0068] Optionally, the actual detection results can be obtained by processing the training samples through the convolutional neural network model to be trained in the following ways: processing the sample text encoding features in the training samples based on the preset forward feedback module in the convolutional neural network model to be trained to obtain text prompt embedding features; performing convolution processing on the sample coal pile image in the training samples based on the convolutional neural network module to be trained to obtain sample image features; performing feature alignment processing on the sample image features and text prompt embedding features based on the feature alignment module in the convolutional neural network model to be trained to obtain sample alignment features; and processing the sample alignment features through the convolutional layer to be adjusted to obtain the actual detection results.

[0069] The text prompt embedding features can be features obtained by processing the sample text encoding features through a preset forward feedback module. Optionally, the preset forward feedback module can be as follows: Figure 3 As shown, in Figure 3 In this process, the sample text encoding features are processed by two parallel linear layers of the preset forward feedback module, resulting in two linear output features. The two linear output features are then fused and concatenated by the feature fusion and concatenation layer of the preset forward feedback module to obtain a fused feature. This fused feature is then processed by the linear layer of the preset forward feedback module to obtain a linear fused output feature. Finally, the sample text encoding features and the linear fused output feature are concatenated to obtain the text prompt embedding feature in the text prompt embedding dimension.

[0070] The sample image features can be image features obtained after extracting features from sample coal pile images. Optionally, the input dimension of the convolutional neural network module to be trained can be 16*3*256*256, where 16 represents the number of input sample coal pile images, 3 is the number of channels, and 256 represents the width and height of the sample coal pile images. The sample alignment features can be features obtained by aligning the sample image features and text prompt embedding features. Optionally, the feature alignment module can align the text embedding features and image features to obtain sample alignment features. The feature dimension of the sample alignment features can be 16*4*8*8. Optionally, processing the sample alignment features through the convolutional layer to be adjusted to obtain the actual detection result can be done by processing the sample alignment features through the convolutional layer to be adjusted, and performing pooling and flattening operations to meet the actual detection result under a preset feature dimension, where the preset feature dimension can be 16*4.

[0071] Specifically, the text encoding features of the training samples are processed sequentially through at least two linear layers, a feature fusion and concatenation layer, and at least one linear layer in the pre-defined feedforward module of the convolutional neural network model to obtain text prompt embedding features. The coal pile image is then convolved using the convolutional neural network module to obtain sample image features. The sample image features and text prompt embedding features are aligned using the feature alignment module to obtain aligned features. Finally, the aligned features are convolved using an adjustment convolutional layer, and pooling and flattening operations are performed on the convolved features to obtain the actual detection results. These results are then used to correct the model parameters of the convolutional neural network model to be trained.

[0072] S240. Adjust the trained convolutional neural network model to obtain the first model.

[0073] Specifically, the trained convolutional neural network model includes: a pre-defined feedforward module, a convolutional neural network module to be trained, a feature alignment module, and a convolutional layer to be adjusted. A reparameterization method can be used to fuse the model parameters of the pre-defined feedforward module in the trained convolutional neural network model with the model parameters of the convolutional layer to be adjusted, resulting in the target convolutional layer with fused model parameters. The pre-defined feedforward module is then removed to obtain the first model.

[0074] It should be noted that after obtaining the first model, sample coal pile images, sample text encoding features of sample prompt words, and corresponding theoretical detection results can also be obtained. The first model can be trained by using sample coal pile images, sample prompt words, and the corresponding theoretical detection results to ensure the accuracy of the first model.

[0075] In this embodiment of the invention, the method for adjusting the trained convolutional neural network model may be as follows: the model parameters corresponding to the preset forward feedback module in the trained convolutional neural network model are fused with the model parameters corresponding to the convolutional layer to be adjusted to obtain the target convolutional layer; the preset forward feedback module in the trained convolutional neural network model is removed, and the convolutional neural network module to be trained in the trained convolutional neural network model is taken as the target convolutional neural network module; based on the target convolutional neural network module, the feature alignment module, and the target convolutional layer in the trained convolutional neural network model, a first model is obtained.

[0076] The target convolutional layer can be the convolutional layer to be adjusted after fusing the model parameters corresponding to the preset forward feedback module with the model parameters corresponding to the convolutional layer to be adjusted. The target convolutional neural network module is the convolutional neural network module to be trained in the trained convolutional neural network model.

[0077] Specifically, the model parameters corresponding to the preset feedforward module in the trained convolutional neural network model are fused with the model parameters corresponding to the convolutional layer to be adjusted, resulting in the target convolutional layer with fused model parameters. The preset feedforward module in the trained convolutional neural network model is then removed, and the convolutional neural network module to be trained in the trained model is used as the target convolutional neural network module. The first model is constructed using the target convolutional neural network module, the feature alignment module, and the target convolutional layer. That is, the first model, from the perspective of input and output, includes: the target convolutional neural network module, the feature alignment module, and the target convolutional layer.

[0078] It should be noted that after obtaining the first model, the target coal pile detection model can be determined based on the text encoding module and the first model.

[0079] Additionally, it should be noted that, since the model weight parameters corresponding to the prompt words in the target coal pile detection model are different from the model weight parameters corresponding to the image to be processed, if the input of the target coal pile detection model is only the image to be processed, the target coal pile detection model can also output the evaluation attributes for each degree of coal pile, such as the probability values ​​of no pile blockage, slight pile blockage, moderate pile blockage, and severe pile blockage.

[0080] The technical solution of this embodiment obtains a pre-defined convolutional neural network (CNN) module by removing the last average pooling layer and the last fully connected layer of the pre-defined CNN module, thus obtaining a CNN module to be trained. A pre-defined feedforward module is then obtained, and based on the pre-defined feedforward module, the CNN module to be trained, the feature alignment module, and the convolutional layer to be adjusted, a CNN model to be trained is obtained. Introducing the pre-defined feedforward module allows for the introduction of only a small number of training parameters, enabling the network to learn the feature parameters after text encoding, facilitating the reparameterization of this module into the main network during the inference phase. The CNN model to be trained is then trained using training samples, resulting in a trained CNN model. The model parameters of the pre-defined feedforward module are reparameterized into the convolutional layer to be adjusted through a reparameterization mechanism, resulting in a first model, which facilitates the deployment and application of subsequent models. After obtaining the trained first model, the target coal pile detection module can be determined based on the text encoding module and the first model. The target coal pile detection model is then used to process the image to be processed and the prompt word information, realizing the processing of image features and semantic features. This makes the severity of coal pile-up on the belt conveyor clearer, improves the efficiency and accuracy of coal pile detection, and ensures the safety of coal transportation.

[0081] Example 3

[0082] Figure 4 This is a schematic diagram of a coal pile detection device provided in Embodiment 3 of the present invention. Figure 4 As shown, the device includes an image information determination module 310 and a detection result determination module 320.

[0083] Image information determination module 310 is used to acquire an image to be processed including a target object and determine prompt word information corresponding to the image to be processed. The target object is coal located at the connection point of multiple belt conveyors, and the prompt word information is used to characterize the expected coal accumulation degree. Detection result determination module 320 is used to process the image to be processed and the prompt word information based on the target coal accumulation detection model to obtain the target coal accumulation detection result corresponding to the prompt word information. The target coal accumulation detection model includes a text encoding module and a first model, which is obtained by adjusting a trained convolutional neural network model.

[0084] The technical solution of this embodiment acquires an image to be processed, including the target object, and determines the corresponding prompt word information. A target coal pile detection model processes the image to be processed and the prompt word information to obtain the target coal pile detection result corresponding to the prompt word information. This enables the detection of coal accumulation at the connection points of multiple belt conveyors, providing data support for subsequent early warning. The target coal pile detection model processes the prompt word information and the image to be processed, achieving the processing of image features and semantic features, thereby more clearly defining the severity of coal accumulation on the belt conveyor. This invention solves the problems of low efficiency and accuracy, and high labor costs caused by manual monitoring of coal piles in existing technologies. By processing the prompt word information and the image to be processed using a target coal pile detection model, the efficiency and accuracy of coal pile detection are improved, ensuring the safety of coal transportation.

[0085] Based on the above embodiments, optionally, the first model includes: a target convolutional neural network module, a feature alignment module, and a target convolutional layer; and a detection result determination module, used to encode the prompt word information based on the text encoding module in the target coal pile detection model, and determine the text encoding features corresponding to the prompt word information; to perform convolution processing on the image to be processed according to the target convolutional neural network module, and determine the image features; to perform feature alignment processing on the text encoding features and image features through the feature alignment module, and obtain the alignment features; and to process the alignment features based on the target convolutional layer, and obtain the target coal pile detection result.

[0086] Optionally, the device further includes: a first model determination module, comprising: a convolutional neural network module determination unit, configured to obtain a preset convolutional neural network module and remove the last average pooling layer and the last fully connected layer of the preset convolutional neural network module to obtain a convolutional neural network module to be trained; a convolutional neural network model determination unit, configured to obtain a preset feedforward module and obtain a convolutional neural network model to be trained based on the preset feedforward module, the convolutional neural network module to be trained, the feature alignment module, and the convolutional layer to be adjusted; wherein, the preset feedforward module, from the perspective of input and output, includes at least: at least two linear layers, a feature fusion and concatenation layer, and at least one linear layer; a model training unit, configured to train the convolutional neural network model to be trained based on training samples to obtain a trained convolutional neural network model; and a first model determination unit, configured to adjust the trained convolutional neural network model to obtain a first model.

[0087] Optionally, the model training unit includes: a sample text encoding feature determination subunit, used to acquire multiple sample coal pile images and sample prompt word information corresponding to the sample coal pile images, and to encode each sample prompt word information based on a pre-trained text encoding module to obtain sample text encoding features corresponding to each sample prompt word information; a training sample determination subunit, used to construct training samples based on sample coal pile images, sample text encoding features, and corresponding theoretical detection results; an actual detection result determination subunit, used to input the training samples into the convolutional neural network model to be trained to obtain actual detection results; and a parameter modification subunit, used to determine the cross-entropy loss value based on the theoretical detection results and actual detection results, and to modify the model parameters of the convolutional neural network model to be trained based on the cross-entropy loss value to obtain the trained convolutional neural network model.

[0088] Optionally, the actual detection result determination subunit is used to process the sample text encoding features in the training samples based on the preset forward feedback module in the convolutional neural network model to obtain text prompt embedding features; to perform convolution processing on the sample coal pile images in the training samples based on the convolutional neural network module to be trained in the convolutional neural network model to obtain sample image features; to perform feature alignment processing on the sample image features and text prompt embedding features based on the feature alignment module in the convolutional neural network model to obtain sample alignment features; and to process the sample alignment features through the convolutional layer to be adjusted to obtain the actual detection result.

[0089] Optionally, the first model determining unit is used to fuse the model parameters corresponding to the preset forward feedback module in the trained convolutional neural network model with the model parameters corresponding to the convolutional layer to be adjusted to obtain the target convolutional layer; remove the preset forward feedback module in the trained convolutional neural network model, and take the convolutional neural network module to be trained in the trained convolutional neural network model as the target convolutional neural network module; and obtain the first model based on the target convolutional neural network module, feature alignment module, and target convolutional layer in the trained convolutional neural network model.

[0090] Optionally, the device further includes: an early warning processing module, used to perform early warning processing based on the target coal pile detection result and the first coal pile detection result if the target coal pile detection result meets the preset coal pile detection result, and the first coal pile detection result corresponding to the coal images collected within a preset time after the image acquisition time corresponding to the image to be processed all meet the preset coal pile detection result.

[0091] The coal pile detection device provided in the embodiments of the present invention can execute the coal pile detection method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the method.

[0092] Example 4

[0093] Figure 5 This is a schematic diagram of the structure of an electronic device provided in Embodiment 4 of the present invention. The electronic device 10 is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices (such as helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.

[0094] like Figure 5 As shown, the electronic device 10 includes at least one processor 11 and a memory, such as a read-only memory (ROM) 12 or a random access memory (RAM) 13, communicatively connected to the at least one processor 11. The memory stores computer programs executable by the at least one processor. The processor 11 can perform various appropriate actions and processes based on the computer program stored in the ROM 12 or loaded from storage unit 18 into the RAM 13. The RAM 13 may also store various programs and data required for the operation of the electronic device 10. The processor 11, ROM 12, and RAM 13 are interconnected via a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0095] Multiple components in electronic device 10 are connected to I / O interface 15, including: input unit 16, such as keyboard, mouse, etc.; output unit 17, such as various types of displays, speakers, etc.; storage unit 18, such as disk, optical disk, etc.; and communication unit 19, such as network card, modem, wireless transceiver, etc. Communication unit 19 allows electronic device 10 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0096] Processor 11 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. Processor 11 performs the various methods and processes described above, such as coal pile detection methods.

[0097] In some embodiments, the coal pile detection method may be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on electronic device 10 via ROM 12 and / or communication unit 19. When the computer program is loaded into RAM 13 and executed by processor 11, one or more steps of the coal pile detection method described above may be performed. Alternatively, in other embodiments, processor 11 may be configured to perform the coal pile detection method by any other suitable means (e.g., by means of firmware).

[0098] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), system-on-a-chip (SoCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0099] Computer programs for implementing the coal pile detection method of the present invention can be written in any combination of one or more programming languages. These computer programs can be provided to the processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the functions / operations specified in the flowcharts and / or block diagrams are implemented. The computer programs can be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0100] In particular, according to embodiments of the present invention, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of the present invention include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication unit 19, or installed from storage unit 18, or installed from ROM 12. When the computer program is executed by processor 11, it performs the functions defined in the methods of the embodiments of the present invention.

[0101] Example 5

[0102] Embodiment 5 of the present invention also provides a computer-readable storage medium storing computer instructions for causing a processor to execute a coal pile detection method, the method comprising:

[0103] The process involves acquiring an image to be processed that includes the target object and determining the prompt word information corresponding to the image. The target object is coal located at the connection point of multiple belt conveyors, and the prompt word information is used to characterize the expected degree of coal accumulation. The image to be processed and the prompt word information are processed based on the target coal accumulation detection model to obtain the target coal accumulation detection result corresponding to the prompt word information. The target coal accumulation detection model includes a text encoding module and a first model, which is obtained by adjusting a pre-trained convolutional neural network model.

[0104] In the context of this invention, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0105] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0106] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or computing systems that include middleware components (e.g., application servers), or computing systems that include frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0107] A computing system can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system to address the shortcomings of traditional physical hosts and VPS services, such as high management difficulty and weak business scalability.

[0108] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and this is not limited herein.

[0109] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A method for detecting coal piles, characterized in that, Applied to the coal transportation process, including: Acquire an image to be processed including a target object, and determine prompt word information corresponding to the image to be processed, wherein the target object is coal located at the connection point of multiple belt conveyors, and the prompt word information is used to characterize the degree of coal accumulation to be detected; The target coal pile detection model is used to process the image to be processed and the prompt word information to obtain the target coal pile detection result corresponding to the prompt word information; The target coal pile detection model includes a text encoding module and a first model, which is obtained by adjusting a pre-trained convolutional neural network model.

2. The method according to claim 1, characterized in that, The first model includes: a target convolutional neural network module, a feature alignment module, and a target convolutional layer. The step of processing the image to be processed and the prompt word information based on the target coal pile detection model to obtain the target coal pile detection result corresponding to the prompt word information includes: The text encoding module in the target coal pile detection model is used to encode the prompt word information to determine the text encoding features corresponding to the prompt word information. The target convolutional neural network module performs convolution processing on the image to be processed to determine image features; The feature alignment module performs feature alignment processing on the text encoding features and the image features to obtain aligned features; The alignment features are processed based on the target convolutional layer to obtain the target coal pile detection result.

3. The method according to claim 1, characterized in that, The method further includes: Obtain a preset convolutional neural network module, and remove the last average pooling layer and the last fully connected layer of the preset convolutional neural network module to obtain the convolutional neural network module to be trained. A preset forward feedback module is obtained, and a convolutional neural network model to be trained is obtained based on the preset forward feedback module, the convolutional neural network module to be trained, the feature alignment module, and the convolutional layer to be adjusted; wherein, the preset forward feedback module, from the perspective of input and output, includes at least: at least two linear layers, a feature fusion and splicing layer, and at least one linear layer. The training process is performed on the convolutional neural network model to be trained based on the training samples to obtain a trained convolutional neural network model. The trained convolutional neural network model is adjusted to obtain the first model.

4. The method according to claim 3, characterized in that, The step of training the convolutional neural network model to be trained based on training samples to obtain a trained convolutional neural network model includes: Multiple sample coal pile images and sample prompt words corresponding to the sample coal pile images are acquired, and each sample prompt word is encoded based on a pre-trained text encoding module to obtain sample text encoding features corresponding to each sample prompt word. Training samples are constructed based on the sample coal pile images, the sample text encoding features, and the corresponding theoretical detection results. The training samples are input into the convolutional neural network model to be trained to obtain the actual detection results; Based on the theoretical detection results and the actual detection results, the cross-entropy loss value is determined, and the model parameters of the convolutional neural network model to be trained are corrected based on the cross-entropy loss value to obtain the trained convolutional neural network model.

5. The method according to claim 4, characterized in that, The step of inputting the training samples into the convolutional neural network model to be trained to obtain the actual detection results includes: Based on the preset forward feedback module in the convolutional neural network model to be trained, the sample text encoding features in the training samples are processed to obtain the text prompt embedding features; Based on the convolutional neural network module in the model to be trained, the sample coal pile image in the training samples is convolved to obtain the sample image features; Based on the feature alignment module in the convolutional neural network model to be trained, feature alignment processing is performed on the sample image features and the text prompt embedding features to obtain sample alignment features; The sample alignment features are processed by the convolutional layer to be adjusted to obtain the actual detection results.

6. The method according to claim 3, characterized in that, The process of adjusting the trained convolutional neural network model to obtain the first model includes: The model parameters corresponding to the preset forward feedback module in the trained convolutional neural network model are fused with the model parameters corresponding to the convolutional layer to be adjusted to obtain the target convolutional layer. Remove the preset forward feedback module from the trained convolutional neural network model, and use the convolutional neural network module to be trained in the trained convolutional neural network model as the target convolutional neural network module; Based on the target convolutional neural network module, feature alignment module, and target convolutional layer in the trained convolutional neural network model, the first model is obtained.

7. The method according to claim 1, characterized in that, The method further includes: If the target coal pile detection result meets the preset coal pile detection result, and the first coal pile detection result corresponding to the coal image collected within a preset time after the image acquisition time corresponding to the image to be processed all meet the preset coal pile detection result, then an early warning processing is performed based on the target coal pile detection result and the first coal pile detection result.

8. A coal pile detection device, characterized in that, include: The image information determination module is used to acquire an image to be processed including a target object and determine prompt word information corresponding to the image to be processed, wherein the target object is coal located at the connection point of multiple belt conveyors, and the prompt word information is used to characterize the expected coal accumulation degree. The detection result determination module is used to process the image to be processed and the prompt word information based on the target coal pile detection model to obtain the target coal pile detection result corresponding to the prompt word information; The target coal pile detection model includes a text encoding module and a first model, which is obtained by adjusting a pre-trained convolutional neural network model.

9. An electronic device, characterized in that, The electronic device includes: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the coal pile detection method according to any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that cause a processor to execute the coal pile detection method according to any one of claims 1-7.

Images