End-to-end dust identification method, device, equipment and medium
By combining the backbone network and detection branches in dust recognition, using the overall classification head and composite loss function, the problems of low dust recognition efficiency and high misidentification rate in the existing technology are solved, and efficient and accurate dust recognition is achieved, which is suitable for Environmental monitoring and industrial safety.
Patent Information
- Application Number
- CN202411318113.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-20
- Publication Date
- 2025-01-10
AI Technical Summary
The existing dust recognition method has high error recognition rate when dealing with spray and dust, resulting in low computational redundancy and inference speed, making it difficult to effectively distinguish spray and dust.
The end-to-end dust recognition method is adopted, and the backbone network is combined with the detection branch, the overall classification head is added, the feature extraction is shared, and the composite loss function is designed to achieve the seamless fusion of classification and detection, reducing calculation redundancy, and improving identification efficiency.
It improves the accuracy and inference speed of dust recognition, reduces misidentification, and achieves efficient identification in the fields of environmental monitoring and industrial safety.
Smart Images
Figure CN119295805A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of image recognition technology, and in particular to an end-to-end dust recognition method, device, equipment and medium. Background Art
[0002] Dust identification is of great significance in environmental monitoring, industrial safety and other fields.
[0003] In the prior art, dust recognition methods mainly rely on the cascade of local feature detection models and classification models. The classification model has the characteristics of a large receptive field, can judge the entire image, and effectively distinguish between spray and dust.
[0004] However, when dealing with spray and dust, the local feature detection model often leads to a large number of misidentifications due to the high similarity of the local features of the two. Using the classification model in cascade with the detection model will introduce computational redundancy and significantly reduce the inference speed. Summary of the invention
[0005] Based on this, it is necessary to provide an end-to-end dust identification method, device, equipment and medium to address the above technical problems, which can simultaneously realize dust detection and classification and improve identification efficiency.
[0006] An end-to-end dust identification method, comprising:
[0007] Obtain an image to be recognized;
[0008] Constructing an image recognition model, wherein the image recognition model includes: a backbone network, an overall classification head, and a detection branch;
[0009] Using the backbone network, extracting features of the image to be identified to obtain image features;
[0010] Using the overall classification head, judging whether there is spray in the image to be identified according to the image features;
[0011] If it is determined that there is spray in the image to be identified, outputting a spray identification result;
[0012] If it is determined that there is no spray in the image to be identified, the detection branch is used to detect the image to be identified according to the image features, and a dust identification result is output.
[0013] In one embodiment, after outputting the spray recognition result or the dust recognition result, the method further includes:
[0014] Calculating the image loss of the image to be recognized, and updating the image recognition model according to the image loss to obtain an optimized image recognition model;
[0015] The image to be identified is identified by using an optimized image recognition model, and a spray identification result or a dust identification result is output.
[0016] In one embodiment, the image loss includes:
[0017] L=L cls-global +y spray *L cls-local +y spray *L reg
[0018] in,
[0019] L cls-global =-w1*y spray *log(p spray +ε)+w0*(1-y spray )*log(1-p spray +ε)
[0020] Where L is the image loss, L cls-global is the classification loss of the overall classification head, L cls-local is the classification loss of the detection branch, L reg is the regression loss of the detection branch, y spray For labels, p spray is the probability that the model predicts that there is spray in the image; w1 and w0 are the weights of positive and negative samples respectively; ε is a very small positive number.
[0021] In one embodiment, obtaining an image to be recognized includes:
[0022] Acquire an image to be detected, and preprocess the image to be detected to obtain an image to be recognized;
[0023] The preprocessing includes: size standardization, normalization and enhancement to ensure the consistency of input data and improve the robustness of the model.
[0024] In one embodiment, the backbone network includes: a feature extraction layer to perform feature extraction and obtain image features;
[0025] The image features are shared by the overall classification head and the detection branch.
[0026] In one embodiment, the overall classification head includes: a global average pooling layer and a fully connected layer;
[0027] The global average pooling layer is used to compress the feature map into a vector of fixed length;
[0028] The fully connected layer is used to output the probability value of whether spray exists or not.
[0029] In one embodiment, the detection branch includes: a feature pyramid network, a classifier, and a bounding box regressor;
[0030] The feature pyramid network is used to extract multi-scale features;
[0031] The classifier is used to determine the category of the object to be detected;
[0032] The bounding box regressor is used to determine the position of the object to be detected.
[0033] An end-to-end dust identification device, comprising:
[0034] An acquisition module, used for acquiring an image to be recognized;
[0035] A construction module, used to construct an image recognition model, wherein the image recognition model includes: a backbone network, an overall classification head, and a detection branch;
[0036] An extraction module, used to extract features of the image to be identified by using the backbone network to obtain image features;
[0037] A judgment module, using the overall classification head to judge whether there is spray in the image to be identified according to the image features;
[0038] A spray module, configured to output a spray recognition result if it is determined that there is spray in the image to be recognized;
[0039] The dust module is used to detect the image to be identified according to the image features by using the detection branch if it is determined that there is no spray in the image to be identified, and output a dust identification result.
[0040] A computer device comprises a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the above method when executing the computer program.
[0041] A computer-readable storage medium stores a computer program, which implements the steps of the above method when executed by a processor.
[0042] The above-mentioned end-to-end dust identification method, device, equipment and medium add an overall classification head between the backbone network and the detection branch, perform dust detection based on the overall classification result, combine classification with detection, and realize the seamless integration of classification and detection functions; it can not only take advantage of the large receptive field of the classification model to quickly determine whether the image contains dust, effectively distinguish between spray and dust, reduce misidentification, and effectively reduce the amount of calculation for subsequent local detection; it can also avoid computational redundancy by sharing the backbone network, improve the reasoning speed and dust identification efficiency. At the same time, the present application optimizes the overall classification head and the detection branch in the same training process by sharing the backbone network, which means that the model can complete the whole process from image input to the final recognition result in one forward propagation; through this end-to-end training method, the accuracy of the model is further guaranteed and its operation efficiency is further improved. In addition, a composite loss function is designed, which includes: the classification loss of the overall classification head, the classification loss of the detection branch, and the regression loss of the detection branch, which can ensure that the model effectively learns the difference between dust and spray, and accurately locates dust and objects that produce dust. Therefore, compared with traditional methods, this application has higher recognition accuracy and faster reasoning speed, and has broad application prospects in environmental monitoring, industrial safety and other fields. BRIEF DESCRIPTION OF THE DRAWINGS
[0043] Figure 1 This is an application scenario diagram of an end-to-end dust identification method in an embodiment;
[0044] Figure 2 A schematic diagram of a process of an end-to-end dust identification method in an embodiment;
[0045] Figure 3 A schematic diagram of determining the presence of spray and outputting a spray recognition result in an embodiment;
[0046] Figure 4 A schematic diagram of determining that there is no spray and outputting a dust identification result in an embodiment;
[0047] Figure 5 is a structural block diagram of an end-to-end dust identification device in an embodiment;
[0048] Figure 6 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION
[0049] In order to make the purpose, technical solutions and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present application.
[0050] In addition, the descriptions of "first", "second", etc. in this application are only for descriptive purposes and cannot be understood as indicating or implying their relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" or "second" may explicitly or implicitly include at least one of the features. In the description of this application, "multiple groups" means at least two groups, such as two groups, three groups, etc., unless otherwise clearly and specifically defined.
[0051] In this application, unless otherwise clearly specified and limited, the terms "connection", "fixation", etc. should be understood in a broad sense. For example, "fixation" can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection, an electrical connection, a physical connection, or a wireless communication connection; it can be a direct connection, or an indirect connection through an intermediate medium, or it can be the internal connection of two elements or the interaction relationship between two elements, unless otherwise clearly defined. For ordinary technicians in this field, the specific meanings of the above terms in this application can be understood according to specific circumstances.
[0052] In addition, the technical solutions between the various embodiments of the present application can be combined with each other, but it must be based on the fact that ordinary technicians in the field can implement it. When the combination of technical solutions is contradictory or cannot be implemented, it should be deemed that such combination of technical solutions does not exist and is not within the scope of protection required by this application.
[0053] The end-to-end dust identification method provided in this application can be applied to Figure 1 In the application scenario diagram shown, the terminal 102 communicates with the server 104 through the network, and the terminal 102 may include but is not limited to various personal computers, laptops, smart phones, tablet computers and portable wearable devices, and the server 104 may be various portal websites, servers corresponding to the backend of the work system, etc.
[0054] This application provides an end-to-end dust identification method, such as Figure 2 The flowchart shown in FIG. 1 shows a method for applying the method to Figure 1 The following is an example of a terminal in the example, including:
[0055] Step 202: Obtain an image to be recognized.
[0056] Specifically, an image to be detected is acquired, and the image to be detected is preprocessed to obtain an image to be identified; wherein the preprocessing includes: size standardization, normalization, and enhancement (such as random cropping, flipping, etc.).
[0057] In this step, preprocessing can ensure the consistency of input data and improve the robustness of the model.
[0058] Step 204, constructing an image recognition model, the image recognition model includes: a backbone network, an overall classification head, and a detection branch.
[0059] In this step, the overall classification head is connected to the backbone network, and the detection branch is connected to both the backbone network and the overall classification head.
[0060] Step 206: Use the backbone network to extract features from the image to be identified to obtain image features.
[0061] Specifically, the backbone network is used for feature extraction and adopts advanced convolutional neural networks, such as ShuffleNet, ResNet, EfficientNet, etc., including: feature extraction layer, the image features output by the feature extraction layer are shared by the overall classification head and the detection branch, realizing the fusion of classification and detection functions and avoiding computational redundancy.
[0062] In this step, the design of the backbone network needs to take into account the characteristics of the dust identification task to ensure that the extracted features contain both local details and global information.
[0063] Step 208: Using the overall classification head, judging whether there is spray in the image to be identified based on the image features.
[0064] Specifically, the overall classification head is used to perform overall classification judgment on the image of spray and dust, including: a global average pooling layer and a fully connected layer; the global average pooling layer is used to compress the feature map into a vector of fixed length; the fully connected layer is used to output the probability value of whether spray exists or not.
[0065] In this step, the global classification head is located at the top of the backbone network, and the specific global average pooling layer and the fully connected layer are both existing technologies.
[0066] Step 210: If it is determined that there is spray in the image to be identified, the spray identification result is output.
[0067] In this step, the spray recognition result is output, and the detection step of the detection branch is skipped.
[0068] Step 212: If it is determined that there is no spray in the image to be identified, the detection branch is used to detect the image to be identified according to the image features, and the dust identification result is output.
[0069] Specifically, the detection branch is used to perform detection using regression and classification algorithms to accurately locate and classify local dust and dust-producing objects, including: feature pyramid network, classifier and bounding box regressor; feature pyramid network (FPN) is used to extract multi-scale features; the classifier is used to determine the category of the object to be detected; the bounding box regressor is used to determine the position of the object to be detected.
[0070] In this step, the detection branch can adopt the existing target detection architecture, such as YOLO, DETR, etc. The specific feature pyramid network, classifier and bounding box regressor are all existing technologies.
[0071] In this embodiment, only when the scene is in a non-spray state without spray, detection (local classification and regression) is performed, and the spray recognition result is output as follows: Figure 3 As shown, the dust identification results are output as Figure 4 As shown, cup_work indicates that the category name is suction cup operation, cup_off indicates that the category name is suction cup non-operation, and dust_off indicates dust diffusion, all of which indicate confidence levels.
[0072] In another embodiment, after outputting the spray recognition result or the dust recognition result, the method further includes:
[0073] The image loss of the image to be identified is calculated, and the image recognition model is updated according to the image loss to obtain an optimized image recognition model; the image to be identified is identified using the optimized image recognition model, and a spray recognition result or a dust recognition result is output.
[0074] Image losses include:
[0075] L=L cls-global +y spray *L cls-local +y spray *L reg
[0076] in,
[0077] L cls-global =-w1*y spray *log(p spray +ε)+w0*(1-y spray )*log(1-p spray +ε)
[0078] Where L is the image loss, Lcls-global is the classification loss of the overall classification head, L cls-local is the classification loss of the detection branch (which can be obtained according to the existing technology), L reg is the regression loss of the detection branch (which can be obtained according to the existing technology), y spray is the label, 0 if spray is present, 1 otherwise, p spray is the probability that the model predicts that there is spray in the image; w1 and w0 are the weights of positive samples and negative samples respectively, which can be adaptively adjusted according to the category distribution of the data set so that the model can perform well on different data sets. For example, if the number of positive samples is much less than the number of negative samples (that is, the number of positive samples differs from the number of negative samples by one order of magnitude), w1 can be set greater than w0 to increase the loss contribution of positive samples; ε is a very small positive number used for smoothing to avoid numerical instability problems, especially when the model predicts that the probability is close to 0 or 1, which helps to improve the stability of the loss function and the training effect of the model.
[0079] The above end-to-end dust recognition method adds an overall classification head between the backbone network and the detection branch, performs dust detection based on the overall classification results, combines classification with detection, and realizes the seamless integration of classification and detection functions; it can not only take advantage of the large receptive field of the classification model to quickly determine whether the image contains dust, effectively distinguish between spray and dust, reduce misidentification, and effectively reduce the amount of calculation for subsequent local detection; it can also avoid computational redundancy by sharing the backbone network, improve the reasoning speed and dust recognition efficiency. At the same time, the present application optimizes the overall classification head and the detection branch simultaneously in the same training process by sharing the backbone network, which means that the model can complete the entire process from image input to the final recognition result in one forward propagation; through this end-to-end training method, the accuracy of the model is further guaranteed and its operating efficiency is further improved. In addition, a composite loss function is designed, which includes: the classification loss of the overall classification head, the classification loss of the detection branch, and the regression loss of the detection branch, which can ensure that the model effectively learns the difference between dust and spray, and accurately locates dust and objects that produce dust. Therefore, compared with traditional methods, this application has higher recognition accuracy and faster reasoning speed, and has broad application prospects in environmental monitoring, industrial safety and other fields.
[0080] It should be understood that although Figure 2 The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Moreover, Figure 2At least part of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.
[0081] The present application also provides an end-to-end dust identification device, such as Figure 5 As shown, in one embodiment, it includes: an acquisition module 502, a construction module 504, an extraction module 506, a judgment module 508, a spray module 510 and a dust module 512, wherein:
[0082] An acquisition module 502 is used to acquire an image to be recognized;
[0083] A construction module 504 is used to construct an image recognition model, which includes: a backbone network, an overall classification head, and a detection branch;
[0084] An extraction module 506 is used to extract features of the image to be identified by using a backbone network to obtain image features;
[0085] The judgment module 508 uses the overall classification head to judge whether there is spray in the image to be identified according to the image features;
[0086] The spray module 510 is used to output a spray recognition result if it is determined that there is spray in the image to be recognized;
[0087] The dust module 512 is used to detect the image to be identified according to the image features and output the dust identification result by using the detection branch if it is determined that there is no spray in the image to be identified.
[0088] For the specific definition of an end-to-end dust identification device, please refer to the definition of an end-to-end dust identification method above, which will not be repeated here. Each module in the above device can be implemented in whole or in part by software, hardware and a combination thereof. Each of the above modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to each of the above modules.
[0089] In one embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as follows: Figure 6As shown. The computer device includes a processor, a memory, a network interface, a display screen and an input device connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, an end-to-end dust identification method is implemented. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covering the display screen, or a key, trackball or touchpad set on the computer device housing, or an external keyboard, touchpad or mouse, etc.
[0090] Those skilled in the art will understand that Figure 6 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.
[0091] In one embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the method in the above embodiment when executing the computer program.
[0092] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps of the method in the above embodiment are implemented.
[0093] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in the present application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).
[0094] The contents not described in detail in this specification belong to the prior art known to professional and technical personnel in this field.
[0095] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.
[0096] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.
Claims
1. An end-to-end dust identification method, characterized in that: include: Obtain an image to be recognized; Constructing an image recognition model, wherein the image recognition model includes: a backbone network, an overall classification head, and a detection branch; Using the backbone network, extracting features of the image to be identified to obtain image features; Using the overall classification head, judging whether there is spray in the image to be identified according to the image features; If it is determined that there is spray in the image to be identified, outputting a spray identification result; If it is determined that there is no spray in the image to be identified, the detection branch is used to detect the image to be identified according to the image features, and a dust identification result is output.
2. The end-to-end dust identification method according to claim 1, characterized in that: After outputting the spray identification result or dust identification result, it also includes: Calculating the image loss of the image to be recognized, and updating the image recognition model according to the image loss to obtain an optimized image recognition model; The image to be identified is identified by using an optimized image recognition model, and a spray identification result or a dust identification result is output.
3. The end-to-end dust identification method according to claim 2, characterized in that: The image losses include: L=L cls-global +y spray *L cls-local +y spray *L reg in, L cls-global =-w1*y spray *log(p spray +ε)+w0*(1-y spray )*log(1-p spray +e) Where L is the image loss, L cls-global is the classification loss of the overall classification head, L cls-local is the classification loss of the detection branch, L reg is the regression loss of the detection branch, y spray For labels, p spray is the probability that the model predicts that there is spray in the image; w1 and w0 are the weights of positive and negative samples respectively; ε is a very small positive number.
4. An end-to-end dust identification method according to any one of claims 1 to 3, characterized in that: Get the image to be recognized, including: Acquire an image to be detected, and preprocess the image to be detected to obtain an image to be recognized; The preprocessing includes: size standardization, normalization and enhancement to ensure the consistency of input data and improve the robustness of the model.
5. An end-to-end dust identification method according to any one of claims 1 to 3, characterized in that: The backbone network includes: a feature extraction layer to extract features and obtain image features; The image features are shared by the overall classification head and the detection branch.
6. An end-to-end dust identification method according to any one of claims 1 to 3, characterized in that: The overall classification head includes: a global average pooling layer and a fully connected layer; The global average pooling layer is used to compress the feature map into a vector of fixed length; The fully connected layer is used to output the probability value of whether spray exists or not.
7. An end-to-end dust identification method according to any one of claims 1 to 3, characterized in that: The detection branch includes: a feature pyramid network, a classifier, and a bounding box regressor; The feature pyramid network is used to extract multi-scale features; The classifier is used to determine the category of the object to be detected; The bounding box regressor is used to determine the position of the object to be detected.
8. An end-to-end dust identification device, characterized in that: include: An acquisition module, used for acquiring an image to be recognized; A construction module, used to construct an image recognition model, wherein the image recognition model includes: a backbone network, an overall classification head, and a detection branch; An extraction module, used to extract features of the image to be identified by using the backbone network to obtain image features; A judgment module, using the overall classification head to judge whether there is spray in the image to be identified according to the image features; A spray module, configured to output a spray recognition result if it is determined that there is spray in the image to be recognized; The dust module is used to detect the image to be identified according to the image features by using the detection branch if it is determined that there is no spray in the image to be identified, and output a dust identification result.
9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.
10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.