An abnormal state recognition method, an abnormal state recognition device, and a terminal device

Abstract

The application belongs to the technical field of image processing, and provides an abnormal state recognition method, an abnormal state recognition device and a terminal device. The recognition method comprises the following steps: inputting a to-be-recognized image into a first recognition model to obtain a first recognition result of a first detection area and a second detection area output by the first recognition model; if it is determined according to the first recognition result that an object in the first detection area is in an abnormal state, inputting a first cut image corresponding to the second detection area into a second recognition model to obtain a second recognition result output by the second recognition model; and determining whether the object in the to-be-recognized image is in an abnormal state according to the first recognition result and the second recognition result. Due to the above scheme, the characteristics of the coal leakage detection scene are fully considered, the coal leakage is detected in different regions, and different recognition models are used in each region. Therefore, the scheme has high detection accuracy.

Description

Technical Field

[0001] This invention belongs to the technical field of image processing, and particularly relates to a method, device and terminal equipment for identifying abnormal states. Background Technology

[0002] In the operation and maintenance of thermal power plants, coal leakage detection of coal conveyor belts (coal leakage refers to coal falling into areas outside the conveyor belt) is one of the most important tasks. Timely detection of coal leakage on the coal conveyor belt can effectively reduce coal loss during coal transportation, improve production efficiency, and reduce production costs.

[0003] With the development of computer vision technology, it is gradually being used for coal leakage detection. However, traditional computer vision technology has low detection accuracy for coal leakage, which is a technical problem that urgently needs to be solved. Summary of the Invention

[0004] In view of this, embodiments of the present invention provide a method, device and terminal equipment for identifying abnormal states, in order to solve the technical problem of low detection accuracy of traditional computer vision technology for coal leakage detection.

[0005] A first aspect of this invention provides a method for identifying abnormal states, comprising:

[0006] The image to be recognized is input into the first recognition model to obtain a first recognition result of the first detection area and a second detection area output by the first recognition model; the first recognition result is used to indicate whether the object in the first detection area is in an abnormal state; wherein, the first detection area is the desired area for placing the object, and the second detection area is the undesired area for placing the object;

[0007] If it is determined that the object in the first detection area is in an abnormal state based on the first recognition result, then the first cropped image corresponding to the second detection area is input into the second recognition model to obtain the second recognition result output by the second recognition model; the second recognition result is used to indicate whether the object exists in the second detection area;

[0008] Based on the first recognition result and the second recognition result, it is determined whether the object in the image to be recognized is in an abnormal state.

[0009] Further, determining whether an object in the image to be identified is in an abnormal state based on the first recognition result and the second recognition result includes:

[0010] If it is determined that the object in the first detection area is in an abnormal state based on the first recognition result, and it is determined that the object exists in the second detection area based on the second recognition result, then it is determined that the object in the image to be recognized is in an abnormal state.

[0011] If the object in the first detection area is determined to be in an abnormal state based on the first recognition result, and the object is determined not to exist in the second detection area based on the second recognition result, then the object in the image to be recognized is determined to be in a normal state.

[0012] Furthermore, after the step of inputting the image to be recognized into the first recognition model to obtain the first recognition result of the first detection region and the second detection region output by the first recognition model, the method further includes:

[0013] If the object in the first detection area is determined to be in a normal state based on the first recognition result, then the object in the image to be recognized is determined to be in a normal state.

[0014] Furthermore, before the step of inputting the image to be recognized into the first recognition model to obtain the first recognition result of the first detection region and the second detection region output by the first recognition model, the method further includes:

[0015] Acquire training sample images and corresponding annotation data for the training sample images; the annotation data includes a first training result for a first training region, a second training region, and a final training result.

[0016] The first training sample image is input into the first initial model to obtain a first initial result and a second initial region output by the first initial model; the first initial result is used to indicate whether the first initial region is in an abnormal state; wherein, the first initial region and the second initial region are both image regions in the image to be identified;

[0017] Calculate the first loss value between the first initial result and the first training result;

[0018] Calculate the second loss value between the second initial region and the second training region;

[0019] Adjust the first initial model based on the first loss value and the second loss value;

[0020] Using other training sample images as the first training sample images, the steps of inputting the first training sample images into the first initial model, obtaining the first initial result of the first initial region and the second initial region output by the first initial model, and subsequent steps are performed repeatedly to obtain the first recognition model.

[0021] Furthermore, the identification method further includes:

[0022] For each of the second cropped images, the following steps are performed to obtain the second recognition model:

[0023] The second cropped image corresponding to the second training region is input into the second initial model to obtain the second initial result output by the second initial model.

[0024] Calculate a third loss value between the second initial result and the final training result;

[0025] The second initial model is adjusted based on the third loss value.

[0026] Further, acquiring the training sample images and the corresponding annotation data of the training sample images includes:

[0027] Acquire multiple initial images;

[0028] Each initial image is transformed to obtain multiple transformed images; the transformation process includes optical deformation, geometric deformation, color transformation, or simulated occlusion; wherein the number of transformed images is greater than the number of initial images;

[0029] The transformed image is input into a generative adversarial network (GAN) to obtain the training sample image output by the GAN; wherein the number of training sample images is greater than the number of transformed images.

[0030] Further, the step of inputting the transformed image into a generative adversarial network to obtain the training sample image output by the generative adversarial network includes:

[0031] The image region containing the target object is cropped from the transformed image to obtain a third cropped image;

[0032] The third cropped image is input into the generative adversarial network to obtain a simulated image output by the generative adversarial network;

[0033] The simulated image is combined with the transformed image to obtain the training sample image.

[0034] A second aspect of the present invention provides an abnormal state identification device, comprising:

[0035] The processing unit is configured to input the image to be recognized into a first recognition model to obtain a first recognition result of a first detection region and a second detection region output by the first recognition model; the first recognition result is used to indicate whether the object in the first detection region is in an abnormal state; wherein, the first detection region is the desired region for placing the object, and the second detection region is the undesired region for placing the object;

[0036] The judgment unit is configured to, if determined based on the first recognition result that the object in the first detection area is in an abnormal state, input the first cropped image corresponding to the second detection area into the second recognition model to obtain a second recognition result output by the second recognition model; the second recognition result is used to indicate whether the object exists in the second detection area;

[0037] The determining unit is configured to determine whether an object in the image to be identified is in an abnormal state based on the first identification result and the second identification result.

[0038] A third aspect of the present invention provides a terminal device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the method described in the first aspect.

[0039] A fourth aspect of the present invention provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the method described in the first aspect.

[0040] The beneficial effects of this invention compared to existing technologies are as follows: This invention inputs an image to be identified into a first recognition model to obtain a first recognition result for a first detection region and a second detection region output by the first recognition model. The first recognition result indicates whether an object in the first detection region is in an abnormal state. The first detection region is the desired region for placing the object, and the second detection region is the undesired region for placing the object. If the first recognition result determines that an object in the first detection region is in an abnormal state, a first cropped image corresponding to the second detection region is input into a second recognition model to obtain a second recognition result output by the second recognition model. The second recognition result indicates whether the object exists in the second detection region. Based on the first and second recognition results, it is determined whether the object in the image to be identified is in an abnormal state. Because the above scheme fully considers the characteristics of the coal leakage detection scenario, coal leakage is detected in zones, and each zone uses a different recognition model. For the first detection region, the first recognition model is used to identify whether the object in the first detection region is in an abnormal state. Furthermore, the second recognition model is used to identify whether there is an object in the second detection area only when the object in the first detection area is in an abnormal state. Then, based on the recognition results output by the first and second recognition models, it is determined whether the object in the image to be recognized is in an abnormal state, thus making the scheme have high detection accuracy. Attached Figure Description

[0041] To more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0042] Figure 1 A schematic flowchart of an abnormal state identification method provided by the present invention is shown;

[0043] Figure 2 A schematic flowchart illustrating step 103 of an abnormal state identification method provided by the present invention is shown.

[0044] Figure 3 A schematic diagram of the overall process of an abnormal state identification method provided by the present invention is shown;

[0045] Figure 4 A schematic flowchart of another abnormal state identification method provided by the present invention is shown;

[0046] Figure 5A schematic flowchart illustrating step 103 of an abnormal state identification method provided by the present invention is shown.

[0047] Figure 6 A schematic flowchart illustrating step 103 of an abnormal state identification method provided by the present invention is shown.

[0048] Figure 7 A schematic diagram illustrating the generation process of training sample images provided by the present invention is shown;

[0049] Figure 8 A schematic diagram of an abnormal state identification device provided by the present invention is shown;

[0050] Figure 9 This is a schematic diagram of a terminal device provided in an embodiment of the present invention. Detailed Implementation

[0051] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of the invention. However, those skilled in the art will understand that the invention can be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods are omitted so as not to obscure the description of the invention with unnecessary detail.

[0052] It should be noted that this solution can be applied to multiple scenarios, such as sand and gravel transportation inspection, warehouse cargo inspection, or coal leakage detection, which require multi-area collaborative detection. This invention does not limit the application scenarios; however, to better understand this invention, the coal leakage detection scenario is used as an example to explain the technical solution. Technical solutions for other application scenarios can be derived by analogy and will not be elaborated upon here.

[0053] First, this invention provides a method for identifying abnormal states. Please refer to [link / reference needed]. Figure 1 , Figure 1 A schematic flowchart of an abnormal state identification method provided by the present invention is shown. Figure 1 As shown, the identification method may include the following steps:

[0054] Step 101: Input the image to be recognized into the first recognition model to obtain the first recognition result of the first detection area and the second detection area output by the first recognition model; the first recognition result is used to indicate whether the object in the first detection area is in an abnormal state; wherein, the first detection area is the desired area for placing the object, and the second detection area is the undesired area for placing the object.

[0055] The image to be identified includes image regions corresponding to a first detection region and a second detection region. The first detection region is the desired area where the object is placed; in a coal leakage detection scenario, this refers to the conveyor belt area used for transporting coal. The second detection region is the undesired area where the object is placed; in a coal leakage detection scenario, this refers to a non-conveyor belt area used for transporting coal (e.g., the ground or machine surface). The first identification result indicates whether the object in the first detection region is in an abnormal state; in a coal leakage detection scenario, this means that coal is missing or absent from the conveyor belt area.

[0056] The first recognition model is a target detection model used to perform target detection in the first and second detection regions, i.e., selecting the first and second detection regions in the image to be recognized. It then identifies the coal material within the first detection region, obtaining a first recognition result for that region. This first recognition result indicates whether the coal material in the first detection region is in an abnormal state, i.e., whether there is missing or no coal material in the conveyor belt area. If there is missing or no coal material, the object is determined to be in an abnormal state. If there is no missing or no coal material, the object is determined to be in a normal state.

[0057] It is worth noting that the first detection area is the conveyor belt area, while the second detection area is a non-conveyor belt area. The coal volume in the conveyor belt area is relatively large, making it easier to identify whether the coal is abnormal. However, the leaked coal volume in the non-conveyor belt area is smaller and more dispersed, resulting in a high false detection rate for leak detection. Therefore, the first recognition model in this invention only recognizes the first recognition result of the first detection area, without needing to recognize and detect the second detection area; only the second detection area needs to be selected. The second recognition model then performs refined recognition of the second detection area to improve detection accuracy.

[0058] The object detection model can use object detection networks such as SSD or YOLO. The feature extraction backbone network in the object detection model can be replaced with a lightweight MobileNet network.

[0059] As an optional embodiment of the present invention, after step 101, the method further includes: if it is determined from the first recognition result that the object in the first detection area is in a normal state, then it is determined that the object in the image to be recognized is in a normal state. That is, steps 102 to 103 do not need to be executed.

[0060] If it is determined from the first recognition result that the object in the first detection area is in an abnormal state, then steps 102 to 103 are executed, and the execution logic is as follows:

[0061] Step 102: If it is determined that the object in the first detection area is in an abnormal state based on the first recognition result, then the first cropped image corresponding to the second detection area is input into the second recognition model to obtain the second recognition result output by the second recognition model; the second recognition result is used to indicate whether the object exists in the second detection area.

[0062] When the objects in the first detection area are in a normal state, it indicates that the coal on the conveyor belt is relatively intact, and there is no need to perform coal leakage detection. Therefore, to avoid unnecessary computation, this invention determines whether to perform coal leakage detection using the second recognition model based on the state of the objects in the first detection area before performing coal leakage detection using the second recognition model.

[0063] The second recognition model is used to segment the image of coal leakage in the second detection area, thereby identifying whether coal leakage exists in the second detection area. The second recognition model can be a DeepLab network model or an ENet network model, etc. The second recognition model can both compensate for the weakness of the first recognition model in detecting small, scattered coal targets, and can perform fine-grained segmentation and confirmation of the scattered coal area (second detection area), reducing the overall false detection rate.

[0064] The second detection area is selected by a rectangle. When cropping the first cropped image, it is only necessary to obtain the position coordinates of the upper left and lower right corners of the rectangle to locate the second detection area and then crop the first cropped image.

[0065] Step 103: Based on the first recognition result and the second recognition result, determine whether the object in the image to be recognized is in an abnormal state.

[0066] Since the first recognition result is for the transmission zone region, while the second recognition result is for the non-transmission zone region, this invention aims to improve recognition accuracy by comprehensively judging whether the object in the image to be recognized is in an abnormal state based on both the first and second recognition results.

[0067] Specifically, step 103 includes steps 1031 to 1032. For example... Figure 2 As shown, Figure 2 The diagram shows a schematic flowchart of step 103 in an abnormal state identification method provided by the present invention.

[0068] Step 1031: If it is determined that the object in the first detection area is in an abnormal state according to the first recognition result, and the object is determined to exist in the second detection area according to the second recognition result, then it is determined that the object in the image to be recognized is in an abnormal state.

[0069] When there is a shortage or absence of coal in the conveyor belt area (i.e., the object in the first detection area is in an abnormal state), and there is coal in the non-conveyor belt area, it is determined that the coal in the image to be identified is in an abnormal state (i.e., there is a coal leak).

[0070] Step 1032: If it is determined from the first recognition result that the object in the first detection area is in an abnormal state, and it is determined from the second recognition result that the object does not exist in the second detection area, then it is determined that the object in the image to be recognized is in a normal state.

[0071] When there is a shortage or absence of coal in the conveyor belt area (i.e., the object in the first detection area is in an abnormal state), and there is no coal in the non-conveyor belt area, the coal in the image to be identified is determined to be in a normal state. It is understood that this state may be caused by insufficient loading or omissions during coal loading and unloading, rather than a coal leak. Therefore, this invention uses a combination of the first and second identification results to improve detection accuracy.

[0072] As an optional embodiment of the present invention, when it is determined that an object in the image to be identified is in an abnormal state, the coordinate position of the second detection area is output to better guide the user to handle the coal leakage.

[0073] To better understand the technical solution of this embodiment, it is combined with... Figure 3 The technical solution of this embodiment of the present invention will be described in its entirety. Please refer to [link / reference]. Figure 3 , Figure 3 This diagram illustrates the overall flow of an abnormal state identification method provided by the present invention. An inspection image or video (i.e., the image to be identified) is input into a target detection model (i.e., the first identification model). The target detection model outputs a first identification result for the coal pile area (first detection area) (the first identification result includes whether coal leakage has occurred or not) and a scattered coal area (i.e., the second detection area). The scattered coal area is input into an image segmentation model (i.e., the second identification model) to obtain a second identification result (the second identification result includes whether there is scattered coal or no scattered coal). Based on the first and second identification results, a comprehensive determination is made as to whether coal leakage has occurred.

[0074] In this embodiment, the image to be identified is input into a first recognition model to obtain a first recognition result and a second detection region output by the first recognition model. The first recognition result indicates whether the object in the first detection region is in an abnormal state. The first detection region is the desired area for placing the object, and the second detection region is the undesired area for placing the object. If the object in the first detection region is determined to be in an abnormal state based on the first recognition result, the first cropped image corresponding to the second detection region is input into a second recognition model to obtain a second recognition result output by the second recognition model. The second recognition result indicates whether the object exists in the second detection region. Based on the first and second recognition results, it is determined whether the object in the image to be identified is in an abnormal state. Because the above scheme fully considers the characteristics of the coal leakage detection scenario, coal leakage is detected in zones, and each zone uses a different recognition model. For the first detection region, the first recognition model is used to identify whether the object in the first detection region is in an abnormal state. Furthermore, the second recognition model is used to identify whether there is an object in the second detection area only when the object in the first detection area is in an abnormal state. Then, based on the recognition results output by the first and second recognition models, it is determined whether the object in the image to be recognized is in an abnormal state, thus making the scheme have high detection accuracy.

[0075] Optionally, steps 104 to 110 are included before step 103. See also... Figure 4 , Figure 4 A schematic flowchart of another abnormal state identification method provided by the present invention is shown.

[0076] Step 104: Obtain training sample images and corresponding annotation data for the training sample images; the annotation data includes the first training result of the first training region, the second training region, and the final training result.

[0077] Training sample images are obtained from a public database, and these images are manually labeled to obtain labeled data. The labeled data includes the first training result for a first training region, a second training region, and the final training result. The first training result for the first training region indicates whether coal leakage occurs in the conveyor belt area; the second training region refers to the non-conveyor belt area; and the final training result indicates whether coal leakage exists in the training sample image. It can be seen that the labeled data in this embodiment has two levels: the first level of labeling consists of the first training result and the second training region, and this first-level labeling is used to train the first recognition model. The second-level labeling is the final training result, used to train the second recognition model.

[0078] In particular, due to the limited number of training sample images in coal leakage detection scenarios, data-driven deep learning models may overfit, resulting in excellent detection performance on the validation set but failing to achieve the same level of performance in field applications. To avoid this situation, this embodiment provides a method for generating training sample images, as follows:

[0079] Specifically, step 104 includes steps 1041 to 1043. For example... Figure 5 As shown, Figure 5 The diagram shows a schematic flowchart of step 103 in an abnormal state identification method provided by the present invention.

[0080] Step 1041: Obtain multiple initial images.

[0081] The initial image refers to an image obtained from a public database.

[0082] Step 1042: Perform transformation processing on each of the initial images to obtain multiple transformed images; the transformation processing includes optical deformation, geometric deformation, color transformation or simulated occlusion; wherein the number of transformed images is greater than the number of initial images.

[0083] This embodiment combines two generation strategies to jointly generate training sample images. First, each initial image is transformed to obtain multiple transformed images. Transformation processing includes, but is not limited to, optical deformation, geometric deformation, color transformation, or a combination of one or more methods such as simulated occlusion. A single initial image can generate multiple transformed images after transformation processing. Then, the second generation strategy is implemented in step 1043.

[0084] Step 1043: Input the transformed image into a generative adversarial network to obtain the training sample image output by the generative adversarial network; wherein the number of training sample images is greater than the number of transformed images.

[0085] Before generating training sample images using a Generative Adversarial Network (GAN), the GAN needs to be trained beforehand. The training process is as follows: The generator in the initial GAN ​​autonomously generates sample images, which are then input along with the initial image into the discriminator in the GAN. The differences between the autonomously generated sample images and the initial images are evaluated, and the network parameters of the generator are further optimized. Through continuous adversarial interaction between the generator and the discriminator, the generator parameters are continuously optimized until the autonomously generated sample images and the initial images reach a certain level of similarity. At this point, the generator parameter adjustment stops, and the final GAN ​​is output.

[0086] Specifically, step 1043 includes steps A1 to A3. For example... Figure 6As shown, Figure 6 The diagram shows a schematic flowchart of step 103 in an abnormal state identification method provided by the present invention.

[0087] Step A1: Extract the image region containing the target object from the transformed image to obtain the third cropped image.

[0088] Step A2: Input the third cropped image into the generative adversarial network to obtain a simulated image output by the generative adversarial network.

[0089] To reduce unnecessary computation, this embodiment extracts the image region containing the target object from the transformed image, resulting in a third cropped image. Only this third cropped image needs to be input into the generative adversarial network to obtain a simulated image. This eliminates the need to process the entire transformed image, thus reducing computational load.

[0090] Step A3: Combine the simulated image with the transformed image to obtain the training sample image.

[0091] The simulated image and the transformed image are randomly combined to obtain the training sample image.

[0092] To better understand steps 1042 to 1043, a general explanation of steps 1042 to 1043 is provided here in conjunction with the accompanying drawings. Please refer to... Figure 7 , Figure 7 A schematic diagram illustrating the generation process of training sample images provided by this invention is shown. Figure 7 As shown, firstly, an image of coal leakage at the production site (initial image) is acquired. A GAN network is then trained using this image to obtain the trained GAN network. A simulated image is then generated using the GAN network. Next, the coal leakage image is transformed (optical deformation, geometric deformation, color transformation, and target occlusion) to obtain a transformed image. Finally, the transformed image is combined with the simulated image to obtain the result data (training sample images). The training sample images are then manually labeled to obtain labeled data.

[0093] Step 105: Input the first training sample image into the first initial model to obtain the first initial result and the second initial region output by the first initial model; the first initial result is used to indicate whether the first initial region is in an abnormal state; wherein, the first initial region and the second initial region are both image regions in the image to be identified.

[0094] The first training sample image includes image regions corresponding to a first initial region and a second initial region. The first initial region refers to the desired area for placing the object; in a coal leakage detection scenario, this refers to the conveyor belt area used for transporting coal. The second initial region is the undesired area for placing the object; in a coal leakage detection scenario, this refers to a non-conveyor belt area used for transporting coal (e.g., the ground or machine surface). The first initial result indicates whether the object in the first initial region is in an abnormal state; in a coal leakage detection scenario, this means that coal is missing or absent in the conveyor belt area.

[0095] Step 106: Calculate the first loss value between the first initial result and the first training result.

[0096] Step 107: Calculate the second loss value between the second initial region and the second training region.

[0097] Step 108: Adjust the first initial model based on the first loss value and the second loss value.

[0098] Step 109: Using other training sample images as the first training sample images, repeatedly execute the steps of inputting the first training sample images into the first initial model to obtain the first initial result of the first initial region and the second initial region output by the first initial model, and subsequent steps, to obtain the first recognition model.

[0099] For each training sample image, steps 105 to 109 are executed sequentially to continuously adjust the network parameters of the first initial model until the recognition accuracy of the first initial model converges or the maximum number of training iterations is reached.

[0100] Step 110: For each of the second cropped images, perform the following steps to obtain the second recognition model:

[0101] Step 1101: Input the second cropped image corresponding to the second training region into the second initial model to obtain the second initial result output by the second initial model.

[0102] Step 1102: Calculate the third loss value between the second initial result and the final training result.

[0103] Step 1103: Adjust the second initial model according to the third loss value.

[0104] It can be seen that the first and second initial models are jointly trained, and the input of the second initial model is the second cropped image corresponding to the second training region output by the first initial model. This enables the first and second initial models to have good detection accuracy in the application stage.

[0105] In this embodiment, the first initial model and the second initial model are jointly trained using two levels of labeled data, so that the first initial model and the second initial model have good detection accuracy in the application stage.

[0106] like Figure 8 This invention provides an abnormal state identification device 8, please refer to [link / reference]. Figure 8 , Figure 8 A schematic diagram of an abnormal state identification device provided by the present invention is shown, such as... Figure 8 An abnormal state identification device shown includes:

[0107] Processing unit 81 is used to input the image to be recognized into a first recognition model to obtain a first recognition result of a first detection area and a second detection area output by the first recognition model; the first recognition result is used to indicate whether the object in the first detection area is in an abnormal state; wherein, the first detection area is the desired area for placing the object, and the second detection area is the undesired area for placing the object;

[0108] The judgment unit 82 is configured to, if determined according to the first recognition result that the object in the first detection area is in an abnormal state, input the first cropped image corresponding to the second detection area into the second recognition model to obtain a second recognition result output by the second recognition model; the second recognition result is used to indicate whether the object exists in the second detection area;

[0109] The determining unit 83 is used to determine whether the object in the image to be identified is in an abnormal state based on the first identification result and the second identification result.

[0110] This invention provides an abnormal state identification device. An image to be identified is input into a first identification model to obtain a first identification result for a first detection area and a second detection area output by the first identification model. The first identification result indicates whether an object in the first detection area is in an abnormal state. The first detection area is the desired area for placing the object, and the second detection area is the undesired area for placing the object. If the first identification result determines that an object in the first detection area is in an abnormal state, a first cropped image corresponding to the second detection area is input into a second identification model to obtain a second identification result output by the second identification model. The second identification result indicates whether the object exists in the second detection area. Based on the first and second identification results, it is determined whether the object in the image to be identified is in an abnormal state. This solution fully considers the characteristics of coal leakage detection scenarios, performing zoned detection of coal leakage, and using a different identification model for each area. For the first detection area, the first identification model is used to identify whether the object in the first detection area is in an abnormal state. Furthermore, the second recognition model is used to identify whether there is an object in the second detection area only when the object in the first detection area is in an abnormal state. Then, based on the recognition results output by the first and second recognition models, it is determined whether the object in the image to be recognized is in an abnormal state, thus making the scheme have high detection accuracy.

[0111] Figure 9 This is a schematic diagram of a terminal device provided in an embodiment of the present invention. Figure 9 As shown, a terminal device 100 of this embodiment includes: a processor 9, a memory 1001, and a computer program 1002 stored in the memory 1001 and executable on the processor 9, such as a method program for identifying abnormal states. When the processor 9 executes the computer program 1002, it implements the steps in the various embodiments of the method program for identifying abnormal states described above, for example... Figure 1 Steps 101 to 103 are shown. Alternatively, when the processor 9 executes the computer program 1002, it implements the functions of each unit in the above-described device embodiments, for example... Figure 8 The functions of units 81 to 83 are shown.

[0112] For example, the computer program 1002 can be divided into one or more units, which are stored in the memory 1001 and executed by the processor 9 to complete the present invention. The one or more units can be a series of computer program instruction segments capable of performing specific functions, which describe the execution process of the computer program 1002 in the terminal device 100. For example, the specific functions of each unit of the computer program 1002 can be divided as follows:

[0113] The processing unit is configured to input the image to be recognized into a first recognition model to obtain a first recognition result of a first detection region and a second detection region output by the first recognition model; the first recognition result is used to indicate whether the object in the first detection region is in an abnormal state; wherein, the first detection region is the desired region for placing the object, and the second detection region is the undesired region for placing the object;

[0114] The judgment unit is configured to, if determined based on the first recognition result that the object in the first detection area is in an abnormal state, input the first cropped image corresponding to the second detection area into the second recognition model to obtain a second recognition result output by the second recognition model; the second recognition result is used to indicate whether the object exists in the second detection area;

[0115] The determining unit is configured to determine whether an object in the image to be identified is in an abnormal state based on the first identification result and the second identification result.

[0116] The terminal device includes, but is not limited to, a processor 9 and a memory 1001. Those skilled in the art will understand that... Figure 9 This is merely an example of a terminal device 100 and does not constitute a limitation on a terminal device 100. It may include more or fewer components than shown, or combine certain components, or different components. For example, the terminal device may also include input / output devices, network access devices, buses, etc.

[0117] The processor 9 can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor.

[0118] The memory 1001 can be an internal storage unit of the terminal device 100, such as a hard disk or memory of the terminal device 100. The memory 1001 can also be an external storage device of the terminal device 100, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc., equipped on the terminal device 100. Furthermore, the memory 1001 can include both internal and external storage units of the terminal device 100. The memory 1001 is used to store the computer program and other programs and data required by the roaming control device. The memory 1001 can also be used to temporarily store data that has been output or will be output.

[0119] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0120] It should be noted that the information interaction and execution process between the above-mentioned devices / units are based on the same concept as the method embodiments of the present invention. For details on their specific functions and technical effects, please refer to the method embodiments section, which will not be repeated here.

[0121] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this invention. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0122] This invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps described in the various method embodiments above.

[0123] This invention provides a computer program product that, when run on a mobile terminal, enables the mobile terminal to implement the steps described in the above-described method embodiments.

[0124] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments of the present invention can be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include at least: any entity or device capable of carrying computer program code to a photographic device / terminal device, a recording medium, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium. Examples include USB flash drives, portable hard drives, magnetic disks, or optical disks. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electrical carrier signals or telecommunication signals.

[0125] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0126] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0127] In the embodiments provided by this invention, it should be understood that the disclosed apparatus / network devices and methods can be implemented in other ways. For example, the apparatus / network device embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0128] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; they may be located in one place or distributed across multiple network units.

[0129] It should be understood that, when used in this specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or collections thereof.

[0130] It should also be understood that the term “and / or” as used in this specification and the appended claims refers to any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0131] As used in this specification and the appended claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection." Similarly, the phrase "if determined" or "if [the described condition or event] is detected" may be interpreted, depending on the context, as meaning "once determined," "in response to determination," "once [the described condition or event] is detected," or "in response to detection of [the described condition or event]."

[0132] Furthermore, in the description of this invention and the appended claims, the terms "first," "second," "third," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0133] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of the invention include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including, but not limited to," unless otherwise specifically emphasized.

[0134] The above-described embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.

Claims

1. A method for identifying abnormal states, characterized in that, The identification method includes: The image to be recognized is input into the first recognition model to obtain a first recognition result of the first detection area and a second detection area output by the first recognition model; the first recognition result is used to indicate whether the object in the first detection area is in an abnormal state; wherein, the first detection area is the desired area for placing the object, and the second detection area is the undesired area for placing the object; If it is determined that the object in the first detection area is in an abnormal state based on the first recognition result, then the first cropped image corresponding to the second detection area is input into the second recognition model to obtain the second recognition result output by the second recognition model; the second recognition result is used to indicate whether the object exists in the second detection area; Based on the first recognition result and the second recognition result, it is determined whether the object in the image to be recognized is in an abnormal state.

2. The identification method as described in claim 1, characterized in that, The step of determining whether an object in the image to be identified is in an abnormal state based on the first recognition result and the second recognition result includes: If it is determined that the object in the first detection area is in an abnormal state based on the first recognition result, and it is determined that the object exists in the second detection area based on the second recognition result, then it is determined that the object in the image to be recognized is in an abnormal state. If the object in the first detection area is determined to be in an abnormal state based on the first recognition result, and the object is determined not to exist in the second detection area based on the second recognition result, then the object in the image to be recognized is determined to be in a normal state.

3. The identification method as described in claim 1, characterized in that, After the step of inputting the image to be recognized into the first recognition model and obtaining the first recognition result of the first detection region and the second detection region output by the first recognition model, the method further includes: If the object in the first detection area is determined to be in a normal state based on the first recognition result, then the object in the image to be recognized is determined to be in a normal state.

4. The identification method as described in claim 1, characterized in that, Before the step of inputting the image to be recognized into the first recognition model to obtain the first recognition result of the first detection region and the second detection region output by the first recognition model, the method further includes: Acquire training sample images and corresponding annotation data for the training sample images; the annotation data includes a first training result for a first training region, a second training region, and a final training result. The first training sample image is input into the first initial model to obtain a first initial result and a second initial region output by the first initial model; the first initial result is used to indicate whether the first initial region is in an abnormal state; wherein, the first initial region and the second initial region are both image regions in the image to be identified; Calculate the first loss value between the first initial result and the first training result; Calculate the second loss value between the second initial region and the second training region; Adjust the first initial model based on the first loss value and the second loss value; Using other training sample images as the first training sample images, the steps of inputting the first training sample images into the first initial model, obtaining the first initial result of the first initial region and the second initial region output by the first initial model, and subsequent steps are performed repeatedly to obtain the first recognition model.

5. The identification method as described in claim 4, characterized in that, The identification method further includes: For each second cropped image, perform the following steps to obtain the second recognition model: The second cropped image corresponding to the second training region is input into the second initial model to obtain the second initial result output by the second initial model. Calculate a third loss value between the second initial result and the final training result; The second initial model is adjusted based on the third loss value.

6. The identification method as described in claim 4, characterized in that, The acquisition of training sample images and corresponding annotation data includes: Acquire multiple initial images; Each initial image is transformed to obtain multiple transformed images; the transformation process includes optical deformation, geometric deformation, color transformation, or simulated occlusion; wherein the number of transformed images is greater than the number of initial images; The transformed image is input into a generative adversarial network (GAN) to obtain the training sample image output by the GAN; wherein the number of training sample images is greater than the number of transformed images.

7. The identification method as described in claim 6, characterized in that, The step of inputting the transformed image into a generative adversarial network (GAN) to obtain the training sample image output by the GAN includes: The image region containing the target object is cropped from the transformed image to obtain a third cropped image; The third cropped image is input into the generative adversarial network to obtain a simulated image output by the generative adversarial network; The simulated image is combined with the transformed image to obtain the training sample image.

8. An abnormal state identification device, characterized in that, The identification device includes: The processing unit is configured to input the image to be recognized into a first recognition model to obtain a first recognition result of a first detection region and a second detection region output by the first recognition model; the first recognition result is used to indicate whether the object in the first detection region is in an abnormal state; wherein, the first detection region is the desired region for placing the object, and the second detection region is the undesired region for placing the object; The judgment unit is configured to, if determined based on the first recognition result that the object in the first detection area is in an abnormal state, input the first cropped image corresponding to the second detection area into the second recognition model to obtain a second recognition result output by the second recognition model; the second recognition result is used to indicate whether the object exists in the second detection area; The determining unit is configured to determine whether an object in the image to be identified is in an abnormal state based on the first identification result and the second identification result.

9. A terminal device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the method as described in any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method as described in any one of claims 1 to 7.

Images