Target detection method, electronic device and computer readable storage medium

Abstract

The application discloses a target detection method, an electronic device and a computer readable storage medium, wherein the method comprises: performing feature extraction on a to-be-detected image to obtain a detection image feature of the to-be-detected image; matching the detection image feature of the to-be-detected image with a false detection image feature contained in a false detection image feature library, wherein the false detection image feature is extracted from a false detection image whose target detection result is inconsistent with a target annotation result; in response to the false detection image feature library not containing a false detection image feature matched with the detection image feature, performing target classification by using the detection image feature; and in response to a classification result of the target classification being inconsistent with a target detection result of the to-be-detected image and the classification result of the target classification being consistent with a target annotation result of the false detection image, adding the detection image feature to the false detection image feature library. In the foregoing manner, the application can improve the accuracy of target detection.

Description

Technical Field

[0001] This application relates to the field of machine vision technology, and in particular to a target detection method, electronic device, and computer-readable storage medium. Background Technology

[0002] Object detection is a popular area of ​​computer vision and digital image processing, widely used in fields such as robot navigation, intelligent video surveillance, industrial inspection, and aerospace. Reducing the reliance on human capital through computer vision has significant practical implications. However, the accuracy of object detection is currently still relatively low. Summary of the Invention

[0003] The main technical problem addressed by this application is to provide a target detection method, electronic device, and computer-readable storage medium that can improve the accuracy of target detection.

[0004] To address the aforementioned technical problems, the first aspect of this application provides a target detection method, comprising: extracting features from an image to be detected to obtain detection image features of the image to be detected; matching the detection image features of the image to be detected with false detection image features contained in a false detection image feature library, wherein the false detection image features are extracted from false detection images where the target detection result and the target annotation result are inconsistent; in response to the absence of a false detection image feature in the false detection image feature library that matches the detection image features, classifying the target using the detection image features; and in response to the discrepancy between the classification result of the target classification and the target detection result of the image to be detected, and the consistency between the classification result of the target classification and the target annotation result of the false detection image, adding the detection image features to the false detection image feature library.

[0005] To address the aforementioned technical problems, a second aspect of this application provides an electronic device comprising a memory and a processor coupled to each other, wherein the memory is used to store program data and the processor is used to execute the program data to implement the aforementioned method.

[0006] To address the aforementioned technical problems, a third aspect of this application provides a computer-readable storage medium storing program data, which, when executed by a processor, is used to implement the aforementioned method.

[0007] The beneficial effects of this application are as follows: Unlike existing technologies, this application extracts features from the image to be detected to obtain detection image features. Then, it matches these detection image features with false detection image features contained in a false detection image feature library. False detection image features are extracted from false detection images where the target detection result and target annotation result are inconsistent. Then, in response to the absence of a matching false detection image feature in the false detection image feature library, target classification is performed using the detection image features. Finally, in response to the discrepancy between the target classification result and the target detection result of the image to be detected, and the discrepancy between the target classification result and the false detection image... If the target annotation results are consistent, the detected image features are added to the false detection image feature library. Thus, this application establishes a false detection image feature library to address the false detection problem of images. The false detection image feature library is used to identify false detections of images, in order to further determine whether the image to be detected has been falsely detected in target detection. Furthermore, when there are no false detection image features in the false detection image feature library that match the detected image features, the image is further identified through target classification, and the detected image features corresponding to the image to be detected whose target detection results are consistent with the target annotation results are added to the false detection image feature library, so as to continuously optimize the false detection discrimination capability of the false detection image feature library and further improve the accuracy of target detection. Attached Figure Description

[0008] To more clearly illustrate the technical solutions in this application, the accompanying drawings required in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. Among them:

[0009] Figure 1 This is a flowchart illustrating an embodiment of the target detection method of this application;

[0010] Figure 2 This is a flowchart illustrating another embodiment of the target detection method of this application;

[0011] Figure 3 This is a flowchart illustrating yet another embodiment of the target detection method of this application;

[0012] Figure 4 This is a schematic diagram of the feature matching model of this application;

[0013] Figure 5 This is a flowchart illustrating another embodiment of the target detection method of this application;

[0014] Figure 6 yes Figure 5 A flowchart illustrating an implementation method for step S44;

[0015] Figure 7 yes Figure 6 A flowchart illustrating an implementation method for step S443;

[0016] Figure 8 yes Figure 5 A flowchart illustrating another embodiment of step S44;

[0017] Figure 9 This is a flowchart illustrating yet another embodiment of the target detection method of this application;

[0018] Figure 10 This is a schematic block diagram of an embodiment of the target detection device of this application;

[0019] Figure 11 This is a schematic block diagram of the structure of an embodiment of the electronic device of this application;

[0020] Figure 12 This is a schematic block diagram of an embodiment of a computer-readable storage medium of this application. Detailed Implementation

[0021] In this application, the reference to "embodiment" means that a specific feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0022] The terms "first" and "second" in this application are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this application, "multiple" means at least two, such as two, three, etc., unless otherwise explicitly specified. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to such processes, methods, products, or apparatus.

[0023] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of the embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0024] Please see Figure 1 , Figure 1 This is a schematic flowchart of an embodiment of the target detection method of this application. The implementing entities of this application include target detection devices, electronic devices, etc. The following description uses a target detection device as an example.

[0025] The method may include the following steps:

[0026] Step S11: Extract features from the image to be detected to obtain the detection image features.

[0027] Feature extraction networks can extract features from an image to obtain the corresponding feature vectors. In some implementations, feature extraction networks can be used to extract features from the image to be detected, obtaining the detection image features. The type, structure, and other parameters of the feature extraction network are not limited and can be set according to the actual situation.

[0028] Step S12: Match the detected image features of the image to be detected with the false detection image features contained in the false detection image feature library. The false detection image features are extracted from false detection images where the target detection result and the target annotation result are inconsistent.

[0029] The object detection result is obtained by performing object detection on the image, while the object annotation result is the true result of the image. The object detection result may match the object annotation result, in which case the object detection detected the true result of the image. Conversely, the object detection result may not match the object annotation result, in which case the object detection did not detect the true result of the image, i.e., a false detection occurs.

[0030] Target annotation results can be obtained manually or automatically by equipment. In one example, target annotation results can be obtained by the user manually annotating a batch of images, with one target annotation result corresponding to one image. Then, target detection is performed on this batch of images to obtain the target detection result for each image. If the target detection result of an image is inconsistent with the target annotation result, the image can be regarded as a false detection image. Then, the false detection image features are extracted and added to the false detection image feature library to obtain an initial false detection image feature library. In another example, the target annotation result can be the classification result of the image after target classification. If the classification result of an image is inconsistent with the target detection result of the image, the classification result of the image can be used as the target label result of the image, and the image can be regarded as a false detection image. The detection image features of the image can be added to the false detection image feature library as false detection image features to update the initial false detection image feature library and continuously optimize the false detection discrimination capability of the false detection image feature library. For details, please refer to the following embodiments.

[0031] In some implementations, the target labeling result of the image to be detected is A, but the target detection result after target detection is not A, such as B, C, D, etc., which means that the target detection result of the image to be detected is inconsistent with the target labeling result, indicating that the image to be detected is a false detection image.

[0032] Step S13: In response to the absence of a false detection image feature matching the detected image feature in the false detection image feature library, target classification is performed using the detected image feature.

[0033] If no false detection image feature matches the detected image feature in the false detection image feature library, it means that the image to be detected does not belong to any false detection cases currently included in the false detection image feature library. However, the image to be detected may belong to other false detection cases outside the false detection image feature library. Therefore, this embodiment further performs target classification on the image to be detected to determine whether the image to be detected belongs to other false detection cases outside the false detection image feature library. If so, the detected image feature of the image to be detected is added to the false detection image feature library.

[0034] On the other hand, if there are false detection image features in the false detection image feature library that match the features of the detected image, it means that the image to be detected belongs to the false detection cases included in the current false detection image feature library. This indicates that the target detection result of the image to be detected is inaccurate and there is a false detection. In fact, the true target detection result of the image to be detected should be consistent with the target annotation result. Therefore, the target annotation result can be used as the target detection result of the image to be detected in order to correct the erroneous target detection result and improve the accuracy of target detection.

[0035] Step S14: In response to the discrepancy between the target classification result and the target detection result of the image to be detected, and the consistency between the target classification result and the target annotation result of the false detection image, the features of the detected image are added to the false detection image feature library.

[0036] At this point, it indicates that the image to be detected was falsely detected during the target detection process. In this case, the features of the detected image can be added to the false detection image feature library to update the false detection image feature library.

[0037] The above scheme extracts features from the image to be detected to obtain the detection image features. Then, it matches these detection image features with false positive image features in a false positive image feature library. False positive image features are extracted from false positive images where the target detection result and target annotation result are inconsistent. If no matching false positive image feature exists in the false positive image feature library, the detection image features are used for target classification. Finally, if the target classification result is inconsistent with the target detection result of the image to be detected, but consistent with the target annotation result of the false positive image, the scheme proceeds smoothly. By adding the detected image features to the false detection image feature library, this application establishes a false detection image feature library to address the false detection problem of images. The false detection image feature library is used to identify false detections of images, thereby further confirming that the image to be detected has been falsely detected in target detection. Furthermore, when there are no false detection image features in the false detection image feature library that match the detected image features, the image is further identified through target classification. The detected image features corresponding to the image to be detected whose target detection results are consistent with the target annotation results are added to the false detection image feature library to continuously optimize the false detection identification capability of the false detection image feature library and further improve the accuracy of target detection.

[0038] In some implementations, before performing feature extraction on the image to be detected to obtain the detection image features, target detection is first performed on the image to be detected. Then, based on the obtained target detection results, it is determined whether to further perform false detection judgment on the image to be detected. For details, please refer to the following two embodiments.

[0039] Please see Figure 2 , Figure 2 This is a flowchart illustrating another embodiment of the target detection method of this application.

[0040] In this embodiment, the target annotation result indicates the presence of a target in the falsely detected image. The target detection method specifically includes the following steps:

[0041] Step S21: Perform target detection on the image to be detected to obtain the target detection result.

[0042] This can be achieved by employing appropriate object detection algorithms to detect objects in the image to be detected, thereby obtaining the object detection results. No specific limitations are specified here. For example, an object detection algorithm based on a convolutional neural network (CNN) can be used.

[0043] Step S22: In response to the target detection result indicating that there is no target in the image to be detected, feature extraction is performed on the image to be detected to obtain the detection image features of the image to be detected.

[0044] Since the target annotation result indicates that the target exists in the false detection image, if the target detection result indicates that the target does not exist in the image to be detected, it means that the target detection result is inconsistent with the target annotation result. Therefore, it can be further determined whether the detection image features of the image to be detected are already included in the false detection image feature library. If so, there is no need to add the detection image features of the image to be detected to the false detection image feature library. Otherwise, the detection image features of the image to be detected can be added to the false detection image feature library to continuously optimize the false detection discrimination capability of the false detection image feature library.

[0045] Step S23: Match the detected image features of the image to be detected with the false detection image features contained in the false detection image feature library. The false detection image features are extracted from the false detection images where the target detection results and target annotation results are inconsistent. The target annotation results indicate that there is a target in the false detection image.

[0046] Step S24: In response to the absence of a false detection image feature matching the detection image feature in the false detection image feature library, target classification is performed using the detection image feature.

[0047] For a description of steps S22 to S24, please refer to the aforementioned embodiments, which will not be repeated here.

[0048] Step S25: In response to the target detection result of the image to be detected indicating that there is no target in the image to be detected and the classification result of the target classification indicating that there is a target in the image to be detected, add the detected image features to the false detection image feature library.

[0049] Among them, the target detection result of the image to be detected indicates that there is no target in the image to be detected, and the classification result of the target classification indicates that there is a target in the image to be detected. This indicates that the classification result of the image to be detected is inconsistent with the target detection result, but consistent with the target annotation result. This indicates that the target detection result of the image to be detected is inconsistent with the target annotation result, that is, the image to be detected is a false detection image. Then, the detection image features of the image to be detected can be added to the false detection image feature library to continuously optimize the false detection discrimination capability of the false detection image feature library.

[0050] Optionally, the target in this application can be any object, and can be selected according to actual needs. For example, the target can be a person, animal, fireworks, object, etc.

[0051] Please see Figure 3 , Figure 3 This is a flowchart illustrating another embodiment of the target detection method of this application. Unlike the previous embodiments, the target annotation result indicates that the target does not exist in the falsely detected image. This target detection method specifically includes the following steps:

[0052] Step S31: Perform target detection on the image to be detected to obtain the target detection result.

[0053] Step S32: In response to the target detection result indicating the presence of a target in the image to be detected, feature extraction is performed on the image to be detected to obtain the detection image features of the image to be detected.

[0054] Since the target annotation result indicates that the target does not exist in the false detection image, if the target detection result indicates that the target exists in the image to be detected, it means that the target detection result is inconsistent with the target annotation result. Therefore, it can be further determined whether the detection image features of the image to be detected are already included in the false detection image feature library. If so, there is no need to add the detection image features of the image to be detected to the false detection image feature library. Otherwise, the detection image features of the image to be detected can be added to the false detection image feature library to continuously optimize the false detection discrimination capability of the false detection image feature library.

[0055] Step S33: Match the detected image features of the image to be detected with the false detection image features contained in the false detection image feature library. The false detection image features are extracted from the false detection images where the target detection results and target annotation results are inconsistent. The target annotation results indicate that the target does not exist in the false detection image.

[0056] Step S34: In response to the absence of a false detection image feature matching the detected image feature in the false detection image feature library, target classification is performed using the detected image feature.

[0057] For a description of steps S32 to S34, please refer to the aforementioned embodiments, which will not be repeated here.

[0058] Step S35: In response to the target detection result of the image to be detected indicating that there is a target in the image to be detected and the classification result of the target classification indicating that there is no target in the image to be detected, add the detected image features to the false detection image feature library.

[0059] Among them, the target detection result of the image to be detected indicates that there is a target in the image to be detected, and the classification result of the target classification indicates that there is no target in the image to be detected. This indicates that the target detection result of the image to be detected is inconsistent with the target annotation result, that is, the image to be detected is a false detection image. Then, the detection image features of the image to be detected can be added to the false detection image feature library to continuously optimize the false detection discrimination capability of the false detection image feature library.

[0060] In some embodiments, the image to be detected may be acquired from a monitored area. Specifically, before feature extraction of the image to be detected, an image to be detected from the monitored area may also be acquired. Optionally, the monitored area may include, but is not limited to, construction site areas, industrial park areas, warehouse areas, underground mine areas, and other areas where fires may occur. The image to be detected is, for example, a visible light image, i.e., an RGB image. The mine may be used to mine one of the following minerals: coal, metallic minerals, non-metallic minerals, building material minerals, and chemical minerals. In one example, the monitored area is an underground coal mine area. As one of the main sources of energy in my country, the supply of coal is crucial to the development of my country's national economy. Currently, most of my country's coal resources come from underground coal mining. However, underground coal mines suffer from severely limited production conditions due to their underground location, and the presence of numerous ignition sources and accumulated combustibles in the working space makes them prone to fire accidents during production. Furthermore, once a fire occurs in a coal mine, it can easily trigger secondary combustion and explosions, severely impacting the mine's production efficiency and potentially causing casualties. Furthermore, the vast underground area of ​​coal mines, coupled with the effects of coal dust and moisture, easily causes smoke detectors to malfunction, leading to untimely and inaccurate fire detection and ultimately serious accidents. The complex underground environment also results in a high false detection rate for vision-based smoke and fire detection.

[0061] In some implementations, an image acquisition device is installed in the monitored area to acquire images of the monitored area and obtain images to be detected. The image acquisition device and the target detection device can communicate (wired or wirelessly), and the target detection device can acquire the images to be detected of the monitored area from the image acquisition device. The image acquisition device includes a camera.

[0062] In some implementations, the target detection device may include a camera, so that the target detection device can directly acquire images of the monitored area to obtain the image to be detected.

[0063] In some implementations, the image to be detected can be an image frame extracted from video or an image captured individually. Specifically, an image can be captured from the monitored area at preset time intervals, or video data can be captured from the monitored area in real time, resulting in video data including multiple frames of the image to be detected.

[0064] Among them, the parameters of the camera can be adjusted during the installation of the image acquisition device and the target detection device to ensure that the acquired image is clear and free of obvious noise.

[0065] In some implementations, a comparison network can be used to match the detected image features of the image to be detected with falsely detected image features contained in a falsely detected image feature library. The comparison network is used to compare the similarity between two feature vectors. Specifically, at least one falsely detected image feature from the falsely detected image feature library and the detected image features of the image to be detected can be input into the comparison network to obtain the similarity between the detected image features and each falsely detected image feature. Further, if the similarity between the detected image features and each falsely detected image feature meets a preset similarity requirement, it is determined that a falsely detected image feature matching the detected image features exists in the falsely detected image feature library; otherwise, it is determined that no falsely detected image feature matching the detected image features exists in the falsely detected image feature library. In some implementations, the similarity value ranges from 0 to 1. If the similarity is greater than a preset threshold (e.g., 0.8, 0.9, 0.95, etc.), the match is determined to be successful; otherwise, the match is determined to be unsuccessful.

[0066] Please see Figure 4 , Figure 4 A schematic diagram of the feature matching model of this application.

[0067] In some implementations, the image to be detected can be directly matched with at least one falsely detected image from a falsely detected image library. Specifically, the image to be detected and the falsely detected image can be input into a pre-trained feature matching model. The feature matching model includes a feature extraction network and a comparison network. The feature extraction network includes two sub-networks (sub-network 1 and sub-network 2), each receiving one input. For example, sub-network 1 receives the image to be detected, and sub-network 2 receives the falsely detected image. Then, it outputs the representations embedded in a high-dimensional space, i.e., the features of the detected image and the features of the falsely detected image. Then, the comparison network determines whether the features of the detected image match the features of the falsely detected image to obtain the matching result. The comparison network is specifically used to calculate the distance (e.g., Euclidean distance) between the features of the detected image and the features of the falsely detected image. If the distance between the features of the detected image and the features of the falsely detected image meets the preset distance requirement, it is determined that the features of the detected image and the features of the falsely detected image match, that is, there is a falsely detected image in the falsely detected image that matches the image to be detected. Otherwise, it is determined that the features of the detected image and the features of the falsely detected image do not match, that is, there is no falsely detected image in the falsely detected image that matches the image to be detected. In this way, the similarity between the image to be detected and the falsely detected image can be compared.

[0068] In one example, the feature matching model can be a Siamese neural network. Training the feature matching model can include: establishing an initial image sample database, then training the Siamese neural network model based on the false positive image database, and selecting the optimal training result as the feature matching model.

[0069] Fire brings light and warmth to people, but it also brings much suffering and sorrow. Fire prevention is a crucial task in firefighting. Detecting smoke and fire in the early stages of a fire, providing early warning, can minimize the damage. Currently, smoke detectors are the most widely used method for smoke and fire detection, but these devices must be installed in non-large, enclosed indoor spaces or near the object being detected. In scenarios such as underground coal mines, timely and accurate detection is difficult. Additionally, while machine vision-based target detection methods exist, their accuracy is not high. Therefore, the target detection method of this application can be applied to the field of smoke and fire detection to improve its accuracy. Please refer to the following embodiments for details.

[0070] Please see Figures 5 to 8 , Figure 5 This is a flowchart illustrating another embodiment of the target detection method of this application. Figure 6 yes Figure 5 A flowchart illustrating an implementation method for step S44. Figure 7 yes Figure 6 A flowchart illustrating an implementation method for step S443. Figure 8 yes Figure 5 A flowchart illustrating another embodiment of step S44. In this embodiment, the target is fireworks, which may include at least one of fireworks and flames.

[0071] The method may include the following steps:

[0072] Step S41: Perform dehazing on the image to be detected.

[0073] In some implementations, dehazing algorithms can be used to dehaze the image to be detected. Dehazing algorithms can include, but are not limited to, those based on prior knowledge of dark channel color. Examples include Dr. Kaiming He's dark channel dehazing algorithm, a guided filtering-based dark channel dehazing algorithm, Fattal's single image dehazing algorithm, Tan's single image dehazing algorithm (Visibility in bad weather from a single image), Tarel's fast visibility restoration algorithm (Fast visibility restoration from a single color or gray level image), and Bayesian dehazing algorithms (Single image defogging by multiscale depth fusion). The basic principle of dehazing algorithms is based on atmospheric scattering models. The atmospheric scattering model can be used to explain the degradation process of foggy images as follows:

[0074]

[0075] Where I(x) is the foggy image, A represents the global atmospheric light value, t(x) is the scene transmittance, and J(x) is the defogging image.

[0076] Step S42: Perform image enhancement on the dehazed image to be detected.

[0077] Image enhancement algorithms can be used to enhance the dehazed image to be detected. Common image enhancement algorithms adjust brightness, contrast, saturation, and hue to increase clarity and reduce noise. These algorithms may include, but are not limited to, histogram equalization, Laplacian transformation, Logarithmic transformation, and gamma transform.

[0078] In some implementations, after dehazing, contrast enhancement algorithms (histogram equalization, gamma transform, etc.) can be used to process the image to be detected in order to improve the contrast between the target and the background.

[0079] In this embodiment, before feature extraction from the image to be detected, preprocessing (including dehazing and image enhancement) can be performed to make the target in the image clearer. Specifically, for the target being smoke and fire, dehazing is included in the image preprocessing to reduce the impact of fog in the environment on the smoke and fire target, thus improving the accuracy of target detection. Furthermore, in this embodiment, before feature extraction from the image to be detected, at least one of target detection, smoke movement trend detection, and multispectral image detection can be performed on the image to be detected.

[0080] Step S43: Perform target detection on the image to be detected to obtain the target detection result.

[0081] This involves using object detection algorithms to detect objects in the image to be detected, obtaining the object detection results. The object detection results indicate the presence of at least one of smoke or flame in the image to be detected, along with the corresponding location coordinates.

[0082] In some implementations, a deep learning object detection network can be pre-built and trained using collected sample materials. Specifically, visible light images of the monitored area can be acquired, and image data cleaning and data augmentation can be performed to construct a deep learning sample set. The trained object detection network uses optimal training weights. Then, the trained object detection network is used to detect objects in the image to be detected, identifying objects (e.g., smoke and / or flames) and recording their locations. The object detection network is not limited to a convolutional neural network (CNN).

[0083] Step S44: In response to the target detection result indicating the presence of smoke and / or flame in the image to be detected, feature extraction is performed on the image to be detected to obtain the detection image features of the image to be detected.

[0084] Specifically, if the target detection result indicates the presence of at least one of smoke or flame in the image to be detected, a step of feature extraction is performed on the image to be detected to obtain the detection image features. Thus, through target detection, false positives are only considered after at least one of smoke or flame is detected, thereby avoiding invalid identification.

[0085] In some implementations, step S44 may include steps S441 to S443:

[0086] Step S441: In response to the target detection result being smoke, acquire the smoke region image in the image to be detected.

[0087] Specifically, based on the location coordinates of the smoke in the target detection results, the region of interest (ROI) of the smoke in the image to be detected can be determined, i.e., the smoke region image. Then, the image can be cropped to obtain the smoke region image.

[0088] Step S442: Detect smoke movement trends using smoke region images.

[0089] In some implementations, when using optical flow to detect smoke movement trends, it is necessary to use two images of the monitored area taken at different times. In one example, video data of the monitored area can be acquired, and then images corresponding to the same position in the current frame and the previous or next frame can be extracted from the video data, denoted as smoke area image A1 and smoke area image A2.

[0090] Step S443: In response to the movement trend of the smoke, feature extraction is performed on the image to be detected to obtain the detection image features corresponding to the image to be detected.

[0091] If the smoke does not show any movement, it means the target is not smoke, and step S41 is repeated, i.e., the detection process restarts. If the smoke shows any movement, it meets the characteristics of smoke, and smoke can be detected, thus proceeding to the step of feature extraction of the image to be detected.

[0092] Alternatively, it can be further determined whether the smoke is water mist. If, after water mist identification, the target is still smoke, then the step of feature extraction of the image to be detected is performed. In some embodiments, step S443 may include steps S4431 to S4435:

[0093] Step S4431: In response to the movement trend of the smoke, acquire a multispectral image and a background image of the monitored area, wherein the multispectral image and the image to be detected are images acquired from the same monitored area within a time range, and the background image is an image acquired when the monitored area has not changed.

[0094] In this embodiment, a multispectral image is acquired simultaneously with the image to be detected. The background image can be pre-acquired and stored in the target detection device. In one example, the multispectral image corresponds to a wavelength range of 400–700 nm, with 50 nm intervals; this is not limited here.

[0095] Step S4432: Use the multispectral image and background image to determine the change region image of the smoke region.

[0096] This process involves performing inter-frame difference operations between the multispectral image and the background image. This involves subtracting the absolute values ​​of corresponding pixels in the two images to obtain a difference image. The difference image contains information about the differences between the two images. After obtaining the difference image, based on the location of the smoke region in the image, the corresponding location in the difference image is determined, thus obtaining the image of the changed region.

[0097] Step S4433: Extract features from the image of the changed region to obtain the features of the changed region.

[0098] In some implementations, feature extraction networks can be used to extract features from images of changing regions to obtain the features of the changing regions. The type, structure, and other parameters of the feature extraction network are not limited and can be set according to the actual situation.

[0099] Step S4434: Obtain the similarity between the features of the changed region and the features of the standard smoke.

[0100] Standard smoke features are extracted when smoke is present in the monitored area. This involves acquiring multispectral and background images of the monitored area with smoke present. These images are then used to determine the changing regions within the smoke area. Features are extracted from these changing regions to obtain the standard smoke features. It's understandable that there are various types of smoke, and different types of smoke correspond to different standard smoke features. Therefore, when multiple standard smoke features are available, the similarity between the changing region features and each of the multiple standard smoke features can be obtained separately.

[0101] In some implementations, the Euclidean distance between the variation region feature X and the standard smoke feature S can be calculated, using the following formula:

[0102]

[0103] Where ds is the Euclidean distance between the changed region feature and the standard smoke feature, X(i) is the changed region feature, S(i) is the standard smoke feature, and n represents the dimension of the feature vector. A smaller Euclidean distance corresponds to a higher similarity. By setting an appropriate distance threshold, smoke and water mist can be distinguished. In one example, the distance threshold ranges from 0.15 to 0.25, corresponding to a similarity range of 0.8 to 0.87.

[0104] Step S4435: In response to the similarity meeting the preset requirements, feature extraction is performed on the image to be detected to obtain the detection image features corresponding to the image to be detected.

[0105] In some implementations, if the similarity between the features of the changed area and the features of standard smoke is greater than or equal to a preset similarity threshold, then the target is determined to be smoke; otherwise, the target is determined not to be smoke.

[0106] In some implementations, the similarity value ranges from 0 to 1. If the similarity is greater than or equal to a preset similarity threshold (e.g., 0.6 to 1), the target is determined to be smoke; otherwise, the target is determined not to be smoke.

[0107] In some implementations, multispectral image detection is not performed after smoke motion trend detection, but directly after the target detection result is smoke. Correspondingly, step S44 may also include steps S444 to S448:

[0108] Step S444: In response to the detection result indicating the presence of smoke in the image to be detected, acquire a multispectral image and a background image of the monitored area. The multispectral image and the image to be detected are images acquired from the same monitored area within a time range, and the background image is an image acquired when the monitored area has not changed.

[0109] Step S445: Use the multispectral image and background image to determine the change region image of the smoke region.

[0110] Step S446: Extract features from the image of the changed region to obtain the features of the changed region.

[0111] Step S447: Obtain the similarity between the features of the changed region and the features of the standard smoke.

[0112] Step S448: In response to the similarity meeting the preset requirements, feature extraction is performed on the image to be detected to obtain the detection image features corresponding to the image to be detected.

[0113] There is no fixed order between steps S441 to S443 and steps S444 to S448. For an explanation of steps S444 to S448, please refer to steps S4431 to S4435, which will not be repeated here.

[0114] In some implementations, after step S448 responds to the similarity meeting the preset requirements, the image to be detected can be further used to detect the smoke motion trend, which will not be elaborated here.

[0115] Step S45: Match the detected image features of the image to be detected with the false detection image features contained in the false detection image feature library. The false detection image features are extracted from the false detection images where the target detection results and target annotation results are inconsistent. The target detection results of the false detection images indicate that there is smoke in the false detection images, and the target annotation results indicate that there is no smoke in the false detection images.

[0116] Step S46: In response to the absence of a false detection image feature matching the detection image feature in the false detection image feature library, target classification is performed using the detection image feature.

[0117] In cases where no false detection image features match the features of the detected image in the false detection image feature library, a secondary discrimination is performed on the image to be detected. That is, the target is classified using the features of the detected image corresponding to the image to be detected, and further the classification results are used to determine whether there is smoke or fire in the image to be detected.

[0118] Step S47: In response to the classification result of the target classification indicating that there is no smoke in the image to be detected, the features of the detected image are added to the false detection image feature library.

[0119] Similarly, for a description of steps S46 to S47, please refer to the corresponding locations in the above embodiments, which will not be repeated here.

[0120] In some implementations, in response to the classification result indicating the presence of smoke or fire in the image to be detected, it is determined that smoke or fire exists in the image, indicating that smoke or fire exists in the monitored area. After determining that smoke or fire exists in the image to be detected, an alarm can be triggered, and relevant fire-fighting equipment can be activated for fire prevention and extinguishing. Specifically, when the target detection device detects smoke or fire, it will control the alarm device to emit an audible or visual alarm signal, indicating that a fire has occurred and people should evacuate as soon as possible. In addition, relevant fire-fighting equipment can be activated to extinguish the initial fire and prevent it from spreading.

[0121] Because many objects (such as white clouds and red lights) are very close to smoke and fire, vision-based target detection methods have difficulty distinguishing them, thus detecting images with smoke and fire when there is none, forming the first type of false detection scenario. In addition, there is also the possibility of detecting images with smoke and fire as if there were none, forming the second type of false detection scenario. This embodiment uses the first type of false detection scenario as an example, which corresponds to steps S45-S47. In other embodiments, the target detection result of the false detection image can also indicate that there is no smoke and fire in the false detection image, while the target annotation result indicates that there is smoke and fire in the false detection image. Correspondingly, step S47 can either respond to the classification result of the target classification indicating that there is smoke and fire in the image to be detected and add the detected image features to the false detection image feature library, or respond to the classification result of the target classification indicating that there is no smoke and fire in the image to be detected and determine that there is no smoke and fire in the image to be detected.

[0122] In this embodiment, for the two false detection scenarios mentioned above, false detection image features for each scenario can be collected under the same monitoring area, forming a first false detection image feature library and a second false detection image feature library, respectively. In practical use, the detected image features can be matched with the corresponding false detection image feature libraries. For example, when the target detection result indicates the presence of smoke in the image to be detected, the detected image features are matched with the first false detection image features contained in the first false detection image feature library. The first false detection image features are extracted from false detection images where the target detection result indicates the presence of smoke in the false detection image, and the target annotation result indicates the absence of smoke in the false detection image. If the image to be detected matches the false detection image features corresponding to the first false detection scenario, it indicates that there is actually no smoke in the image to be detected, but it is falsely detected as having smoke. For example, when the target detection result indicates that there is no smoke in the image to be detected, the detected image features are matched with the second false detection image features contained in the second false detection image feature library. The second false detection image features are extracted from false detection images where the target detection result indicates that there is no smoke in the false detection image, but the target annotation result indicates that there is smoke in the false detection image. If the image to be detected matches the false detection image features corresponding to the second false detection case, it means that there is actually smoke in the image to be detected, but it will be falsely detected as not having smoke.

[0123] Please see Figure 9 , Figure 9 This is a flowchart illustrating yet another embodiment of the target detection method of this application.

[0124] The method may include the following steps:

[0125] Step S501: Acquire the image to be detected from the monitored area.

[0126] Step S502: Perform dehazing on the image to be detected.

[0127] Step S503: Perform image enhancement on the image to be detected after dehazing.

[0128] Step S504: Perform target detection on the image to be detected to obtain the target detection result.

[0129] In response to the target detection result being a flame, step S509 is executed;

[0130] In response to the target detection result being smoke, step S505 is executed.

[0131] Step S505: Perform smoke movement trend detection.

[0132] Specifically, an image of the smoke region in the image to be detected can be obtained, and then the smoke region image can be used to detect the smoke motion trend to obtain the motion trend detection result.

[0133] Step S506: Determine whether the smoke shows a movement trend.

[0134] If so, proceed to step S507;

[0135] Otherwise, proceed to step S501.

[0136] Step S507: Perform multispectral smoke target feature discrimination to obtain the discrimination result.

[0137] Specifically, multispectral images and background images of the monitored area can be acquired. The background image is an image acquired when the monitored area remains unchanged. Then, the changed areas of the smoke region are identified using the multispectral and background images. Features are then extracted from these changed areas to obtain their features. The similarity between these features and standard smoke features is then calculated, and finally, a discrimination result is obtained based on this similarity. If the similarity meets a preset requirement, the discrimination result is determined to be smoke; otherwise, the discrimination result is determined to be water mist.

[0138] Step S508: Determine whether the judgment result is smoke.

[0139] If so, proceed to step S509;

[0140] Otherwise, proceed to step S501.

[0141] Step S509: Perform false detection image feature matching.

[0142] Specifically, the process includes extracting features from the image to be detected to obtain the detection image features of the image to be detected, and then matching the detection image features of the image to be detected with the false detection image features contained in the false detection image feature library. The false detection image features are extracted from false detection images where the target detection result and the target annotation result are inconsistent.

[0143] Step S510: Determine whether there are false detection image features in the false detection image feature library that match the features of the detected image.

[0144] If so, proceed to step S501 to restart the detection;

[0145] Otherwise, proceed to step S511.

[0146] Step S511: Classify the target using the features of the detected image.

[0147] Step S512: Determine whether the classification result of the target classification is inconsistent with the target detection result of the image to be detected and whether the classification result of the target classification is consistent with the target annotation result of the falsely detected image.

[0148] If so, proceed to step S513;

[0149] Otherwise, proceed to step S514.

[0150] In one implementation scenario, when the target annotation result indicates that the falsely detected image does not contain smoke, while the target detection result corresponding to the image to be detected indicates that the image to be detected contains smoke, and the target classification result indicates that the image to be detected does not contain smoke, step S513 is executed, and step S501 is repeated; and when the target classification result indicates that the image to be detected contains smoke, step S514 is executed, that is, it is determined that the image to be detected does indeed contain smoke, thereby further performing system alarm and other operations.

[0151] In another implementation scenario, when the target annotation result indicates that the falsely detected image contains fireworks, while the target detection result corresponding to the image to be detected indicates that the image to be detected does not contain fireworks, and the target classification result indicates that the image to be detected contains fireworks, step S513 is executed, and step S501 is repeated; and when the target classification result indicates that the image to be detected does not contain fireworks, step S514 is executed, that is, it is determined that the image to be detected does not contain fireworks.

[0152] Step S513: Add the detected image features to the false detection image feature library.

[0153] Step S514: Determine that the image to be detected has not been falsely detected.

[0154] For a description of the above steps, please refer to the corresponding section of the foregoing embodiments; it will not be repeated here.

[0155] The present application addresses the problem of false alarms in smoke detection in monitored areas (such as underground mines) by establishing an initial false alarm image feature library. The detection image features of the image to be detected are directly matched with the false alarm image features contained in the library. Alternatively, an initial false alarm image library can be established, and based on a feature matching model (such as a Siamese neural network), the image to be detected and the false alarm images contained in the library are used as input to the model. The feature matching model extracts feature vectors from both the image to be detected and the false alarm images. Then, the extracted detection image features of the image to be detected are matched with the false alarm image features of the false alarm images to determine whether smoke or fire exists in the image to be detected. Furthermore, a target classification network is used for secondary discrimination of the target. If the target classification result is inconsistent with the target detection result of the image to be detected, but consistent with the target annotation result of the false alarm image, the detection image features of the image to be detected are registered in the false alarm image feature library, continuously optimizing the system's false alarm detection capability.

[0156] Furthermore, to address the interference of water mist in the monitored area on smoke detection, a defogging algorithm is first used to filter out some of the lighter fog in the image, enhancing image clarity and improving the detection capability of subsequent algorithms. After the target in the image is detected, its multispectral features are analyzed to determine whether the target is smoke or water mist, improving the recall rate of the system algorithm. In addition, compared to smoke detectors used in non-large indoor enclosed spaces or near the detected object, the target detection method of this application provides higher timeliness and accuracy in detecting smoke and fire.

[0157] Please see Figure 10 , Figure 10 This is a schematic block diagram of an embodiment of the target detection device of this application.

[0158] The target detection device 100 includes a feature extraction module 110, a feature matching module 120, a label classification module 130, and a feature addition module 140. The feature extraction module 110 extracts features from the image to be detected to obtain the detection image features. The feature matching module 120 matches the detection image features of the image to be detected with false detection image features contained in a false detection image feature library. False detection image features are extracted from false detection images where the target detection result and target labeling result are inconsistent. The target classification module 130 performs target classification using the detection image features in response to the absence of a matching false detection image feature in the false detection image feature library. The feature addition module 140 adds the detection image features to the false detection image feature library in response to a situation where the target classification result is inconsistent with the target detection result of the image to be detected, but the target classification result is consistent with the target labeling result of the false detection image.

[0159] In some implementations, the target annotation result indicates that a target exists in the false detection image. The feature addition module 140 is specifically used to add the detection image features to the false detection image feature library in response to the target detection result indicating that there is no target in the image and the target classification result indicating that there is a target in the image.

[0160] In some implementations, the target annotation result indicates that there is no target in the false detection image. The feature addition module 140 is specifically used to add the detection image features to the false detection image feature library in response to the target detection result indicating that there is a target in the image and the classification result indicating that there is no target in the image.

[0161] In some embodiments, the target detection device further includes a target detection module (not shown), which is used to perform target detection on the image to be detected before performing feature extraction on the image to be detected to obtain the detection image features of the image to be detected, and to obtain a target detection result; in response to the target detection result indicating that a target exists in the image to be detected, the step of performing feature extraction on the image to be detected to obtain the detection image features of the image to be detected is performed.

[0162] In some implementations, the target includes smoke and / or flame.

[0163] In some embodiments, when the target includes smoke, the target detection device further includes a motion trend detection module (not shown). The motion trend detection module is used to acquire a smoke region image in the image to be detected in response to the target detection result indicating the presence of smoke in the image to be detected; to perform smoke motion trend detection using the smoke region image; and to perform feature extraction on the image to be detected in response to the motion trend detection result indicating the presence of motion trend, thereby obtaining the detection image features of the image to be detected.

[0164] In some embodiments, the target detection device 100 further includes a multispectral detection module (not shown). The multispectral detection module is used to acquire a multispectral image and a background image of the monitored area in response to a target detection result indicating the presence of smoke in the image to be detected. The multispectral image and the image to be detected are images acquired from the same monitored area within a time range, and the background image is an image acquired when the monitored area remains unchanged. The module then uses the multispectral image and the background image to determine the changing region image of the smoke area; performs feature extraction on the changing region image to obtain changing region features; obtains the similarity between the changing region features and standard smoke features; and, in response to the similarity meeting a preset requirement, performs feature extraction on the image to be detected to obtain the detection image features corresponding to the image to be detected.

[0165] In some embodiments, the image to be detected is an image acquired from an underground area in a mine; and / or, the target detection device further includes an image preprocessing module (not shown), which is used to perform dehazing on the image to be detected before extracting features from the image to be detected to obtain the detection image features; and to perform image enhancement on the dehazed image to be detected.

[0166] Please see Figure 11 , Figure 11 This is a schematic block diagram of the structure of an embodiment of the electronic device of this application.

[0167] The electronic device 200 includes a memory 210 and a processor 220 coupled to each other. The memory 210 is used to store program data, and the processor 220 is used to execute the program data to implement the steps in any of the above method embodiments.

[0168] Electronic device 200 may include, but is not limited to: image acquisition device, personal computer (e.g., desktop computer, laptop computer, tablet computer, handheld computer, etc.), mobile phone, server, monitoring equipment, etc., without limitation.

[0169] Specifically, processor 220 controls itself and memory 210 to implement the steps in any of the above method embodiments. Processor 220 may also be referred to as a Central Processing Unit (CPU). Processor 220 may be an integrated circuit chip with signal processing capabilities. Processor 220 may also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. A general-purpose processor may be a microprocessor or any conventional processor. Furthermore, processor 220 may be implemented by multiple integrated circuit chips.

[0170] Please see Figure 12 , Figure 12 This is a schematic block diagram of an embodiment of a computer-readable storage medium of this application.

[0171] The computer-readable storage medium 300 stores program data 310, which, when executed by a processor, is used to implement the steps in any of the above method embodiments.

[0172] The computer-readable storage medium 300 can be a USB flash drive, external hard drive, read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk, or a medium capable of storing computer programs. It can also be a server storing the computer program, which can send the stored computer program to other devices for execution, or it can run the stored computer program itself.

[0173] In this application, the term "and / or" is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent three cases: A existing alone, A and B existing simultaneously, and B existing alone. Furthermore, the term "at least one" in this application means any combination of at least two of any one or more of a plurality of elements. For example, including at least one of A, B, and C can mean including any one or more elements selected from the set consisting of A, B, and C.

[0174] In the several embodiments provided in this application, it should be understood that the disclosed methods and apparatus can be implemented in other ways. For example, the apparatus implementations described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0175] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment, depending on actual needs.

[0176] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0177] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute all or part of the steps of the methods of various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0178] The above are merely embodiments of this application and do not limit the patent scope of this application. Any equivalent structural or procedural transformations made using the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.

Claims

1. A target detection method, characterized in that, include: Feature extraction is performed on the image to be detected to obtain the detection image features of the image to be detected; The detected image features of the image to be detected are matched with the false detection image features contained in the false detection image feature library, wherein the false detection image features are extracted from false detection images where the target detection result and the target annotation result are inconsistent; In response to the absence of a false detection image feature matching the detected image feature in the false detection image feature library, target classification is performed using the detected image feature; In response to the discrepancy between the classification result of the target classification and the target detection result of the image to be detected, and the discrepancy between the classification result of the target classification and the target annotation result of the falsely detected image, the features of the detected image are added to the falsely detected image feature library. The target annotation result indicates that a target exists in the falsely detected image, and the target includes smoke; Wherein, when the target includes smoke, the step of performing feature extraction on the image to be detected in response to the target detection result indicating the presence of a target in the image to be detected, to obtain the detection image features of the image to be detected, includes: in response to the target detection result indicating the presence of smoke in the image to be detected, acquiring a smoke region image in the image to be detected; using the smoke region image to detect smoke motion trends; and in response to the motion trend detection result indicating the presence of a motion trend, performing feature extraction on the image to be detected to obtain the detection image features of the image to be detected. Wherein, when the target includes smoke, the step of performing feature extraction on the image to be detected in response to the target detection result being the target, to obtain the detection image features of the image to be detected, includes: in response to the target detection result indicating the presence of smoke in the image to be detected, acquiring a multispectral image and a background image collected from the monitoring area, wherein the multispectral image and the image to be detected are images collected from the same monitoring area within a time range, and the background image is an image collected when the monitoring area has not changed; using the multispectral image and the background image to determine the changing area image of the smoke area; performing feature extraction on the changing area image to obtain changing area features; obtaining the similarity between the changing area features and standard smoke features; and in response to the similarity meeting a preset requirement, performing feature extraction on the image to be detected to obtain the detection image features corresponding to the image to be detected.

2. The method according to claim 1, characterized in that, In response to the fact that the classification result of the target classification is inconsistent with the target detection result of the image to be detected, and the classification result of the target classification is consistent with the target annotation result of the falsely detected image, the detected image features are added to the falsely detected image feature library, including: In response to the target detection result of the image to be detected indicating that there is no target in the image to be detected and the classification result of the target classification indicating that there is a target in the image to be detected, the detected image features are added to the false detection image feature library.

3. The method according to claim 1, characterized in that, The target annotation result indicates that the target does not exist in the falsely detected image. The step of adding the detected image features to the false detection image feature library in response to the fact that the classification result of the target classification is inconsistent with the target detection result of the image to be detected and the classification result of the target classification is consistent with the preset classification result includes: In response to the target detection result of the image to be detected indicating that a target exists in the image to be detected and the classification result of the target classification indicating that a target does not exist in the image to be detected, the detected image features are added to the false detection image feature library.

4. The method according to claim 3, characterized in that, Before performing feature extraction on the image to be detected to obtain the detection image features, the method further includes: Target detection is performed on the image to be detected to obtain the target detection result; In response to the target detection result indicating the presence of a target in the image to be detected, a step is performed to extract features from the image to be detected to obtain the detection image features of the image to be detected.

5. The method according to any one of claims 2-4, characterized in that, The target includes flames.

6. The method according to claim 1, characterized in that, The image to be detected is an image acquired from an underground mine area; and / or, Before performing feature extraction on the image to be detected to obtain the detection image features, the method further includes: The image to be detected is dehazed; Image enhancement is performed on the image to be detected after dehazing.

7. An electronic device, characterized in that, The electronic device includes a memory and a processor coupled to each other, the memory being used to store program data and the processor being used to execute the program data to implement the method as described in any one of claims 1-6.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores program data, which, when executed by a processor, is used to implement the method as described in any one of claims 1-6.

Images