Method and device for determining oil mist, storage medium and electronic device

Abstract

Embodiments of the present application provide a kind of oil mist determination method, device, storage medium and electronic device, wherein the method comprises: obtaining the target image collected by image acquisition equipment;First image is detected, and the target texture information of first image is obtained;Second image is input into first classification network, and first classification result is obtained;Third image is input into second classification network, and second classification result is obtained;According to target texture information, first classification result and second classification result, determine whether there is oil mist in target image. By the present application, the problem that whether there is oil mist on the lens cannot be accurately identified is solved, and the effect of improving the accuracy of identifying whether there is oil mist on the lens is achieved.

Description

TECHNICAL FIELD

[0001] Embodiments of the present application relate to the field of computers, and in particular, to a method and apparatus for determining oil mist, a storage medium and an electronic device. BACKGROUND

[0002] In the field of image processing, specific operations can be performed based on the captured image, wherein the definition of the image captured by the camera device directly affects the accuracy of the execution of the specific operation. For example, in the case of oil mist on the lens, the definition of the image will decrease. The following takes an endoscope as an example for illustration:

[0003] With the help of artificial intelligence technology, doctors can identify oil mist in the camera of an endoscope when cutting off the lesion tissue. For example, when a doctor uses an electric knife or an ultrasonic knife to cut off human tissue, a large amount of oil mist will be generated due to high temperature, and a large amount of oil mist will adhere to the camera of the endoscope, which will block the doctor's view and prevent the doctor from performing the operation normally. Therefore, it is necessary to automatically detect whether there is oil mist on the endoscope, and in the case of oil mist, an oil mist cleaning operation can be automatically initiated to ensure the normal performance of the doctor's operation.

[0004] In the prior art, a single deep network model is used to detect oil mist on an endoscope. However, due to the complex scene of the endoscope, such as the relatively light smoke, the data labeling process has strong subjectivity, and the smoke concentration is highly dependent on the inter-frame difference. In addition, the endoscope can be moved out of the body for cleaning, wiping, etc. during use. Therefore, the single deep network model has a high risk of false detection and cannot accurately identify whether there is oil mist on the lens.

[0005] In view of the above problems in the related art, no effective solution has been proposed so far. SUMMARY

[0006] Embodiments of the present application provide a method and apparatus for determining oil mist, a storage medium and an electronic device to at least solve the problem that the related art cannot accurately identify whether there is oil mist on the lens.

[0007] According to one embodiment of the present application, a method for determining oil mist is provided, comprising: acquiring a target image collected by an image collection device; detecting a first image to obtain target texture information of the first image, wherein the first image is obtained by transforming the target image; inputting a second image into a first classification network to obtain a first classification result, wherein the second image is obtained by transforming the first image, and the first classification result is used to indicate whether the second image is an image taken in a target scene; inputting a third image into a second classification network to obtain a second classification result, wherein the third image is obtained by transforming the second image, the second classification result is used to indicate an oil mist concentration in the third image, and a color space corresponding to the second image and a color space corresponding to the third image are different; and determining whether the target image contains oil mist according to the target texture information, the first classification result and the second classification result.

[0008] Optionally, determining whether the target image contains oil mist according to the target texture information, the first classification result and the second classification result comprises: in a case where the texture degree indicated by the target texture information exceeds a predetermined degree threshold, the first classification result indicates that the second image is an image taken in the target scene, and the oil mist concentration determined based on the second classification result exceeds a target concentration threshold, it is determined that the target image contains oil mist.

[0009] Optionally, the detecting the first image to obtain the target texture information of the first image comprises: acquiring a grayscale image of the first image; performing a denoising operation on the grayscale image of the first image to obtain a denoised image; and using a non-maximum suppression algorithm to obtain an edge texture map of the denoised image according to a gradient intensity and a gradient direction of the denoised image, and acquiring the target texture information according to the edge texture map.

[0010] Optionally, the first image is a YUV image, the second image is an RGB image, and the third image is an HSV image.

[0011] Optionally, before the inputting the second image into the first classification network to obtain the first classification result, the method further comprises: obtaining a classification network model, wherein the classification network model comprises a convolution layer, a pooling layer, a Flatten layer, a full connection layer and a Softmax layer, and the convolution layer, the pooling layer, the Flatten layer, the full connection layer and the Softmax layer are connected in sequence; and training the classification network model using a first image sample set to obtain the first classification network.

[0012] Optionally, before inputting the third image into the second classification network to obtain the second classification result, the method further comprises: training the classification network model using the second image sample set to obtain the second classification network.

[0013] Optionally, the convolutional layer comprises six convolutional layers, wherein, in the convolutional layer: the output of a first convolutional layer is connected with the input of a second convolutional layer, and the output of the first convolutional layer is also connected with the input of a third convolutional layer, and the third convolutional layer comprises two convolutional layers; the output of the second convolutional layer and the output of the third convolutional layer are both connected with the input of a first merging module, and the first merging module is used for merging the output data of the second convolutional layer and the output data of the third convolutional layer; the output of the first merging module is connected with the input of a fourth convolutional layer and the input of a second merging module, the output of the fourth convolutional layer is connected with the input of the second merging module, and the second merging module is used for merging the output data of the first merging module and the output data of the fourth convolutional layer and inputting the merged data into the pooling layer, wherein the fourth convolutional layer comprises two convolutional layers.

[0014] According to another embodiment of the present application, a device for determining oil mist is provided, comprising: an acquisition module configured to acquire a target image collected by an image acquisition device;

[0015] The first obtaining module is configured to detect a first image to obtain target texture information of the first image, wherein the first image is obtained by transforming the target image; the second obtaining module is configured to input a second image into a first classification network to obtain a first classification result, wherein the second image is obtained by transforming the first image, and the first classification result is used to indicate whether the second image is an image captured in a target scene; the third obtaining module is configured to input a third image into a second classification network to obtain a second classification result, wherein the third image is obtained by transforming the second image, the second classification result is used to indicate the concentration of oil mist in the third image, and the color space corresponding to the second image is different from the color space corresponding to the third image; and the determining module is configured to determine whether oil mist exists in the target image according to the target texture information, the first classification result and the second classification result.

[0016] According to still another embodiment of the present application, a computer readable storage medium is also provided, and the computer readable storage medium stores a computer program, wherein the computer program is executed by a processor to implement the steps of the method described in any of the above embodiments.

[0017] According to another embodiment of the present invention, an electronic device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to perform the steps in any one of the above method embodiments.

[0018] The present invention first detects a first image transformed from an original target image to obtain target texture information. Next, the first image is transformed into a second image, which is input into a first classification network to obtain a first classification result. Finally, the second image is transformed into a third image, which is input into a second classification network to obtain a second classification result. Based on the target texture information, the first classification result, and the second classification result, the presence of oil mist on the target image is determined. This solves the problem of being unable to accurately identify the presence of oil mist on a lens, thereby improving the accuracy of identifying the presence of oil mist on a lens. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 is a block diagram of the hardware structure of a mobile terminal according to a method for determining oil mist according to an embodiment of the present invention;

[0020] Figure 2 is a flow chart of a method for determining oil mist according to an embodiment of the present invention;

[0021] Figure 3 is a schematic diagram of an optionally classified network structure according to an embodiment of the present invention;

[0022] Figure 4 is an overall flow chart of an optional oil mist determination method according to an embodiment of the present invention;

[0023] Figure 5 4 is a structural block diagram of an oil mist determination device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0024] Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings and in combination with embodiments.

[0025] It should be noted that the terms "first", "second", etc. in the description and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0026] The method embodiments provided in the embodiments of the present application can be executed in a mobile terminal, a computer terminal or a similar computing device. Taking running on a mobile terminal as an example, Figure 1 FIG. 1 is a hardware structure diagram of a mobile terminal for determining oil mist according to an embodiment of the present invention. Figure 1 As shown, the mobile terminal may include one or more ( Figure 1Only one is shown) a processor 102 (the processor 102 may include but is not limited to a microprocessor MCU or a programmable logic device FPGA and other processing devices) and a memory 104 for storing data, wherein the mobile terminal may also include a transmission device 106 and an input and output device 108 for communication functions. It will be understood by those skilled in the art that Figure 1 The structure shown is only for illustration and does not limit the structure of the mobile terminal. Figure 1 More or fewer components than shown, or with Figure 1 Different configurations shown.

[0027] Memory 104 can be used to store computer programs, such as application software programs and modules, such as the computer program corresponding to the oil mist determination method described in the embodiments of the present invention. Processor 102 executes the computer programs stored in memory 104 to execute various functional applications and data processing, thereby implementing the aforementioned method. Memory 104 may include high-speed random access memory (RAM) and non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some embodiments, memory 104 may further include memory located remotely from processor 102, which can be connected to a mobile terminal via a network. Examples of such networks include, but are not limited to, the Internet, an intranet, a local area network (LAN), a mobile communication network, and combinations thereof.

[0028] The transmission device 106 is used to receive or send data via a network. A specific example of the aforementioned network may include a wireless network provided by the mobile terminal's communications provider. In one embodiment, the transmission device 106 includes a network interface controller (NIC), which can be connected to other network devices via a base station to enable communication with the Internet. In another embodiment, the transmission device 106 may be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0029] In this embodiment, a method for determining oil mist is provided. Figure 2 FIG. 1 is a flow chart of a method for determining oil mist according to an embodiment of the present invention. Figure 2 As shown, the process includes the following steps:

[0030] Step S202, acquiring a target image captured by an image acquisition device;

[0031] The image acquisition device can be an endoscope camera used in a medical operation, and the target image can be an initial image captured by the endoscope camera. In addition, the image acquisition device can also be other types of camera devices, such as a camera device having an imaging principle similar to that of the endoscope camera.

[0032] In step S204, the first image is detected to obtain target texture information of the first image, wherein the first image is obtained by transforming the target image.

[0033] For example, the first image can be a YUV image obtained by transforming the original image. The texture information of the YUV image can be obtained by detecting the YUV image. The texture information can be the texture degree of the image, such as the canny texture degree of the YUV image.

[0034] In step S206, the second image is input into the first classification network to obtain a first classification result, wherein the second image is obtained by transforming the first image, and the first classification result is used to indicate whether the second image is an image captured in a target scene.

[0035] The second image can be obtained by transforming the first image. The first image can be a YUV image, and the second image can be an RGB image. The first classification network can be a residual network model constructed based on ResNet. The second image can be input into the first classification network to obtain the first classification result. The first classification result can be used to determine whether the second image is an image captured in an in-vivo scene or an in-vitro scene.

[0036] In step S208, the third image is input into the second classification network to obtain a second classification result, wherein the third image is obtained by transforming the second image, the second classification result is used to indicate the oil mist concentration in the third image, and the color space corresponding to the second image is different from the color space corresponding to the third image.

[0037] The third image can be obtained by transforming the second image. The second image can be an RGB image, and the third image can be an HSV image. The second classification network can be a residual network model constructed based on ResNet. The third image can be input into the second classification network to obtain the second classification result. The second classification result can be used to determine the concentration of the oil mist in the third image. The second image and the third image have different color spaces.

[0038] In step S210, whether the oil mist exists in the target image is determined according to the target texture information, the first classification result, and the second classification result.

[0039] The target texture information can include texture degree information obtained in the first image, the first classification result can be used to determine the collection scene of the image, and the first classification result can be used to determine whether the second image is an image taken in an in-vivo scene or an in-vitro scene, the second classification result can determine the concentration of oil mist in the third image, and whether the target image contains oil mist is determined according to the target texture information, the first classification result and the second classification result.

[0040] Optionally, the execution subject of the above steps can be a background processor or other device with similar processing capability, and can also be a machine integrated with at least an image acquisition device and a data processing device, wherein the image acquisition device can include a camera and other image acquisition modules, and the data processing device can include a computer, a mobile phone and other terminals, but is not limited thereto.

[0041] Through the above steps, first, the target texture information is obtained by detecting the first image transformed from the original target image, second, the first classification result is obtained by inputting the second image transformed from the first image into the first classification network, and finally, the second classification result is obtained by inputting the third image transformed from the second image into the second classification network, and whether the target image contains oil mist is determined through the above target texture information, the first classification result and the second classification result. Therefore, the problem that the oil mist on the endoscope cannot be accurately identified can be solved, and the effect of improving the accuracy of identifying the oil mist on the lens is achieved.

[0042] In an optional embodiment, according to the target texture information, the first classification result and the second classification result, whether the target image contains oil mist is determined, including: in a case where the texture degree indicated by the target texture information is less than a predetermined degree threshold, the first classification result is used to indicate that the second image is an image taken in the target scene, and in a case where it is determined based on the second classification result that the oil mist concentration exceeds a target concentration threshold, it is determined that the target image contains oil mist.

[0043] When a large amount of oil mist appears in the image, the image texture level is significantly reduced. Therefore, the image texture level feature can be used as an important feature to measure the presence of oil mist. The texture level of the first image can be evaluated based on the Canny feature. If the texture level of the first image is less than a predetermined threshold, the presence of oil mist is determined. For example, the predetermined threshold can be set to 10,000 (other values ​​are also possible, such as 8,000, 9,000, 11,000, 15,000, etc.). That is, if the texture level of the first image is less than 10,000, the presence of oil mist can be determined. The predetermined threshold can be determined based on actual conditions, and the value of the oil mist concentration threshold is not limited here. The target scene can be an in-vivo scene. The first classification result can be used to determine whether the second image was captured in an in-vivo or in-vitro scene. If the first classification result determines that the second image was captured in an in-vivo scene, the determination of whether oil mist is present can be continued. If the first classification result determines that the second image was captured in an in-vivo scene, it can be determined that the determination of whether oil mist is present in the in-vitro scene is unnecessary, and it can be determined that no oil mist is present in the second image. A second classification result is obtained by classifying the third image. The oil mist concentration can be determined in the second classification result. If the concentration exceeds a target concentration threshold, the presence of oil mist in the third image can be determined. For example, the target concentration threshold can be 0.5 (of course, other values, such as 0.4, 0.6, 0.8, etc., are also possible). That is, if the oil mist concentration is greater than 0.5, the presence of oil mist in the third image can be determined. The above target concentration threshold can be determined based on actual conditions, and there is no restriction on the value of the oil mist concentration threshold. When the texture degree obtained through the first image is less than a predetermined degree threshold, the second image can be determined as an in-vivo scene in the first classification result, and the oil mist concentration can be determined to exceed the target concentration threshold in the second classification result, it is determined that oil mist exists in the target image obtained through the endoscope.

[0044] In an optional embodiment, the detecting of the first image to obtain target texture information of the first image includes: obtaining a grayscale image of the first image; performing a denoising operation on the grayscale image of the first image to obtain a denoised image; obtaining an edge texture map of the denoised image using a non-maximum suppression algorithm based on the gradient strength and gradient direction of the denoised image; and obtaining the target texture information based on the edge texture map.

[0045] Here, the canny texture degree of the first input image is calculated. First, the canny texture detection is performed on the first input image:

[0046] The first step is to use Gaussian filtering to perform denoising on the grayscale image of the first image;

[0047] The second step calculates the gradient intensity and the gradient direction of the denoised image:

[0048]

[0049] In the above formula, G is the gradient intensity; θ is the gradient direction; f x , f y are the x and y direction gradients, respectively.

[0050] The third step uses a non-maximum suppression technique based on the edge gradient direction to obtain a canny edge texture map. After obtaining the canny edge texture map, the canny texture degree of the image is calculated:

[0051]

[0052] where C(x, y) is the canny edge texture map, M and N are the width and height of the image, respectively. When a large amount of oil mist appears in the picture, the texture degree of the image is obviously reduced, so the image texture degree feature can be used as an important feature to measure whether oil mist appears or not.

[0053] In an optional embodiment, the first image is a YUV image, the second image is an RGB image, and the third image is an HSV image.

[0054] Among them, the original image can be collected by the camera in the endoscope, the original image is converted into a YUV image, the image texture degree is detected, then the YUV image can be converted into an RGB image, and it is detected whether the shooting scene in the RGB image is an in-vivo scene or an in-vitro scene, finally the RGB image is converted into an HSV image, and whether oil mist exists in the HSV image is determined by detecting the concentration of oil mist in the HSV image.

[0055] In an optional embodiment, before the second image is input into the first classification network to obtain the first classification result, the method further comprises: obtaining a classification network model, wherein the classification network model comprises a convolutional layer, a pooling layer, a Flatten layer, a fully connected layer and a Softmax layer, and the convolutional layer, the pooling layer, the Flatten layer, the fully connected layer and the Softmax layer are connected in sequence; training the classification network model using a first image sample set to obtain the first classification network.

[0056] Among them, a neural network model can be constructed, and after training the neural network model, it can be determined whether it is an in-vivo scene or an in-vitro scene. Considering the real-time requirement of the algorithm, we construct a light network based on the excellent resnet, which has six convolutional layers, to improve the running speed of the algorithm, such as Figure 3Fig. 1 shows a schematic diagram of an optional classification network structure according to an embodiment of the present application. The classification network model can be as shown in Fig. 1. The model can be trained by inputting image samples, and the convolutional layer, the pooling layer, the Flatten layer, the fully connected layer and the Softmax layer can be sequentially connected to obtain a final neural network model. The images in the first image sample set can be RGB images. The neural network model as shown in Fig. 1 can be trained using the first sample set to obtain a first classification network. Figure 3 Fig. 1 shows a schematic diagram of an optional classification network structure according to an embodiment of the present application. The classification network model can be as shown in Fig. 1. The model can be trained by inputting image samples, and the convolutional layer, the pooling layer, the Flatten layer, the fully connected layer and the Softmax layer can be sequentially connected to obtain a final neural network model. The images in the first image sample set can be RGB images. The neural network model as shown in Fig. 1 can be trained using the first sample set to obtain a first classification network. Figure 3 Fig. 1 shows a schematic diagram of an optional classification network structure according to an embodiment of the present application. The classification network model can be as shown in Fig. 1. The model can be trained by inputting image samples, and the convolutional layer, the pooling layer, the Flatten layer, the fully connected layer and the Softmax layer can be sequentially connected to obtain a final neural network model. The images in the first image sample set can be RGB images. The neural network model as shown in Fig. 1 can be trained using the first sample set to obtain a first classification network.

[0057] In an optional embodiment, before inputting the third image into the second classification network to obtain a second classification result, the method further comprises: training the classification network model using the second image sample set to obtain the second classification network.

[0058] Fig. 2 shows a schematic diagram of an optional classification network structure according to an embodiment of the present application. The classification network model can be as shown in Fig. 2. The model can be trained by inputting image samples, and the convolutional layer, the pooling layer, the Flatten layer, the fully connected layer and the Softmax layer can be sequentially connected to obtain a final neural network model. The images in the second image sample set can be HSV images. The neural network model as shown in Fig. 2 can be trained using the second image sample set to obtain a second classification network. It should be noted that other neural network models can also be used to train the second classification network using the second image sample set, such as a deep network model. The model used by the second classification network is not limited herein. Figure 3 Fig. 2 shows a schematic diagram of an optional classification network structure according to an embodiment of the present application. The classification network model can be as shown in Fig. 2. The model can be trained by inputting image samples, and the convolutional layer, the pooling layer, the Flatten layer, the fully connected layer and the Softmax layer can be sequentially connected to obtain a final neural network model. The images in the second image sample set can be HSV images. The neural network model as shown in Fig. 2 can be trained using the second image sample set to obtain a second classification network. It should be noted that other neural network models can also be used to train the second classification network using the second image sample set, such as a deep network model. The model used by the second classification network is not limited herein.

[0059] In an optional embodiment, the convolutional layer includes six convolutional layers. In the convolutional layer: the output of the first convolutional layer is connected to the input of the second convolutional layer, and the output of the first convolutional layer is also connected to the input of the third convolutional layer, which includes two convolutional layers; the output of the second convolutional layer and the output of the third convolutional layer are both connected to the input of a first merging module, which is used to merge the output data of the second convolutional layer and the output data of the third convolutional layer; the output of the first merging module is connected to the input of a fourth convolutional layer and the input of a second merging module, the output of the fourth convolutional layer is connected to the input of the second merging module, and the second merging module is used to merge the output data of the first merging module and the output data of the fourth convolutional layer and input the merged data into the pooling layer, wherein the fourth convolutional layer includes two convolutional layers.

[0060] Fig. 3 shows a schematic diagram of a convolutional layer of a neural network model according to an embodiment of the present application. The convolutional layer of the neural network model can be as shown in Fig. 3. The neural network model can include six convolutional layers, as shown in Fig. 3. Figure 3 Fig. 3 shows a schematic diagram of a convolutional layer of a neural network model according to an embodiment of the present application. The convolutional layer of the neural network model can be as shown in Fig. 3. The neural network model can include six convolutional layers, as shown in Fig. 3. Figure 3As shown, the 6 convolutional layers can include Conv1, Conv2, Conv3, Conv4, Conv5 and Conv6, the first convolutional layer can be Conv1, the second convolutional layer can be Conv2, the third convolutional layer can be Conv3 and Conv4, and the fourth convolutional layer can be Conv5 and Conv6. It should be noted that the outputs of Conv1, Conv2, Conv3, Conv4, Conv5 and Conv6 need to be connected to an activation function, which can be a relu function. The first convolutional layer can be as follows Figure 3 As shown, the output of the first convolutional layer Conv1 can be connected to the input of the second convolutional layer Conv2, and at the same time, the output of the first convolutional layer Conv1 is also connected to the input of the third convolutional layer (including Conv2 and Conv3), the output of the second convolutional layer Conv2 and the output of the third convolutional layer (including Conv2 and Conv3) are both connected to the input of the first merging module (including Elewise and relu), the output of the first merging module is connected to the input of the fourth convolutional layer (including Conv5 and Conv6) and the input of the second merging module (including Elewise and relu), the output of the fourth convolutional layer (including Conv5 and Conv6) is connected to the input of the second merging module (including Elewise and relu), and the output data of the first merging module (including Elewise and relu) and the output data of the fourth convolutional layer (including Conv5 and Conv6) can be merged by the second merging module (including Elewise and relu), and the merged data is input into the pooling layer (Pooling).

[0061] As an optional embodiment, the most significant feature of the oil mist in the endoscope is that the picture saturation is greatly reduced, and a large amount of detail texture is lost. Considering that the HSV space of the oil mist image will produce a large visual difference, a trained neural network is used in the embodiment of the present application to classify images in the HSV space, which helps to identify the oil mist image. At the same time, due to the complexity of the extracorporeal scene, another classification network is trained in the embodiment of the present application to identify the extracorporeal scene, so as to reduce the false detection risk of the whole algorithm. Finally, the texture degree of the image is evaluated based on the canny feature. However, due to the complexity of the endoscope use scene, the oil mist detection based on the endoscope image has the following difficulties: 1) part of the smoke is light, and the data labeling process has strong subjectivity, and the smoke concentration is highly dependent on the inter-frame difference; 2) the endoscope may be moved out of the body for cleaning and wiping during use, so the false detection risk of the pure depth method is great.

[0062] The application utilizes mixed features to detect whether there is oil mist phenomenon in the endoscope image, simultaneously gives the judgment result in real time, and reduces false positives by combining a series of post-processing operations. As shown in Figure 4 The overall flowchart of the optional oil mist determination method according to the embodiment of the application is shown in the figure:

[0063] Step one, calculate the canny texture degree of the image through the input YUV image. First, perform canny texture detection on the input first image:

[0064] 1. First, use Gaussian filtering on the gray image of the first image to perform denoising operation;

[0065] 2. Calculate the gradient intensity and gradient direction of the denoised image:

[0066]

[0067] In the above formula, G is the gradient intensity; θ is the gradient direction; f x , f y are the x and y direction gradients respectively;

[0068] 3. Use non-maximum suppression technology based on the edge gradient direction to obtain the canny edge texture image. After obtaining the canny edge texture image, calculate the canny texture degree CS of the image:

[0069]

[0070] Where C(x, y) is the canny edge texture image, M and N are the width and height of the image respectively. When a large amount of oil mist appears in the picture, the image texture degree is obviously reduced, therefore, the image texture degree feature can be used as an important feature to measure whether the oil mist appears.

[0071] Step two, convert the YUV data into an RGB image and then use the classification network to determine whether it is an extracorporeal image. Since the RGB image is easy to obtain, it is convenient to collect and train, the YUV image can be converted into RGB format using the following formula,

[0072] R=Y+1.4075(V-128)

[0073] G=Y-0.3455(U-128)-0.7169(V-128)

[0074] B=Y+1.779(U-128)

[0075] Then construct a neural network model, and after training, obtain the first classification network to determine whether it is an extracorporeal scene. Considering the real-time requirement of the algorithm, a light network can be constructed based on the excellent resnet, such as Figure 3As shown in the figure, the network has six convolutional layers to improve the algorithm's running speed. At this time, if the image is identified by the network as an in vitro scene, it will not be subsequently tested for oil mist to reduce the probability of false positives.

[0076] Step 3: Convert the RGB image into an HSV image and use the deep network to detect oil mist. Due to the particularity of the HSV space found in the experiment, the following formula can be used to convert the RGB image into an HSV image:

[0077] R'=R / 255,G'=G / 255,B'=B / 255

[0078] C max=max(R',G',B'),C min=min(R',G',B'),Δ=c max-c min

[0079]

[0080]

[0081] V=Cmax

[0082] The obtained HSV image is fed into a trained deep network to output the oil mist concentration score. Combined with the results of step one and step two, a comprehensive judgment is made on whether there is oil mist.

[0083] Furthermore, it should be noted that special scenarios may occur during oil mist detection, so a fourth step, post-processing of the endoscope oil mist detection, is required. This selectively filters out false positives caused by special circumstances, such as textureless plastic pipes and images of accumulated water.

[0084] This method uses the canny texture map as a fundamental feature to assess image texture, demonstrating robustness. By analyzing the presence of oil mist from two different color spaces—the RGB image and the HSV image space—the algorithm significantly improves detection rates and reduces false positive rates.

[0085] Through the description of the above embodiments, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus the necessary general hardware platform, and of course it can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for enabling a terminal device (which can be a mobile phone, computer, server, or network device, etc.) to execute the methods described in each embodiment of the present invention.

[0086] This embodiment also provides an oil mist determination device for implementing the aforementioned embodiments and preferred implementations. Details already described will not be repeated. As used below, the term "module" may refer to a combination of software and / or hardware that implements a predetermined function. While the devices described in the following embodiments are preferably implemented using software, implementation using hardware, or a combination of software and hardware, is also possible and contemplated.

[0087] Figure 5 is a structural block diagram of an oil mist determination device according to an embodiment of the present invention, as shown in FIG. Figure 5 As shown, the device includes: an acquisition module 52, which is used to acquire a target image captured by an image acquisition device; a first acquisition module 54, which detects the first image and obtains target texture information of the first image, wherein the first image is obtained by transforming the target image; a second acquisition module 56, which is used to input the second image into a first classification network to obtain a first classification result, wherein the second image is obtained by transforming the first image, and the first classification result is used to indicate whether the second image is an image taken under the target scene; a third acquisition module 58, which is used to input the third image into a second classification network to obtain a second classification result, wherein the third image is obtained by transforming the second image, and the second classification result is used to indicate the oil mist concentration in the third image, and the color space corresponding to the second image is different from the color space corresponding to the third image; a determination module 510, which is used to determine whether there is oil mist in the target image based on the target texture information, the first classification result and the second classification result.

[0088] In an optional embodiment, the above-mentioned determination module 510 can determine whether there is oil mist in the target image based on the target texture information, the first classification result and the second classification result in the following manner: when the texture degree indicated by the target texture information is less than a predetermined degree threshold, the first classification result is used to indicate that the second image is an image taken under the target scene, and when it is determined based on the second classification result that the oil mist concentration exceeds the target concentration threshold, it is determined that there is oil mist in the target image.

[0089] In an optional embodiment, the above-mentioned first acquisition module 54 can detect the first image and obtain the target texture information of the first image in the following manner: obtain a grayscale image of the first image; perform a denoising operation on the grayscale image of the first image to obtain a denoised image; use a non-maximum suppression algorithm based on the gradient strength and gradient direction of the denoised image to obtain an edge texture map of the denoised image; and obtain the target texture information based on the edge texture map.

[0090] In an optional embodiment, the above-mentioned device includes that the first image is a YUV image, the second image is an RGB image, and the third image is an HSV image.

[0091] In an optional embodiment, the above-mentioned device is also used to obtain a classification network model before inputting the second image into the first classification network to obtain the first classification result, wherein the classification network model includes a convolution layer, a pooling layer, a Flatten layer, a fully connected layer and a Softmax layer, and the convolution layer, the pooling layer, the Flatten layer, the fully connected layer and the Softmax layer are connected in sequence; the classification network model is trained using the first image sample set to obtain the first classification network.

[0092] In an optional embodiment, the above-mentioned device is also used to use the second image sample set to train the classification network model to obtain the second classification network before inputting the third image into the second classification network to obtain the second classification result.

[0093] In an optional embodiment, the convolution layer includes six convolution layers, wherein in the convolution layer: the output of the first convolution layer is connected to the input of the second convolution layer, and the output of the first convolution layer is also connected to the input of the third convolution layer, and the third convolution layer includes two convolution layers; the output of the second convolution layer and the output of the third convolution layer are both connected to the input of the first merging module, and the first merging module is used to merge the output data of the second convolution layer with the output data of the third convolution layer; the output of the first merging module is connected to the input of the fourth convolution layer and the input of the second merging module, and the output of the fourth convolution layer is connected to the input of the second merging module, and the second merging module is used to merge the output data of the first merging module with the output data of the fourth convolution layer, and input the merged data into the pooling layer, wherein the fourth convolution layer includes two convolution layers.

[0094] It should be noted that the above modules can be implemented through software or hardware. For the latter, it can be implemented in the following ways, but not limited to: the above modules are all located in the same processor; or the above modules are located in different processors in any combination.

[0095] An embodiment of the present invention further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, wherein when the computer program is executed by a processor, the steps of any of the above methods are implemented.

[0096] In an exemplary embodiment, the computer-readable storage medium may include, but is not limited to, various media that can store computer programs, such as a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk, or an optical disk.

[0097] Optionally, in this embodiment, the storage medium may be configured to store a computer program for performing the following steps:

[0098] S1, obtaining a target image captured by an image acquisition device;

[0099] S2, detecting a first image to obtain target texture information of the first image, wherein the first image is obtained by transforming the target image;

[0100] S3, inputting the second image into the first classification network to obtain a first classification result, wherein the second image is obtained by transforming the first image, and the first classification result is used to indicate whether the second image is an image captured in the target scene;

[0101] S4, inputting the third image into a second classification network to obtain a second classification result, wherein the third image is obtained by transforming the second image, the second classification result is used to indicate the oil mist concentration in the third image, and the color space corresponding to the second image is different from the color space corresponding to the third image;

[0102] S5. Determine whether oil mist exists in the target image according to the target texture information, the first classification result, and the second classification result.

[0103] An embodiment of the present invention further provides an electronic device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to execute the steps in any one of the above method embodiments.

[0104] In an exemplary embodiment, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor, and the input / output device is connected to the processor.

[0105] Optionally, in this embodiment, the processor may be configured to execute the following steps through a computer program:

[0106] S1, obtaining a target image captured by an image acquisition device;

[0107] S2, detecting the first image to obtain target texture information of the first image, wherein the first image is obtained by transforming the target image;

[0108] S3, inputting the second image into the first classification network to obtain a first classification result, wherein the second image is obtained by transforming the first image, and the first classification result is used to indicate whether the second image is an image taken in a target scene;

[0109] S4, inputting the third image into the second classification network to obtain a second classification result, wherein the third image is obtained by transforming the second image, the second classification result is used to indicate an oil mist concentration in the third image, and a color space corresponding to the second image is different from a color space corresponding to the third image;

[0110] S5, determining whether the oil mist exists in the target image according to the target texture information, the first classification result and the second classification result.

[0111] The specific examples in the embodiment can refer to the examples described in the above embodiments and exemplary embodiments, and the embodiment will not be described here again.

[0112] Obviously, those skilled in the art should understand that the modules or steps of the present application described above can be realized by general computing devices, which can be concentrated on a single computing device or distributed on a network composed of multiple computing devices, and they can be realized by program codes executable by the computing devices, so that they can be stored in storage devices and executed by the computing devices, and in some cases, the steps shown or described can be executed in different order, or they can be manufactured into individual integrated circuit modules, or multiple modules or steps can be manufactured into a single integrated circuit module. Thus, the present application is not limited to any specific combination of hardware and software.

[0113] The above only describes the preferred embodiments of the present application and is not used to limit the present application. For those skilled in the art, the present application can have various modifications and changes. Any modification, equivalent replacement, improvement, etc. within the principles of the present application shall be included in the protection scope of the present application.

Claims

1. A method for determining oil mist, characterized in that: include: Acquire a target image captured by an image acquisition device; Detecting a first image to obtain target texture information of the first image, wherein the first image is obtained by transforming the target image; Inputting a second image into a first classification network to obtain a first classification result, wherein the second image is obtained by transforming the first image, and the first classification result is used to indicate whether the second image is an image captured in a target scene, wherein the target scene includes an in-vivo scene; inputting the third image into a second classification network to obtain a second classification result, wherein the third image is obtained by transforming the second image, the second classification result is used to indicate the oil mist concentration in the third image, and a color space corresponding to the second image is different from a color space corresponding to the third image; determining whether oil mist exists in the target image according to the target texture information, the first classification result, and the second classification result; Determining whether oil mist exists in the target image based on the target texture information, the first classification result, and the second classification result includes: when a texture degree indicated by the target texture information is less than a predetermined degree threshold, the first classification result is used to indicate that the second image is an image captured under the target scene, and when it is determined based on the second classification result that the oil mist concentration exceeds a target concentration threshold, determining that oil mist exists in the target image.

2. The method according to claim 1, characterized in that The detecting the first image to obtain target texture information of the first image includes: Acquire a grayscale image of the first image; performing a denoising operation on the grayscale image of the first image to obtain a denoised image; Obtaining an edge texture map of the denoised image using a non-maximum suppression algorithm according to the gradient strength and gradient direction of the denoised image; The target texture information is obtained according to the edge texture map.

3. The method according to claim 1, characterized in that The first image is a YUV image, the second image is an RGB image, and the third image is an HSV image.

4. The method according to claim 1, wherein Before inputting the second image into the first classification network to obtain the first classification result, the method further includes: Obtaining a classification network model, wherein the classification network model includes a convolutional layer, a pooling layer, a Flatten layer, a fully connected layer, and a Softmax layer, and the convolutional layer, the pooling layer, the Flatten layer, the fully connected layer, and the Softmax layer are connected in sequence; The classification network model is trained using a first image sample set to obtain the first classification network.

5. The method according to claim 4, characterized in that Before inputting the third image into the second classification network to obtain a second classification result, the method further includes: The classification network model is trained using a second image sample set to obtain the second classification network.

6. The method according to claim 4, characterized in that The convolutional layer includes six convolutional layers, wherein in the convolutional layer: The output of the first convolutional layer is connected to the input of the second convolutional layer, and the output of the first convolutional layer is also connected to the input of the third convolutional layer, where the third convolutional layer includes two convolutional layers; The output of the second convolutional layer and the output of the third convolutional layer are both connected to the input of a first merging module, and the first merging module is used to merge the output data of the second convolutional layer and the output data of the third convolutional layer; The output of the first merging module is connected to the input of the fourth convolutional layer and the input of the second merging module, the output of the fourth convolutional layer is connected to the input of the second merging module, and the second merging module is used to merge the output data of the first merging module with the output data of the fourth convolutional layer, and input the merged data into the pooling layer, wherein the fourth convolutional layer includes two convolutional layers.

7. An oil mist determination device, characterized in that: include: An acquisition module is used to acquire a target image acquired by an image acquisition device; A first acquisition module detects a first image to obtain target texture information of the first image, wherein the first image is obtained by transforming the target image; a second obtaining module, configured to input a second image into the first classification network to obtain a first classification result, wherein the second image is obtained by transforming the first image, and the first classification result is used to indicate whether the second image is an image captured in a target scene, wherein the target scene includes an in-vivo scene; a third obtaining module, configured to input a third image into a second classification network to obtain a second classification result, wherein the third image is obtained by transforming the second image, the second classification result is used to indicate the oil mist concentration in the third image, and the color space corresponding to the second image is different from the color space corresponding to the third image; a determination module, configured to determine whether oil mist exists in the target image based on the target texture information, the first classification result, and the second classification result; The determination module can determine whether oil mist exists in the target image based on the target texture information, the first classification result, and the second classification result in the following manner: when the texture degree indicated by the target texture information is less than a predetermined degree threshold, the first classification result is used to indicate that the second image is an image captured under the target scene, and when it is determined based on the second classification result that the oil mist concentration exceeds a target concentration threshold, it is determined that oil mist exists in the target image.

8. A storage medium, characterized in that: The storage medium stores a computer program, wherein the program can be executed by a terminal device or a computer to execute the method described in any one of claims 1 to 6.

9. An electronic device comprising a memory and a processor, characterized in that: A computer program is stored in the memory, and the processor is configured to run the computer program to perform the method according to any one of claims 1 to 6.

Images