A liquid level detection method based on semantic segmentation
By combining RGB and infrared images with a semantic segmentation-based liquid level detection method, and using a SegNet segmentation network and multimodal information, the detection error caused by changes in illumination and observation position is solved, achieving high-precision and low-cost liquid level detection.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- HEFEI SIWILL INTELLIGENT
- Filing Date
- 2023-04-04
- Publication Date
- 2026-06-05
Smart Images
Figure CN116739972B_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the technical field of liquid level detection, and particularly relates to a liquid level detection method based on semantic segmentation. Background Technology
[0002] Liquid level detection has many applications, and there are relatively many existing technologies, such as:
[0003] 1. (201310282969.8) Liquid level detection method and system
[0004] This invention discloses a liquid level detection method and system. The liquid level detection method includes acquiring a first morphological information image of the interior of a casting container, obtaining scales from the first morphological information image, the scales being used to indicate the height or depth values corresponding to one or more locations within the casting container, acquiring a second morphological information image of the interior of the casting container when the container contains molten metal, and obtaining the liquid level value of the molten metal in the casting container based on the positional relationship between the edge line of the second morphological information image and the scales. The edge line of the liquid surface represents the boundary between the surface of the molten metal and the inner wall of the casting container. This invention observes the liquid level based on a frontal view. Calculations based on traditional image processing algorithms often result in significant errors in the detected liquid level due to variations in lighting and minute changes in the observation position.
[0005] 2. (201910722825.7) Liquid level detection method and system
[0006] This patent provides a visual method and apparatus for liquid level detection. The method includes acquiring an image of the liquid level information of a transparent container to be measured, inputting the liquid level information image into a liquid level detection model, processing the liquid level information image through the liquid level detection model, and outputting a measurement result image including the position information of the liquid level line of the container to be measured. This patent application provides a method for detecting liquid level using machine vision. It inputs the liquid level information image of the container to be measured into a trained liquid level detection model, which processes the image and outputs a measurement result image including the position information of the liquid level line of the container to be measured. This detection method ignores the influence of light emission and shadow on the liquid level and cannot accurately detect liquid level using only traditional visual algorithms.
[0007] 3. (200810023127.X) Liquid level detection device
[0008] This invention discloses a liquid level detection device, comprising at least one magnetic sensor, which is a zero-power magnetic sensor. A vertical track is positioned within the liquid to be measured, opposite the magnetic sensor. A float is located on the liquid surface, and a positioning plate is fixed to the float. One end of the positioning plate is fixed to a magnet, and the other end is mounted within the track. The magnetic sensor, made of a magnetic bistable alloy wire, senses the position of the floating magnet, converting the liquid level into an electrical signal. This signal is then transmitted remotely via a cable. The liquid level is detected by the sensor's sensing of the magnet at a corresponding height. No power supply is required, and there are no mechanical contacts. However, this invention is problematic for sealed liquids where the float cannot be added, and also for some liquids where the float cannot be added. The device is complex to configure and has high maintenance costs. Summary of the Invention
[0009] To enable the model to adapt to various lighting and noise interferences and improve its robustness, this invention proposes a liquid level detection method based on semantic segmentation, the specific scheme of which is as follows:
[0010] A liquid level detection method based on semantic segmentation includes the following steps:
[0011] S1. Training the model: First, collect data, then train the encoder and decoder separately.
[0012] S2. Based on the image and infrared sensor, the liquid level detection area is detected. The side area of the liquid level segment is obtained through the SegNet segmentation network. At the same time, the actual liquid level data is calculated by the ratio of the side area of the liquid level segment to that of the container.
[0013] Specifically, the steps for collecting data are as follows:
[0014] S111. Collect the RGB three-channel images and corresponding infrared images of the liquid level to be detected, and divide the dataset into corresponding training set, validation set and test set. Then use the training set for model training. The training set not only contains the collected image information and infrared information, but also its corresponding liquid level semantic segmentation information.
[0015] S112. Apply color jitter augmentation and Gaussian noise augmentation to the data in the training set, validation set, and test set, and perform consistent random cropping on the input images, heatmaps, and corresponding labeled data.
[0016] Specifically, the training steps for the encoder are as follows:
[0017] S121. Obtain feature F by performing a convolution operation on the input RGB image. cv The obtained feature is then processed by deformable convolution to obtain feature F. tcThe heatmap information is processed through spatially separable convolution and depthwise separable convolution to obtain feature F respectively. sc and feature F dc Feature F tc F sc and feature F dc Concatenate to obtain feature F cc ;
[0018] S122, Feature F cc The input is passed sequentially through cascaded residual convolutional blocks, dilated convolutional blocks, deformable convolutional blocks, and grouped convolutional blocks to obtain feature F. cr F cd F tc and feature F gc ;
[0019] S123, Feature F cr Feature F is obtained by resizing and upsampling. re Feature F cd Feature F is obtained through upsampling. us F tc Deconvolution yields feature F m and feature F gc Feature F is obtained through 3x3 convolution. n In addition, feature F gc Feature F is obtained through the FAB mechanism. fab The FAB mechanism processes the input features through dilated convolutions with a kernel size of 3x3 and a stride of 1, followed by SynBN operations, and finally ReLU activation and Dropout operations.
[0020] S124, Feature F fab With feature F m The feature F is obtained through the concate operation. p Next, feature F p Feature F is obtained by paying attention to the CBAM mechanism. cbam , will feature F cbam With feature F us Concatenate to obtain feature F e ;
[0021] S124, Feature F e Feature F is obtained through the CFGB module. cfgb In the CFGB module, the input feature F e The output feature F is obtained by performing dilated convolution with a kernel size of 3x3 and a stride of 2, followed by SynBN and ReLU, and then Dropout. cfgb ;
[0022] S125, Feature F cfgb With feature F re Feature F is obtained after concatenation. g Then the obtained feature F g Compared with the previously obtained feature F cc Adding them together yields the characteristic F. ccg ;
[0023] S126, Feature F ccg With feature F n The input is fed into the APFM mechanism to obtain the final feature F. encoder .
[0024] Specifically, step S126 is as follows:
[0025] S1261, Feature F ccg Feature F is obtained through 1x1 convolution. a Feature F n Feature F is obtained through upsampling. b Adding the corresponding positions of the two yields the feature F. ab ; additional feature F a Feature F is obtained through 3x3 convolution. k F b Feature F is also obtained through 3x3 convolution. v ,
[0026] S1262, Feature F ab Feature F is obtained through the FRB module. frb The FRB module consists of cascaded 1x1 convolutions and 3x3 convolutions;
[0027] S1263, Feature F frb The features F are obtained by the CAM module and the SSAM module respectively. CAM and feature F SSAM ;
[0028] The CAM module employs a channel attention mechanism, where the input features are processed through average pooling, a 1x1 convolution, ReLU, and another 1x1 convolution, finally connected to a sigmoid activation function to obtain the final output features. The SSAM module, on the other hand, processes the input features through average pooling and max pooling to obtain feature F. ap and feature F mp The features of both are then input into a 1x1 convolution to obtain feature F. apc and feature F mc Then, these two features are concatenated and merged to obtain feature F. am And the feature F is obtained by using the activation function softmax.cs ; feature F apc With feature F cs Multiplication yields characteristic F acs , feature F cs With feature F mc Multiplication yields characteristic F ms ; feature F acs With feature F ms The features F are obtained by summing the results and then scaling them up. sf The input is used to obtain features F through convolution operations. c Finally, through formula F SSAM =αF SFS +(1-α)F c Obtain feature F ssam , where α is the trainable parameter weight of the model;
[0029] S1264, will obtain feature F CAM With feature F k Multiplication yields characteristic F q Feature F SSAM With feature F v Multiplication yields characteristic F t The final feature F q With feature F t The final output feature F of the encoder is obtained by concatenation. encoder .
[0030] Specifically, the features in the encoder are decoded to form a decoder model; the specific steps are as follows:
[0031] S131, convert the final output feature F of the encoder. encoder Two features are obtained through upsampling and deconvolution operations, respectively.
[0032] S132. After adding the two features, perform bilinear interpolation to enlarge them through the Rescale operation, then perform deconvolution and BN operations, and finally enlarge them to the original image size through 1x1 convolution and resize to obtain the final semantic segmentation result.
[0033] Specifically, the model consisting of the decoder and encoder calculates the loss function by comparing the predicted semantic segmentation result with the labeled semantic segmentation result using the DICE loss function.
[0034] The beneficial effects of this invention are as follows:
[0035] 1. This patent uses multimodal methods, including not only image processing but also infrared imaging, to detect liquid levels, thereby improving the accuracy of detection.
[0036] 2. Based on this task, this patent invented SegNet for liquid level detection, which can detect liquid level from any angle. Attached Figure Description
[0037] Figure 1 This is a flowchart of the encoder training process in this method.
[0038] Figure 2 This is a flowchart of the FAB mechanism.
[0039] Figure 3 This is a flowchart of the CFGB module.
[0040] Figure 4 This is a flowchart of the APFM mechanism.
[0041] Figure 5 This is a flowchart of the CAM module.
[0042] Figure 6 This is a flowchart of the SSAM module.
[0043] Figure 7 This is a diagram of the decoder model. Detailed Implementation
[0044] like Figure 1 As shown, a liquid level detection method based on semantic segmentation includes the following steps:
[0045] S1. Training the model; first, collect data, then train the encoder and decoder separately; details are as follows:
[0046] S11. The specific steps for collecting data are as follows:
[0047] S111. Collect the RGB three-channel images and corresponding infrared images of the liquid level to be detected, and divide the dataset into corresponding training set, validation set and test set. Then use the training set for model training. The training set not only contains the collected image information and infrared information, but also its corresponding liquid level semantic segmentation information.
[0048] S112. Color jitter augmentation and Gaussian noise augmentation are applied to the data in the training set, validation set and test set. At the same time, the input images, heatmaps and corresponding labeled data are uniformly randomly cropped to make the model more robust.
[0049] S12. The training steps for the encoder are as follows:
[0050] S121. Obtain feature F by performing a convolution operation on the input RGB image. cv The obtained feature is then processed by deformable convolution to obtain feature F. tcThe heatmap information is processed through spatially separable convolution and depthwise separable convolution to obtain feature F respectively. sc and feature F dc Feature F tc F sc and feature F dc Concatenate to obtain feature F cc ;
[0051] S122, Feature F cc The input is sequentially passed through cascaded residual convolutional blocks, dilated convolutional blocks, deformable convolutional blocks, and grouped convolutional blocks to obtain feature F. cr F cd F tc and feature F gc ;
[0052] S123, Feature F cr Feature F is obtained by resizing and upsampling. re Feature F cd Feature F is obtained through upsampling. us F tc Deconvolution yields feature F m and feature F gc Feature F is obtained through 3x3 convolution. n In addition, feature F gc Feature F is obtained through the FAB mechanism. fab The FAB mechanism processes the input features through dilated convolutions with a kernel size of 3x3 and a stride of 1, followed by SynBN operations, and finally ReLU activation and Dropout operations.
[0053] S124, Feature F fab With feature F m The feature F is obtained through the concate operation. p Next, feature F p Feature F is obtained by paying attention to the CBAM mechanism. cbam , will feature F cbam With feature F us Concatenate to obtain feature F e ;
[0054] S124, Feature F e Feature F is obtained through the CFGB module. cfgb Specifically, in the CFGB module, the input feature F e The output feature F is obtained by performing dilated convolution with a kernel size of 3x3 and a stride of 2, followed by SynBN and ReLU, and then Dropout. cfgb ;
[0055] S125, Feature F cfgb With feature F re Feature F is obtained after concatenation. g Then the obtained feature F g Compared with the previously obtained feature F cc Adding them together yields the characteristic F. ccg ;
[0056] S126, Feature F ccg With feature F n The input is fed into the APFM mechanism to obtain the final feature F. encoder The specific steps are as follows:
[0057] S1261, Feature F ccg Feature F is obtained through 1x1 convolution. a Feature F n Feature F is obtained through upsampling. b Adding the corresponding positions of the two yields the feature F. ab ; additional feature F a Feature F is obtained through 3x3 convolution. k F b Feature F is also obtained through 3x3 convolution. v ,
[0058] S1262, Feature F ab Feature F is obtained through the FRB module. frb The FRB module consists of cascaded 1x1 convolutions and 3x3 convolutions;
[0059] S1263, Feature F frb The features F are obtained by the CAM module and the SSAM module respectively. CAM and feature F SSAM ;
[0060] The CAM module employs a channel attention mechanism, where the input features are processed through average pooling, a 1x1 convolution, ReLU, and another 1x1 convolution, finally connected to a sigmoid activation function to obtain the final output features. The SSAM module, on the other hand, processes the input features through average pooling and max pooling to obtain feature F. ap and feature F mp The features of both are then input into a 1x1 convolution to obtain feature F. apc and feature F mc Then, these two features are concatenated and merged to obtain feature F. am And the feature F is obtained by using the activation function softmax. cs . The feature Fapc With feature F cs Multiplication yields characteristic F acs , feature F cs With feature F mc Multiplication yields characteristic F ms . The feature F acs With feature F ms The features F are obtained by summing the results and then scaling them up. sf The input is processed through a convolution operation to obtain the feature F. c Finally, through formula F SSAM =αF SFS +(1-α)F c Obtain feature F SSAM , where α is the trainable parameter weight of the model.
[0061] S1264, will obtain feature F CAM With feature F k Multiplication yields characteristic F q Feature F SSAM With feature F v Multiplication yields characteristic F t The final feature F q With feature F t The final output feature F of the encoder is obtained by concatenation. encoder .
[0062] S13. Decode the features in the encoder to form a decoder model; the specific steps are as follows:
[0063] S131, convert the final output feature F of the encoder. encoder Two features are obtained through upsampling and deconvolution operations, respectively.
[0064] S132. After adding the two features, perform bilinear interpolation to enlarge them through the Rescale operation, then perform deconvolution and BN operations, and finally enlarge them to the original image size through 1x1 convolution and resize to obtain the final semantic segmentation result.
[0065] In this application, the model consisting of the decoder and encoder selects the predicted semantic segmentation result and the labeled semantic segmentation result to calculate the loss function using the Dice loss function.
[0066] S2. Based on images and infrared sensors, the liquid level detection area is detected. Through the SegNet segmentation network, we can obtain the lateral area of the liquid level segmentation. At the same time, the actual liquid level data is calculated by the ratio of the lateral area of the liquid level to that of the container, which facilitates automated liquid level monitoring and management by maintenance personnel.
[0067] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.
Claims
1. A liquid level detection method based on semantic segmentation, characterized in that, Includes the following steps: S1. Training the model: First, collect data, then train the encoder and decoder separately. S2. Based on the image and infrared sensor, the liquid level detection area is detected. The side area of the liquid level segment is obtained through the SegNet segmentation network. At the same time, the actual liquid level data is calculated by the ratio of the side area of the liquid level segment to that of the container. The specific steps for collecting data are as follows: S111. Collect the RGB three-channel images and corresponding infrared images of the liquid level to be detected, and divide the dataset into corresponding training set, validation set and test set. Then use the training set for model training. The training set not only contains the collected image information and infrared information, but also its corresponding liquid level semantic segmentation information. S112. Apply color jitter augmentation and Gaussian noise augmentation to the data in the training set, validation set and test set, and perform consistent random cropping on the input images, heatmaps and corresponding labeled data. The training steps for the encoder are as follows: S121. Obtain features from the input RGB image through a convolution operation. The obtained feature is then processed by deformable convolution to obtain the feature. The heatmap information is used to obtain features through spatially separable convolution and depthwise separable convolution, respectively. and features ,feature , and features Features are obtained by concatenating ; S122, Features The input is sequentially passed through cascaded residual convolutional blocks, dilated convolutional blocks, deformable convolutional blocks, and grouped convolutional blocks to obtain features. , and features ; S123, Features Features are obtained through resizing and upsampling. ,feature Features are obtained through upsampling , Deconvolution yields features and features Features are obtained through 3x3 convolution. In addition, features Features are obtained through the FAB mechanism. The FAB mechanism processes the input features through dilated convolutions with a kernel size of 3x3 and a stride of 1, followed by SynBN operations, and finally ReLU activation and Dropout operations. S124, Features With features Features are obtained through the concate operation. Next, the features Features are obtained through attention CBAM mechanism. , will feature With features Features are obtained by concatenating ; S124, Features Features are obtained through the CFGB module. In the CFGB module, the input features The output features are obtained by performing dilated convolution with a kernel size of 3x3 and a stride of 2, followed by SynBN and ReLU, and then Dropout. ; S125, Features With features Features are obtained after concatenation. Then the obtained features Features obtained previously Adding them together yields the features. ; S126, Features With features The final features are obtained by inputting them into the APFM mechanism. ; Step S126 is as follows: S1261, Features Features are obtained through 1x1 convolution. ,feature Features are obtained through upsampling Adding the corresponding positions of the two yields the feature. ; Other features Features are obtained through 3x3 convolution. , Features are also obtained through 3x3 convolution. , S1262, Features Features are obtained through the FRB module. The FRB module consists of cascaded 1x1 convolutions and 3x3 convolutions; S1263, Features Features were obtained through the CAM module and the SSAM module respectively. and features ; The CAM module employs a channel attention mechanism, where the input features are processed through average pooling, a 1x1 convolution, then ReLU, another 1x1 convolution, and finally a sigmoid activation function to obtain the final output features. The SSAM module, on the other hand, processes the input features through average pooling and max pooling respectively to obtain the output features. and features The features from both are then input into a 1x1 convolution to obtain the features. and features Then, the two features are concatenated and merged to obtain the feature. Features are obtained through the softmax activation function. ; Features With features Multiplication yields characteristics , will feature With features Multiplication yields characteristics ; Features With features The features are obtained by adding the features together and then scaling them up using a scale operation. Features are obtained from the input through convolution operations. Ultimately, through the formula Obtain features ,in These are the weights of the trainable parameters of the model; S1264, Features will be obtained With features Multiplication yields characteristics ,feature With features Multiplication yields characteristics final features With features The final output features of the encoder are obtained by concatenation. ; The features in the encoder are decoded to form a decoder model; the specific steps are as follows: S131, The features output by the encoder at the end Two features are obtained through upsampling and deconvolution operations, respectively. S132. After adding the two features, perform bilinear interpolation to enlarge them through the Rescale operation, then perform deconvolution and BN operations, and finally enlarge them to the original image size through 1x1 convolution and resize to obtain the final semantic segmentation result.
2. The liquid level detection method based on semantic segmentation according to claim 1, characterized in that, The model consisting of the decoder and encoder calculates the loss function by comparing the predicted semantic segmentation result with the labeled semantic segmentation result using the Dice loss function.