A method for generating a water meter water mist image

Abstract

The application discloses a kind of water meter water mist image generation method, including the stage of generating image of adversarial network, specifically is using the idea of generating image to train water mist image generator and discriminator;Contrast learning constraint generation image stage, specifically is to design query encoder and key encoder respectively encode the image block from water meter water mist image and clean water meter image, obtain the feature vector corresponding to each image block;The feature vector obtained from the image block in the same position in two images is regarded as positive sample pair;The feature vector obtained from the image block in different positions in two images is pressed into negative sample queue, and all negative sample feature vectors in negative sample queue are regarded as negative sample pair;One positive sample is calculated infoNCE loss with all negative sample pairs, obtains the infoNCE loss of all positive samples, and the final water mist image generator is obtained by updating water mist image generator in reverse direction.The application effectively reduces training parameters and improves training efficiency.

Description

Technical Field

[0001] This invention relates to the field of image processing, and in particular to a method for generating water mist images from water meters. Background Technology

[0002] With the advancement of smart cities, remote meter reading systems and technologies are attracting increasing attention. Remote meter reading systems can significantly reduce the manpower and material resources required by energy operators and management departments for collecting data on different types of energy such as water, electricity, gas, and heat. They also promote the effective utilization of this data in subsequent big data intelligent analysis and data mining. Smart camera meters utilize image acquisition terminals (such as smartphones and acquisition hardware devices) to acquire meter reading images and use high-performance recognition models for automatic identification, representing a new type of remote meter reading mode. Smart camera meters have the following characteristics: (1) low cost; (2) plug-and-play functionality: no need to disassemble existing meters and pipes; (3) good scalability: convenient for connecting meters of different types and specifications; and (4) objective and accurate image data collected. They have broad application prospects.

[0003] Current smart camera meters, under normal circumstances, can achieve high recognition accuracy by employing high-performance recognition methods and models, such as artificial intelligence algorithms. However, water meters have unique characteristics; they are highly susceptible to water mist contamination, and current computer vision methods for recognizing such water meters exhibit certain errors. To address this issue, the most common approach is to first remove the water mist from the water meter image, creating a clean image, before feeding it into the subsequent detection and recognition system. However, in practical applications, the datasets currently used for training water meter mist removal networks are very limited, hindering effective network training. Consequently, the trained networks are not effectively able to remove water mist from water meters. Therefore, there is an urgent need for methods to expand the water meter mist dataset or augment the network training data to address the current dataset limitations. Summary of the Invention

[0004] In order to overcome the above-mentioned shortcomings and deficiencies of the prior art, the purpose of this invention is to provide a method for generating water mist images of water meters.

[0005] This invention combines generative adversarial learning and contrastive learning, and utilizes a momentum encoder to improve model performance. This not only ensures the model's ability to generate water surface mist images but also significantly reduces training parameters and shortens training time. Furthermore, this invention can augment or supplement data for training water surface mist removal networks, enhancing the effectiveness and reliability of network training.

[0006] The objective of this invention is achieved through the following technical solution:

[0007] A method for generating water mist images from a water meter includes:

[0008] The image generation stage of adversarial networks: specifically:

[0009] Input a clean water meter image into the water mist image generator, and output a water meter water mist image;

[0010] The water mist image from the water meter is input into the discriminator, which outputs a feature map. The feature map is then compared with the true label to calculate the loss and generate an adversarial loss, which is used to update the water mist image generator in reverse.

[0011] Input clean water meter images and water mist images from water meters into the discriminator, calculate the loss with real labels and fake labels respectively, generate adversarial loss, and update the discriminator in reverse;

[0012] The contrastive learning constraint generation image stage specifically includes:

[0013] Design a query encoder and a key encoder to encode image patches from water meter mist images and clean water meter images respectively, and obtain the feature vector corresponding to each image patch;

[0014] The feature vectors obtained from image patches at the same location in two images are considered as positive sample pairs.

[0015] The feature vectors obtained from image patches at different locations in the two images are pushed into the negative sample queue, and all negative sample feature vectors in the negative sample queue are regarded as negative sample pairs.

[0016] The infoNCE loss is calculated for each positive sample and each of the negative sample pairs to obtain the infoNCE loss for all positive samples. This loss is then used to update the water mist image generator to obtain the final water mist image generator.

[0017] Furthermore, by utilizing the outputs of multiple intermediate layers of the query encoder and key encoder, the image generation stage of learning constraints is repeatedly compared and contrasted to calculate the overall infoNCE loss, which is then integrated and used to update the water mist image generator in reverse.

[0018] Furthermore, the main structure of the water mist image generator is a resNet network.

[0019] Furthermore, the discriminator is specifically patchGAN.

[0020] Furthermore, the query encoder consists of a cropped water mist image generator and an MLP layer, sharing parameters with the water mist image generator.

[0021] Furthermore, the key encoder refers to the query encoder finding vectors similar to the query feature vector from the encoded key feature vector. The network structure of the key encoder is the same as that of the query encoder, but the key encoder does not share parameters with the query encoder and adopts a momentum update method.

[0022] Furthermore, the negative sample update process is as follows:

[0023] The negative sample queue stores the negative sample feature vectors calculated each time, and uses all the sample features in it as negative samples for this training.

[0024] The size of the negative sample queue is 65536. If the negative sample queue is not full at this time, the sample feature vector calculated by the current training will continue to be pushed in.

[0025] If the negative sample queue is full, then according to the first-in-first-out characteristic of the queue, the oldest batch of sample features at the head of the queue will be pushed out of the queue, and the most recently obtained sample features will be pushed into the queue.

[0026] Furthermore, the outputs of multiple intermediate layers of the query encoder and key encoder are used to randomly select neurons at the same location as intermediate sample features. This process is repeated in the image generation stage to compare and learn constraints, resulting in multiple infoNCE losses. These losses are then integrated and used to update the water mist image generator in reverse. Specifically:

[0027] The neurons of the corresponding image blocks in the intermediate layer output feature maps of the query encoder and key encoder are used as sample features and input into the corresponding MLP layers to calculate the feature vectors respectively.

[0028] The outputs of multiple intermediate layers are selected and neurons at the same position are selected. The contrastive learning constraint image generation stage is repeated, and finally the overall infoNCE loss is calculated.

[0029] Each time an intermediate layer is selected, a new negative sample queue needs to be set up to dynamically update the sample features of this layer and all the corresponding negative sample features.

[0030] Furthermore, four intermediate layers were selected to assist the network learning.

[0031] Furthermore, the cropped water mist image generator specifically refers to cropping the last few convolutional layers and activation layers of the water mist image generator. Preferably, the last three layers are cropped, namely the last activation layer, the convolutional layer, and the activation layer in the transpose block.

[0032] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0033] (1) The water meter mist image generation method provided by this invention, through the ideas of generative adversarial networks and contrastive learning, combined with momentum encoders, negative sample queues and other techniques, enables the network to be trained on unpaired water meter mist datasets and generate water meter mist images using clean water meter images. The model provided by this invention not only ensures the effectiveness of the model, enabling it to generate water meter mist images, but also effectively reduces training parameters, lowers training time, and improves training efficiency, thus having significant advantages.

[0034] (2) The water meter mist image generation method provided by the present invention enables the network to be trained by unpaired water meter mist datasets, so that it can generate water meter mist images from clean water meter images. The generated mist images can be used as data to augment paired water meter mist datasets, or as data augmentation for subsequent water meter mist removal networks. Both of these methods can improve the effectiveness and reliability of subsequent water meter mist network training. Attached Figure Description

[0035] Figure 1 This is the model structure of the present invention;

[0036] Figure 2 This is the model structure of the generator of this invention;

[0037] Figure 3 This is the model structure of the discriminator of the present invention;

[0038] Figures 4(a) and 4(b) are schematic diagrams of the method for selecting positive and negative samples in this invention;

[0039] Figures 5(a) and 5(b) are schematic diagrams of the query encoder of the present invention and the encoder that can calculate sample features, respectively.

[0040] Figure 6 This is a schematic diagram illustrating how the infoNCE loss is calculated for each intermediate layer in this invention;

[0041] Figure 7 This is a schematic diagram comparing the number of parameters and training time of this invention with other models;

[0042] Figure 8 This is an FID diagram showing the water mist image generated by the present invention and the actual images with and without water mist.

[0043] Figure 9 This diagram illustrates the comparison of the effects of using this invention to assist in training water meter mist removal networks. Detailed Implementation

[0044] The present invention will be further described in detail below with reference to the embodiments, but the implementation of the present invention is not limited thereto.

[0045] Please see Figure 1 A model structure for generating water mist images from water meters includes two steps: Step S1: Image generation stage using adversarial network, and Step S2: Image generation stage using contrastive learning constraints.

[0046] Step S1: The adversarial network image generation stage, specifically: A clean water meter image is input to the water mist image generator, which outputs a water meter mist image. The clean water meter image and the generated water meter mist image are then input into a discriminator for training. The discriminator is a binary classifier. The generative adversarial loss of the discriminator is obtained by calculating the loss between the output and its corresponding label, and this loss is used to update the water mist image discriminator. Next, the loss between the generated water mist image output and its corresponding label is calculated, and the generative adversarial loss of the generator is obtained.

[0047] The preferred process in this embodiment is as follows:

[0048] Step S11: Image I of the clean water meter clean Input to Figure 2 The water mist image generator G shown generates a water meter water mist image I after the image is downsampled, processed by a residual network, and then upsampled. mist ;

[0049] Step S12: Use patchGAN as the discriminator D. The network structure of discriminator D is as follows: Figure 3 As shown, the network ultimately outputs a 30*30 prediction image, where each pixel in the prediction image represents the realism of the corresponding region in the original image, with a value ranging from 0 to 1. The generated water meter mist image I... mist The input is fed into the discriminator, which outputs a 30x30 predicted image. This predicted image is then compared with the true label (a 30x30 predicted image with all values ​​set to 1) to calculate the generative adversarial loss L. GAN Inverse gradient update water mist image generator, L GAN The calculation formula is as follows:

[0050]

[0051] Where n is the number of pixels, in this case n = 256 * 256, y i For the prediction graph, For the corresponding tag, at this time This is a 30*30 predicted label image with all values ​​being 1.

[0052] Step S13: Image I of the clean water meter clean Water mist image from water meter I mist The discriminator is input to obtain the predicted image, which is compared with the real label and the fake label (the fake label is a 30*30 predicted image with all values ​​of 0). The loss is calculated according to the formula in step S12 and the discriminator D is updated by backgrading.

[0053] Specifically: For the structure of the water meter mist generator, please refer to [reference needed]. Figure 2 The structure of the water meter mist generator consists of a convolutional block, a residual block, and a transposed convolutional block. The convolutional block performs downsampling to obtain a high-dimensional feature map, the residual block performs the backbone of the model to learn the high-dimensional feature map, and the transposed convolutional block performs upsampling to restore the high-dimensional feature map to the original image size.

[0054] The convolutional block structure comprises, in sequence, a conv layer, a BN regularization layer, and a ReLU activation layer. The residual block structure comprises, in sequence, a conv layer, a ReLU activation layer, and a conv layer. The transposed convolutional block structure comprises, in sequence, a transposconv layer, a BN regularization layer, and a ReLU activation layer.

[0055] For details on the discriminator structure, please refer to [link / reference]. Figure 3 The discriminator is composed of convolutional blocks. After the input image passes through the discriminator, it outputs a 30*30 prediction image. The structure of the convolutional blocks used by the discriminator can be found in [reference needed]. Figure 3 It consists of convolutional layers, batch regularization layers, and activation layers.

[0056] Step S2: The image generation stage involves contrastive learning constraints. Specifically, a query and key encoder are designed using a water mist image generator and a momentum encoder to encode image patches from different images. Multiple 3x3 image patches at different locations are selected from both the water mist image and the clean water meter image. These patches are fed into their respective encoders to generate corresponding feature vectors. Feature vectors from patches at the same location are used as positive sample pairs, while feature vectors from patches at different locations are pushed into a negative sample queue. All negative sample feature vectors from this queue are then used as negative sample pairs to calculate the infoNCE loss. To improve the network's learning ability, the outputs of multiple intermediate layers of the water mist image generator and momentum encoder are also processed using the same steps to calculate the infoNCE loss. Finally, the infoNCE loss of the entire model is integrated. Combined with the generative adversarial loss obtained in Step S1, the water mist image generator is updated in reverse, resulting in the final water mist image generator, which is then applied to clean water meter images to obtain water mist images.

[0057] In this embodiment, the preferred method for step S2 is as follows:

[0058] Step S21: Image I of clean water meter clean 256 3x3 image patches at different locations were randomly selected, and the water mist image I from the water meter was processed in the same way. mist Select image patches of the same number, location, and size. For specific selection methods, please refer to [link / reference / document / etc.]. Figures 4(a)-4(b)Figure 4(a) shows the water mist image of the water meter, and Figure 4(b) shows the clean water meter image. The same number of image blocks need to be selected for both images, and the selection positions are random.

[0059] Step S22: Image I of water mist from the water meter mist The 256 image blocks are fed into the query encoder E. q The feature vector e corresponding to 256 images is obtained. q ;

[0060] Step S23: Image I of the clean water meter clean The 256 image blocks are fed into the key encoder E. k The feature vector e corresponding to 256 image patches is obtained. k ;

[0061] Step S24: Place I clean and I mist Feature vector e' obtained from image patches at the same location q and e' k Treated as positive sample pairs, since 256 image patches were selected, 256 pairs of positive sample pairs can be obtained, where e' q and e' k All taken from e q and e k They are for I clean and I mist Feature vectors encoded at the same location;

[0062] Step S25: e k The sample data is fed into a negative sample queue of size 65536, and all sample features in the negative sample queue are extracted and compared with the e obtained above. q As negative sample pairs, the infoNCE loss is calculated once for each positive sample pair and all negative sample pairs. For a single sample, there are 256 positive sample pairs; the infoNCE loss is calculated for each of these pairs. The infoNCE loss calculation formula is shown below:

[0063]

[0064] Where, sim(z) i ,z j To calculate the cosine similarity between positive sample pairs, let e be the number of positive sample pairs. q and e k The same position in the sample encodes the features; sim(z) i ,z k To calculate the cosine similarity between negative sample pairs, let e be the number of negative sample pairs. qThis is the sum of all negative samples in the negative sample queue; N is 65536, which is the size of the negative sample queue; log() calculates the logarithm, exp() calculates the exponent, and τ is a hyperparameter, which is 0.1 in this case.

[0065] Step S26: Calculate the infoNCE loss for 256 image patches, integrate all infoNCE losses, and update the generator G using the inverse gradient. The integration formula is as follows:

[0066]

[0067] Where L total The infoNCE loss for integrating 256 image patches.

[0068] Furthermore, in step S22:

[0069] query encoder E q The "query" in the name refers to the fact that, in contrastive learning, this encoder essentially needs to find the vector in the key feature vector that is most similar to the query vector it encodes; hence, it is named a query encoder. q It combines the generator G from the last few layers with an MLP layer;

[0070] For a detailed implementation of encoding sample features, please refer to [link / reference]. Figures 5(a)-5(b) After passing through convolutional layers, each neuron can be mapped to a certain region in the original image. Therefore, when an image is fed into the model and passes through the corresponding convolutional layers, all neurons that can be mapped to the image patch region can be obtained. These neurons are collected and used as intermediate sample features. After the intermediate sample features are fed into the MLP, the feature vector corresponding to the image patch can be obtained.

[0071] query encoder E q The network structure is consistent with that of the water meter mist generator G, and it shares parameters with the water meter mist generator G. The water meter mist generator G and the query encoder E... q It remains consistent at all times, and the parameters are updated as the generator is updated.

[0072] In order to improve the model's learning ability, considering that this model utilizes the properties of intermediate layers, we can consider using the outputs of multiple intermediate layers at the same time, randomly selecting neurons at the same position, repeating the above steps to calculate their corresponding infoNCE loss, as shown in Figure 5(a).

[0073] In this embodiment, in the key encoder:

[0074] (1) key encoder E kThe "key" in the name refers to the key feature vector encoded by this encoder, which is essentially the one being searched during contrastive learning; hence, it's named the "key encoder." Where E... k Network structure and E q completely consistent;

[0075] (2) key encoder E k Not with query encoder E q Shared parameters, key encoder E k Update parameters using the idea of ​​momentum update;

[0076] (3) key encoder E k The parameter update utilizes the idea of ​​a momentum encoder, and its update formula is shown below:

[0077] θ k =m×θ k-1 +(1-m)×θ q

[0078] Where θ k For encoder E k The current parameter, θ k-1 Let θ be the parameter from the previous time step. q For E q The parameter value updated in reverse at the current moment

[0079] In the above formula, the value of m is 0.999. This setting makes the encoder E k The current update of each parameter largely comes from the parameter value of the previous time step, thus ensuring the encoder E... k Consistency in encoder E can effectively avoid encoder E q Learning shortcuts;

[0080] key feature vector e k The method for obtaining the query vector is largely the same as that for obtaining the query vector, the only difference being that it uses a clean water meter image as input, and inputs it into encoder E. k .

[0081] In other words, the network structure of the key encoder is the same as that of the query encoder, but they do not share parameters. The parameters of the key encoder are updated through momentum.

[0082] In this embodiment, the update process for negative samples is as follows:

[0083] (1) The negative sample queue stores the calculated e for each time. k And treat all sample features in the negative sample queue as negative samples for this training;

[0084] (2) The size of the negative sample queue is 65536. If the negative sample queue is not full at this time, the currently trained sample features e will continue to be pushed in. k ;

[0085] (3) If the negative sample queue is full, then according to the first-in-first-out characteristic of the queue, the oldest batch of sample features at the head of the queue will be pushed out of the queue, and the latest sample features will be pushed into the queue.

[0086] Furthermore, such as Figure 6 As shown, the outputs of multiple intermediate layers of the query encoder and key encoder are used to randomly select multiple neurons at the same position as intermediate sample features. This process is repeated in the image generation stage to compare and learn constraints, resulting in multiple infoNCE losses. These losses are then integrated and used to update the water mist image generator in reverse. Specifically:

[0087] (1) Considering that the implementation of this model involves using convolutional neurons to collect intermediate sample features and feeding them into an MLP to obtain sample features, in order to improve the learning ability of the model, the query encoder E can be improved. q and key encoder E k In Figure 5(b), 256 neurons at the same position are randomly selected from the output feature maps of multiple intermediate layers and treated as intermediate sample features. The intermediate sample features of different intermediate layers are then fed into the corresponding MLP layers to calculate the feature vectors e. q_mid and e k_mid ;

[0088] (2)e q_mid and e k_mid Repeat steps S21-S26 above, e q and e k The steps specifically include selecting positive and negative sample pairs and calculating the infoNCE loss to finally obtain L. mid Combining all the loss functions mentioned above, the encoder E... k The final formula for calculating the loss function is shown below;

[0089]

[0090] Among them, L final For the final calculated loss, In this invention, the outputs of four intermediate layers were also selected according to the calculated L. total The corresponding infoNCE loss is calculated using the method described above. total This is the loss function integrated from step S26 above, L GAN This is the generative adversarial loss calculated in step S12.

[0091] (3) For each additional intermediate layer output used, a new negative sample queue needs to be set up and all negative sample features corresponding to the sample features of this layer are dynamically updated. The negative sample queue is updated as described in requirement 6.

[0092] (4) Each time an intermediate layer is selected to assist in computation, a new negative sample queue needs to be set up. The negative sample queues do not affect each other and each stores the negative sample features corresponding to the sample features of its own layer. In this invention, four intermediate layers are selected to assist in network learning.

[0093] The beneficial effects of the water meter mist image generation method provided in this embodiment are as follows:

[0094] The water meter mist image generation method provided in this embodiment utilizes generative adversarial networks and contrastive learning, combined with momentum encoders, negative sample queues, and other techniques. This allows the network to be trained on unpaired water meter mist datasets and generate water meter mist images using clean water meter images. The model provided by this invention ensures its effectiveness in generating water meter mist images, and further details can be found in [reference needed]. Figure 7 The present invention effectively reduces training parameters, shortens training time, and improves training efficiency, which shows that the present invention has certain advantages.

[0095] Furthermore, the FID indicator can be consulted. Figure 8 This indicates that the FID between the water meter mist image generated by the present invention and the real water meter mist image is very close to that between the real water mist image and the real water mist image, indicating that the water meter mist image generated by the present invention is similar to the real water mist image.

[0096] Also, please refer to Figure 9 The beneficial effects of this invention over other inventions are as follows:

[0097] The water meter mist image generation method provided by this invention enables the training of a network using unpaired water meter mist datasets. This network can then generate water meter mist images from clean water meter images. The generated mist images can be used to augment paired water meter mist datasets or as data augmentation for subsequent water meter mist removal networks. Both methods improve the effectiveness and reliability of the subsequent water meter mist removal network training. This can be seen from… Figure 9 The results show that the dataset expanded by this invention improves the performance of the water mist removal network when used for training.

[0098] The above embodiments are preferred embodiments of the present invention, but the embodiments of the present invention are not limited to the embodiments described above. Any changes, modifications, substitutions, combinations, or simplifications made without departing from the spirit and principle of the present invention shall be considered equivalent substitutions and shall be included within the protection scope of the present invention.

Claims

1. A method for generating water mist images from a water meter, characterized in that, include: The image generation stage of adversarial networks: specifically: Input a clean water meter image into the water mist image generator, and output a water meter water mist image; The water mist image from the water meter is input into the discriminator, which outputs a feature map. The feature map is then compared with the true label to calculate the loss and generate an adversarial loss, which is used to update the water mist image generator in reverse. Input clean water meter images and water mist images from water meters into the discriminator, calculate the loss with real labels and fake labels respectively, generate adversarial loss, and update the discriminator in reverse; The contrastive learning constraint generation image stage specifically includes: Design a query encoder and a key encoder to encode image patches from water meter mist images and clean water meter images respectively, and obtain the feature vector corresponding to each image patch; The feature vectors obtained from image patches at the same location in two images are considered as positive sample pairs. The feature vectors obtained from image patches at different locations in the two images are pushed into the negative sample queue, and all negative sample feature vectors in the negative sample queue are regarded as negative sample pairs. The infoNCE loss is calculated for each positive sample and each of the negative sample pairs to obtain the infoNCE loss for all positive samples. This is then used to update the water mist image generator to obtain the final water mist image generator. By utilizing the outputs of multiple intermediate layers of the query encoder and key encoder, the image generation stage of learning constraints is repeatedly compared and contrasted to calculate the overall infoNCE loss, which is then integrated and used to update the water mist image generator in reverse. The query encoder and key encoder output multiple intermediate layers, randomly select multiple neurons at the same position as intermediate sample features, repeat the comparison and learning constraint image generation stage, calculate multiple infoNCE losses, integrate and back-update the water mist image generator, specifically: The neurons of the corresponding image blocks in the intermediate layer output feature maps of the query encoder and key encoder are used as sample features and input into the corresponding MLP layers to calculate the feature vectors respectively. The outputs of multiple intermediate layers are selected and neurons at the same position are selected. The contrastive learning constraint image generation stage is repeated, and finally the overall infoNCE loss is calculated. Each time an intermediate layer is selected, a new negative sample queue needs to be set up to dynamically update the sample features of this layer and all the corresponding negative sample features.

2. The method according to claim 1, characterized in that, The main structure of the water mist image generator is a resNet network.

3. The method according to claim 1, characterized in that, The discriminator is specifically patchGAN.

4. The method according to any one of claims 1-3, characterized in that, The query encoder consists of a cropped water mist image generator and an MLP layer, and shares parameters with the water mist image generator.

5. The method according to any one of claims 1-3, characterized in that, The key encoder refers to the query encoder finding vectors similar to the query feature vector from the encoded key feature vector. The network structure of the key encoder is the same as that of the query encoder, but the key encoder does not share parameters with the query encoder and uses momentum update.

6. The method according to claim 1, characterized in that, The negative sample update process is as follows: The negative sample queue stores the negative sample feature vectors calculated each time, and uses all the sample features in it as negative samples for this training. The size of the negative sample queue is 65536. If the negative sample queue is not full at this time, the sample feature vector calculated by the current training will continue to be pushed in. If the negative sample queue is full, then according to the first-in-first-out characteristic of the queue, the oldest batch of sample features at the head of the queue will be pushed out of the queue, and the most recently obtained sample features will be pushed into the queue.

7. The method according to claim 1, characterized in that, Four intermediate layers were selected to assist the network in learning.

8. The method according to claim 4, characterized in that, The cropped water mist image generator specifically refers to the one that has had some of its convolutional and activation layers removed.

Images