A method and system for evaluating low-quality images in mine machine vision

Abstract

A method and system for evaluating the quality of low-quality images in mine machine vision, the method includes: designing prompt words related to image distortion type and quality prediction, obtaining distortion distribution probability and quality prediction probability through CLIP model, and performing representative ranking of samples in new scenes in terms of distortion distribution and model prediction uncertainty; using ranking aggregation method, selecting some samples in new scenes for labeling; obtaining quality feature vectors of selected samples of new scenes and all samples of old scenes, and selecting easy-to-transfer samples of old scenes by similarity measurement algorithm; training the selected samples of new and old scenes to complete the construction of evaluation model. The system includes a camera, a sample selector and a feature transfer learning trainer. The present invention reduces the distortion diversity in new scenes while reducing the cost of labeled samples, reduces the risk of overfitting of the model to adapt to new scenes, and improves the prediction ability of the model and the accuracy of image quality evaluation.

Description

Technical Field

[0001] The invention relates to a method and system for evaluating the quality of low-quality images in mine machine vision, and belongs to the technical field of image quality evaluation. Background Art

[0002] The mine environment, due to its unique operating conditions, not only poses a threat to safe mining, but also has a significant impact on the quality of image acquisition. The combination of dusty air, humid climate and insufficient lighting makes the image acquisition process extremely complicated. Among them, the presence of dust will form deposits on the lens, reducing the clarity of the image, while the high temperature and humid environment will affect the stability and performance of the equipment. In addition, the limited space and poor ventilation of the mine make it difficult for light to evenly cover the entire area, which leads to obvious uneven lighting and shadows in the images taken at specific locations, further reducing the quality of the image, making it difficult for the acquired target image to meet the high standards required for subsequent image processing, especially in the critical tasks of image analysis and recognition. Low-quality images may directly affect the effectiveness of key visual tasks such as target detection and behavior recognition, thereby threatening the safe operation of the mine and the lives of workers. With the continuous improvement of the demand for intelligent mine management and safety monitoring, it is particularly important to ensure image quality. Therefore, it is crucial to conduct a comprehensive assessment of the quality of images collected in mines.

[0003] Image quality assessment (IQA) refers to quantifying the degree of image distortion by analyzing the relevant characteristics of image signals. When existing image quality assessment models are applied to actual new scenes, they often have difficulty handling the distortion diversity and poor generalization in new scenes. Researchers often have to annotate the images of new scenes and fine-tune the models to adapt to the new scenes. However, due to the high cost of sample annotation, there is an urgent need to reduce the number of labeled samples. Since deep learning is a data-driven learning model, when there are fewer new scene samples, the model may overfit to the new scene data, which in turn affects the generalization of the model and the reliability of actual application in new scenes. Summary of the invention

[0004] The purpose of the invention is to provide a method and system for evaluating the quality of low-quality images in mine machine vision. The method and system can reduce the problem of distortion diversity in new scenes while reducing the cost of a large number of labeled samples. It can also reduce the risk of overfitting when the model adapts to new scenes, effectively improve the prediction ability of the model, improve the accuracy of image quality evaluation, and ensure safe production in mines.

[0005] In order to achieve the above object, the present invention provides a method for evaluating the quality of low-quality images in mine machine vision, comprising the following steps:

[0006] S1. Design prompts related to the distribution of image distortion types, obtain the probability of distortion distribution by comparing the language-image pre-training model, use clustering methods to classify the samples in the new scene in terms of distortion distribution and rank their representativeness, then design prompts related to image quality, use the method of Dropout operation on the model, and rank the samples in the new scene in terms of contribution from the perspective of the model's prediction uncertainty;

[0007] S2. Use the Dowdall ranking aggregation method, combined with the ranking order of the two dimensions of distortion distribution and prediction uncertainty, to select some representative samples in the new scene, and manually annotate the selected samples to obtain corresponding quality score labels; use the pre-trained model method to fine-tune the pre-trained model using the selected samples, and then repeat the first step to the second step until the number of selected samples in the new scene reaches the specified number of small samples, that is, 25% of the total number of samples;

[0008] S3. By comparing the Image Encoder module in the Language-Image Pre-training (CLIP) model, the quality feature vectors of the selected samples of the new scene and all the samples of the old scene are obtained. At the level of the quality feature space, the similarity of the samples in the new and old scenes is calculated using a similarity measurement algorithm, and the samples in the old scene that are easy to transfer are selected;

[0009] S4. The samples obtained in step S3 are trained based on the feature transfer learning strategy and the method of fine-tuning the model. That is, the model is pre-trained using the samples that are easy to transfer selected from the old scene, and then the new scene samples are used to fine-tune the parameters of the pre-trained model to complete the construction of the image quality evaluation model.

[0010] Furthermore, the specific process of S1 is as follows:

[0011] S1.1. Design prompts related to the distribution of image distortion types, including brightness distortion, clarity distortion, noise distortion, color distortion, and contrast distortion, and set the corresponding prompts as follows: Group 1: the photo features a good / poor distribution of light and shadow; Group 2: sharp / blurry photo; Group 3: the noise enhances / reduces the overall quality of this photo; Group 4: colorful / dull photo; Group 5: good / poor contrast photo;

[0012] S1.2. Use the contrastive language-image pre-training model to perform probability prediction and cluster analysis on the distortion distribution of all images in the new scene: Note the prompts in S1.1 as Where n = 1, 2, ..., 5, m = 0, 1, represents the nth group of prompts and the mth category in each group of prompts; represents sharp photo; let the image and text encoders in the contrastive language-image pre-training model be E img and E text , then the i-th image I obtained by the model output i The distortion distribution probability is:

[0013]

[0014] In the formula, F is the image feature obtained by the image encoder, is the distorted text feature obtained by the text encoder, is the cosine similarity between the distorted text features and image features of image I, e (·) represents the natural exponential function, represents the probability of the nth type of distortion of the i-th image, i = 1, 2, ..., N, N represents the total number of new scene images;

[0015] According to K-means clustering, we obtain the k distortion distribution class centers of the entire data set and the category to which each image belongs. Where k = 1, 2, ..., 7, and the distance between each image distortion distribution and the centroid within the class is calculated as:

[0016]

[0017] In the formula, represents the five distortion type probabilities of the k-th centroid obtained by the clustering algorithm, Represents the Euclidean distance between the i-th image and the centroid of its k-th class; the smaller the distance, the more similar the image is to the overall data distortion distribution, that is, the more representative it is;

[0018] S1.3. Based on the Euclidean distance calculated in S1.2, without considering the category factor of the clustering algorithm, the image data ranking is obtained. That is, the smaller the Euclidean distance, the higher the ranking;

[0019] S1.4. Use the contrastive language-image pre-training model to predict the quality scores of all images in the new scene: By designing prompts related to quality prediction: high / low quality photo, denoted as P m, where m represents the category in prompts; for example, P0 is goodphoto; let the image and text encoders in the contrastive language-image pre-training model be E img and E text , then the i-th image I obtained by the model output i The quality prediction probability value is:

[0020]

[0021] Where, T m is the quality text feature obtained by the text encoder, S m is the cosine similarity between the quality text feature and the image feature of image I, and e(·) represents the natural exponential function Represents the quality prediction probability value of the i-th image;

[0022] By performing a dropout operation, that is, randomly setting some parameters of some layers of the contrast language-image pre-training model to zero, we obtain the quality prediction probability values ​​of the other two sets of new scene images. and and quality prediction probability value Common composition Where j = 0, 1, 2;

[0023] S1.5. Based on the four sets of quality prediction probability values ​​obtained in S1.4, calculate the standard deviation of the prediction probability values ​​of the new scene image as the prediction uncertainty, that is, the prediction uncertainty PD of each image i The calculation formula is:

[0024]

[0025] S1.6. Prediction uncertainty PD calculated according to S1.5 i , get the model-wise ranking of the image That is, the greater the prediction uncertainty, the higher the ranking.

[0026] Furthermore, the specific process of S2 is as follows:

[0027] S2.1. Use the Dowdall ranking aggregation method to select some samples in the new scene by combining the ranking of image data and model:

[0028]

[0029] In the formula, argsort(·) sorts in ascending order and returns the corresponding index value, s newIt represents some samples corresponding to the index value in the selected new scene, and manually annotates the selected samples to obtain the quality score of the selected samples;

[0030] S2.2. After repeating S2.1 for z times, the estimated number of samples finally selected in the new scene is obtained:

[0031]

[0032] In the formula, is the sample selected in round i, Sample new For all samples selected from the new scene, manually label these samples and obtain corresponding quality scores;

[0033] Furthermore, the specific process of S3 is as follows:

[0034] S3.1. Use the image encoder in the contrast language-image pre-training model to obtain the quality features of the selected new scene samples and all old scene samples, and calculate the cosine similarity of the quality features. The quality features of the new and old images and the cosine similarity obtained by the model output are:

[0035]

[0036] Where D old is all samples of the old scene, F new and F old are the corresponding new and old image quality features, S F is the cosine similarity matrix;

[0037] S3.2, taking each sample selected in the new scene as the standard, select the three corresponding samples with the highest similarity in the old scene:

[0038] S old =[D old |Top3(S F )] (8)

[0039] In the formula, S old These are samples in the old scene that are highly similar to the samples selected in the new scene and are easy to transfer.

[0040] Furthermore, the specific process of S4 is as follows:

[0041] S4.1. Pre-train the model using the samples selected from the old scene that are easy to transfer. That is, set the training parameters and use the contrastive language-image pre-training model to train on the easy-to-transfer dataset. Meanwhile, continuously test on the unselected data from the old scene to evaluate the model performance, and then obtain a trained model with higher performance and save it.

[0042] S4.2. Use the pre-trained model to fine-tune the samples selected for the new scene. That is, first load the pre-trained model parameters and set the structural layer where the model adjustable parameters are located. Then use the labeled samples selected for the new scene for training to complete the construction of the image quality evaluation model.

[0043] A low-quality image quality evaluation system for mine machine vision includes a camera, a sample selector and a feature transfer learning trainer, wherein the feature transfer learning trainer is provided with a similarity measurement module and a fine-tuning module;

[0044] The camera and its memory card are used to collect image sample data in new scenes and to save sample data collected and annotated in old scenes;

[0045] The sample selector is used to receive the image sample data collected in the new scene, and send the sample data to two different prompt word Prompts modules to obtain multiple distortion distribution probabilities and multiple prediction quality scores respectively, and then obtain the corresponding distortion distribution distance and quality prediction variance through the clustering and statistics modules; through the Dowdall ranking aggregation module, the function of selecting representative samples in the new scene is realized;

[0046] The feature transfer learning trainer is used to receive samples selected and marked in the new scene and all marked samples in the old scene. The samples first enter the similarity measurement module to calculate the similarity between the samples in the old scene and the selected samples in the new scene. The feature transfer learning trainer selects samples that are easy to transfer in the old scene according to the similarity, and then sends the selected samples in the new scene and the easy-to-transfer samples in the old scene to the comparative language-image pre-training model for training to complete the construction of the image quality evaluation system.

[0047] The present invention selects samples by combining image distortion distribution and model prediction uncertainty, and uses CLIP model as the initial model to achieve zero-sample prediction of image distortion distribution and image quality in new scenes; Dowdall's ranking aggregation method is used to fuse data and model, and the final selection of samples is performed through multiple rounds of model training. The present invention uses a similarity measurement algorithm in the quality feature space to select samples with high similarity to the samples selected in the new scene in the old scene, which not only expands the sample data, but also further enriches the distortion types of samples in the new scene; the strategy based on feature transfer learning and the fine-tuning model are used to complete the training of samples selected from the old and new scenes, reducing the damage caused by too few training samples to model training. The present invention reduces the problem of distortion diversity in new scenes while reducing the cost of a large number of labeled samples, and also reduces the risk of overfitting of the model to adapt to new scenes, effectively improving the prediction ability of the model and the accuracy of image quality evaluation, and ensuring safe production in mines. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 It is a work flow chart of the method of the present invention;

[0049] Figure 2 It is a structural diagram of the system of the present invention;

[0050] Figure 3 is a graph of ablation experiment results of an embodiment of the present invention on the LIVEW dataset;

[0051] Figure 4 It is a graph showing the ablation experiment results of the model generalization ability on the RBID and CID datasets according to the embodiment of the present invention. DETAILED DESCRIPTION

[0052] The present invention will be further described below in conjunction with the accompanying drawings.

[0053] like Figure 1 As shown, a method for evaluating the quality of low-quality images in mine machine vision includes the following steps:

[0054] S1. Design prompts related to the distribution of image distortion types, obtain the probability of distortion distribution by comparing the language-image pre-training model, use clustering methods to classify the samples in the new scene in terms of distortion distribution and rank their representativeness, then design prompts related to image quality, use the method of Dropout operation on the model, and rank the samples in the new scene in terms of contribution from the perspective of the model's prediction uncertainty;

[0055] S2. Use the Dowdall ranking aggregation method, combined with the ranking order of the two dimensions of distortion distribution and prediction uncertainty, to select some representative samples in the new scene, and manually annotate the selected samples to obtain the corresponding quality score labels; use the pre-trained model to fine-tune the pre-trained model using the selected samples, and then repeat the first step to the second step until the number of samples selected in the new scene reaches the specified number of small samples;

[0056] S3. By comparing the Image Encoder module in the Language-Image Pre-training (CLIP) model, the quality feature vectors of the selected samples of the new scene and all the samples of the old scene are obtained. At the level of the quality feature space, the similarity of the samples in the new and old scenes is calculated using a similarity measurement algorithm, and the samples in the old scene that are easy to transfer are selected;

[0057] S4. The samples obtained in step S3 are trained based on the feature transfer learning strategy and the method of fine-tuning the model. That is, the model is pre-trained using the samples that are easy to transfer selected from the old scene, and then the new scene samples are used to fine-tune the parameters of the pre-trained model to complete the construction of the image quality evaluation model.

[0058] like Figure 2 As shown, the system of the present invention includes a sample selection module based on data and model and a training module based on feature transfer learning. Among them, the sample selection module based on data and model adopts a ranking aggregation method to fuse the two aspects of distortion distribution and prediction uncertainty to select representative and contributing samples in the new scene. Specifically, the image and prompts considering different aspects are first input into the CLIP model to obtain the corresponding prediction results. In terms of distortion distribution, the clustering method is used to rank the images. In terms of prediction uncertainty, the Dropout strategy is used to obtain and calculate the standard deviation to rank the images. Then, the Dowdall ranking aggregation method is used to combine the ranking of data and model. After repeating the above operations for many times, the representative samples of the new scene are finally selected; the training module based on feature transfer learning includes two parts, namely, the selection of easy-to-transfer samples of the old scene and model training. The selection of samples in the old scene is based on the similarity measurement algorithm to calculate the similarity of all samples in the old scene with the selected samples in the new scene, and sort and select them. The model training adopts a method of training the selected samples based on feature transfer learning strategy and fine-tuning model. That is, the model is first pre-trained with easy-to-transfer samples from the old scene, and then the model is fine-tuned on the selected samples of the new scene to finally complete the construction of the quality evaluation model.

[0059] Embodiment: The present invention uses 25% of the images selected from the real distortion dataset LIVEW and the easy-to-transfer samples in KonIQ-10k to train the model, and compares it with the existing algorithm. The evaluation indicators are Pearson linear correlation coefficient (PLCC) and Spearman rank-order correlation coefficient (SRCC). The closer the two indicators are to 1, the better the performance of the model. The results are shown in the following table:

[0060]

[0061] It can be seen from the above table that, compared with the existing SOTA (State of the Art) algorithm, the PLCC and SRCC coefficients of the algorithm of the present invention are closer to 1, that is, the algorithm of the present invention has achieved the best small sample prediction performance for new scenarios.

[0062] In order to prove the effectiveness of the present invention, an ablation experiment was conducted. The present invention adopts a common strategy, randomly selects images and uses a training method and network model consistent with the algorithm of the present invention for comparison. The results are as follows: Figure 3 As shown. Figure 3It can be seen that the few-sample algorithm for selecting new and old scene samples of the present invention has better prediction ability;

[0063] Ablation experiment of model generalization ability on RBID and CID2013 datasets. Consistent with the above, the present invention uses randomly selected samples and trained with the same training method and network model to perform cross-dataset tests on RBID and CID2013 datasets, and uses the obtained results as baseline model results. Similarly, the present invention also directly performs cross-database tests on these two datasets, and the obtained results are as follows Figure 4 As shown. Figure 4 It can be seen that the algorithm of the present invention has excellent cross-dataset test performance. In summary, the above experiments have proved the effectiveness of the present invention. While alleviating the problem of distortion diversity in new scenes, the present invention also reduces the risk of overfitting when the model is adapted to new scenes, thereby effectively improving the prediction ability of the model.

Claims

1. A method for evaluating low-quality images in mine machine vision, characterized in that: The steps include: S1. Design prompts related to the distribution of image distortion types, obtain the probability of distortion distribution by comparing the language-image pre-training model, use clustering methods to classify the distortion distribution of samples in new scenes and rank their representativeness, then design prompts related to image quality, use the method of Dropout operation on the model, and rank the contribution of samples in new scenes from the perspective of model prediction uncertainty; S2. Use the Dowdall ranking aggregation method, combined with the ranking order of the two dimensions of distortion distribution and prediction uncertainty, to select some representative samples in the new scene, and manually annotate the selected samples to obtain the corresponding quality score labels; use the pre-trained model to fine-tune the pre-trained model using the selected samples, and then repeat the first step to the second step until the number of samples selected in the new scene reaches the specified number of small samples; S3. By comparing the Image Encoder module in the language-image pre-training model, the quality feature vectors of the selected samples of the new scene and all the samples of the old scene are obtained. At the level of the quality feature space, the similarity of the samples in the new and old scenes is calculated using a similarity measurement algorithm, and the samples in the old scene that are easy to transfer are selected; S4. The samples obtained in step S3 are trained based on the feature transfer learning strategy and the method of fine-tuning the model. That is, the model is pre-trained using the samples that are easy to transfer selected from the old scene, and then the new scene samples are used to fine-tune the parameters of the pre-trained model to complete the construction of the image quality evaluation model.

2. The method for evaluating low-quality images in mine machine vision according to claim 1, characterized in that: The specific process of S1 is as follows: S1.

1. Design prompts related to the distribution of image distortion types, including brightness distortion, clarity distortion, noise distortion, color distortion, and contrast distortion, and set the corresponding prompts as follows: Group 1: the photo features a good / poor distribution of light and shadow; Group 2: sharp / blurry photo; Group 3: the noise enhances / reduces the overall quality of this photo; Group 4: colorful / dull photo; Group 5: good / poor contrast photo; S1.

2. Use the contrastive language-image pre-training model to perform probability prediction and cluster analysis on the distortion distribution of all images in the new scene: Note the prompts in S1.1 as Where n = 1, 2, ..., 5, m = 0, 1, representing the nth group of prompts and the mth category in each group of prompts; let the image and text encoders in the contrastive language-image pre-training model be E img and E text , then the i-th image I obtained by the model output i The distortion distribution probability is: In the formula, F is the image feature obtained by the image encoder, is the distorted text feature obtained by the text encoder, is the cosine similarity between the distorted text features and image features of image I, e (·) represents the natural exponential function, represents the probability of the nth type of distortion of the i-th image, i = 1, 2, ..., N, N represents the total number of new scene images; According to K-means clustering, we obtain the k distortion distribution class centers of the entire data set and the category to which each image belongs. Where k = 1, 2, ..., 7, and the distance between each image distortion distribution and the centroid within the class is calculated as: In the formula, represents the five distortion type probabilities of the k-th centroid obtained by the clustering algorithm, represents the Euclidean distance between the i-th image and the centroid of its k-th class; S1.

3. Based on the Euclidean distance calculated in S1.2, without considering the category factor of the clustering algorithm, the image data ranking is obtained. That is, the smaller the Euclidean distance, the higher the ranking; S1.

4. Use the contrastive language-image pre-training model to predict the quality scores of all images in the new scene: By designing prompts related to quality prediction: high / low quality photo, denoted as P m , where m represents the category in prompts; let the image and text encoders in the contrastive language-image pre-training model be E img and E text , then the i-th image I obtained by the model output i The quality prediction probability value is: Where, T m is the quality text feature obtained by the text encoder, S m is the cosine similarity between the quality text feature and the image feature of image I, e (·) represents the natural exponential function, Represents the quality prediction probability value of the i-th image; By performing a dropout operation, that is, randomly setting some parameters of some layers of the contrast language-image pre-training model to zero, we obtain the quality prediction probability values ​​of the other two sets of new scene images. and and quality prediction probability value Common composition Where j = 0, 1, 2; S1.

5. Based on the four sets of quality prediction probability values ​​obtained in S1.4, calculate the standard deviation of the prediction probability values ​​of the new scene image as the prediction uncertainty, that is, the prediction uncertainty PD of each image i The calculation formula is: S1.

6. Prediction uncertainty PD calculated according to S1.5 i , get the model-wise ranking of the image That is, the greater the prediction uncertainty, the higher the ranking.

3. The method for evaluating low-quality images in mine machine vision according to claim 2, characterized in that: The specific process of S2 is as follows: S2.

1. Use the Dowdall ranking aggregation method to select some samples in the new scene by combining the ranking of image data and model: In the formula, argsort(·) sorts in ascending order and returns the corresponding index value, s new It represents some samples corresponding to the index value in the selected new scene, and manually annotates the selected samples to obtain the quality score of the selected samples; S2.

2. After repeating S2.1 for z times, the estimated number of samples finally selected in the new scene is obtained: In the formula, is the sample selected in round i, Sample new For all samples selected from the new scene, these samples are manually labeled and the corresponding quality scores are obtained.

4. The method for evaluating low-quality images in mine machine vision according to claim 3, characterized in that: The specific process of S3 is as follows: S3.

1. Use the image encoder in the contrast language-image pre-training model to obtain the quality features of the selected new scene samples and all old scene samples, and calculate the cosine similarity of the quality features. The quality features of the new and old images and the cosine similarity obtained by the model output are: Where D old is all samples of the old scene, F new and F old are the corresponding new and old image quality features, S F is the cosine similarity matrix; S3.2, taking each sample selected in the new scene as the standard, select the three corresponding samples with the highest similarity in the old scene: S old =[D old |Top3(S F )] (8) In the formula, S old These are samples in the old scene that are highly similar to the samples selected in the new scene and are easy to transfer.

5. The method for evaluating low-quality images in mine machine vision according to claim 1, characterized in that: The specific process of S4 is as follows: S4.

1. Pre-train the model using the samples selected from the old scene that are easy to transfer. That is, set the training parameters and use the contrastive language-image pre-training model to train on the easy-to-transfer dataset. Meanwhile, continuously test on the unselected data from the old scene to evaluate the model performance, and then obtain and save the trained model. S4.

2. Use the pre-trained model to fine-tune the samples selected for the new scene. That is, first load the pre-trained model parameters and set the structural layer where the model adjustable parameters are located. Then use the labeled samples selected for the new scene for training to complete the construction of the image quality evaluation model.

6. A system for evaluating low-quality images in mine machine vision, comprising a camera, characterized in that: It also includes a sample selector and a feature transfer learning trainer, wherein the feature transfer learning trainer is provided with a similarity measurement module and a fine-tuning module; The camera and its memory card are used to collect and save image sample data in new scenes, and to save sample data that has been collected and annotated in old scenes; The sample selector is used to receive the image sample data collected in the new scene, and send the sample data to two different prompt word Prompts modules to obtain multiple distortion distribution probabilities and multiple prediction quality scores respectively, and then obtain the corresponding distortion distribution distance and quality prediction variance through the clustering and statistics modules; through the Dowdall ranking aggregation module, the function of selecting representative samples in the new scene is realized; The feature transfer learning trainer is used to receive samples selected and marked in the new scene and all marked samples in the old scene. The samples first enter the similarity measurement module to calculate the similarity between the samples in the old scene and the selected samples in the new scene. The feature transfer learning trainer selects samples that are easy to transfer in the old scene according to the similarity, and then sends the selected samples in the new scene and the easy-to-transfer samples in the old scene to the comparative language-image pre-training model for training to complete the construction of the image quality evaluation system.

Images