Rail transit engineering video image quality degradation-oriented pre-screening method and system

Abstract

The application discloses a kind of pre-screening method and system for rail transit engineering video image quality degradation, method includes: the image in the image set collected, with traditional method and multimodal big model is combined score, weighted summation obtains degradation score, calculates probability distribution and filters out high-quality image set H.For image in set H, construct positive sample by data enhancement, add degradation feature according to probability distribution to construct negative sample, sample contains image, quality label and normalized score sequence.Based on EfficientNetV2, build discriminator, divide dataset and train, use binary cross entropy and L1 loss as loss function.At the same time, add degradation feature to high-quality image to construct dataset, embedded degradation score by MLP, build repair model based on Transformer-UNet, train using multiple loss functions, realize image high-quality repair.Effectively solve the problem of poor repair effect of traditional repair method, improve the usability of image.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of image processing, and more particularly relates to a pre-screening method and system for video image quality degradation in rail transit engineering. BACKGROUND

[0002] In the process of urban rail transit engineering construction, with the wide application of various monitoring devices, a large amount of video image data is generated. These image data are of great value to various aspects of engineering construction, such as construction progress monitoring, safety hazard identification, and engineering quality control, and can provide intuitive and key information support for engineering decision-making.

[0003] However, due to the constraints of complex environmental factors on the construction site, image quality faces many challenges. The lighting conditions at the construction site are complex and changeable, and strong direct sunlight may cause image overexposure, while in areas with insufficient light such as tunnels, underexposure is prone to occur, resulting in loss of image details and difficulty in identifying key information. At the same time, the continuous vibration of engineering equipment and the equipment shaking caused by the frequent activities of construction personnel can cause image blurring, seriously affecting the clarity and accuracy of the image. In addition, the performance limitations of the shooting equipment itself, such as defects in the sensor, may cause abnormal pixels in the local area of the image; and in adverse weather conditions, such as fog, rain and snow, the image will suffer from atmospheric degradation, with decreased contrast and distorted color, further reducing the image quality.

[0004] At present, existing image quality screening and restoration techniques are difficult to effectively cope with the complex problems faced by urban rail transit engineering construction images. Some simple screening methods only rely on a single image feature for judgment, which cannot comprehensively and accurately identify low-quality images with multiple degradation characteristics, resulting in large errors in the screening results, and many low-quality images are misjudged as high-quality images entering the subsequent processing flow, affecting the accuracy of engineering analysis. In terms of image restoration, the existing restoration methods often have unsatisfactory restoration effects. Some traditional restoration algorithms cannot restore the details and true colors of the image when dealing with complex degradation types, such as images with blurring, noise and color degradation, and the restored images still have obvious defects, which cannot meet the strict requirements of image quality in engineering practical applications.

[0005] In summary, the existing technical means cannot meet the demand of urban rail transit engineering construction on image quality, there are serious deficiencies in the accuracy of image screening and the effectiveness of image repair, and it is difficult to guarantee the efficient and accurate development of subsequent engineering analysis and management such as construction progress monitoring and safety hazard identification. Therefore, it is urgent to develop an efficient pre-screening and repair method for the quality degradation characteristics of urban rail transit engineering construction video images, which has important practical significance for improving the informatization management level of engineering construction and ensuring the quality and safety of engineering construction. SUMMARY

[0006] The present application aims to provide a pre-screening method for the quality degradation characteristics of urban rail transit engineering construction video images. By constructing a multi-modal fusion degradation scoring system, a dynamic degradation synthesis method based on a physical model, and a degradation condition guided Transformer-UNet repair architecture, low-quality images can be accurately selected, image quality can be effectively quantified, and low-quality images can be repaired to high quality, meeting the high-quality data demand of urban rail transit engineering construction in image analysis and application, and providing strong support for subsequent engineering analysis and management.

[0007] In view of the above defects or improvement needs of the prior art, the present application provides a pre-screening method for the quality degradation of rail transit engineering video images, comprising:

[0008] S1. For each image img in the image set IMG, score each image img for degradation characteristics by combining traditional methods and multi-modal large models; weight and sum the two degradation scores of the image to obtain the degradation score of the image; calculate the probability distribution of each degradation characteristic score in the image set IMG; select a high-quality image set H according to the score;

[0009] S2. For the images in the high-quality image set H, construct high-quality positive samples by data augmentation; according to the degradation characteristic probability distribution, randomly select and add single or fused degradation characteristics to construct low-quality negative samples; each sample contains an image, a quality label and a normalized degradation score sequence;

[0010] S3. Build a discriminator based on EfficientNetV2, which can judge the quality of the input image and output the degradation score sequence; divide the data set into training set and validation set according to the set proportion, use the quality label and degradation score sequence of the sample as the true value, and use binary cross entropy and L1 loss as the loss function for training;

[0011] S4. Add multiple degradation features to high-quality images to construct a dataset, perform embedding operations on the degradation scores through a fully connected network MLP, embed the degradation scores in a high-dimensional hidden space, build a repair model based on Transformer-UNet, and train using multiple loss functions to achieve high-quality image repair.

[0012] Further, the degradation features in S1 include:

[0013] Blur: refers to the phenomenon of unclear image edges and details caused by defocus, jitter or motion;

[0014] Noise: refers to the phenomenon of random particles or abnormal pixel points appearing in the image;

[0015] Color degradation: refers to the phenomenon of color shift, fading or abnormal saturation;

[0016] Low resolution: refers to the phenomenon of image detail loss and jagged edges;

[0017] Lighting problem: refers to the phenomenon of loss of bright details or loss of dark details;

[0018] Sensor defect: refers to the phenomenon of bad pixels at fixed positions;

[0019] Atmospheric degradation: refers to the phenomenon of contrast reduction or color distortion caused by haze, rain and snow;

[0020] Motion blur: refers to the phenomenon of dynamic blur caused by camera or object motion.

[0021] Further, the specific method for scoring each image img degradation feature in S1 by combining traditional methods and multi-modal large models is:

[0022] Let the image degradation feature set D = {d1, d2,..., d8};

[0023] Each degradation feature of the image is scored by using traditional methods including gradient calculation and histogram, with the score range being an integer value in 0-10, obtaining the traditional method degradation score sequence S1 = {s 11 , s 12 ,..., s 18} of the image, and the detection and calculation method of each degradation feature is as follows:

[0024] Blur: measured by the Laplacian gradient variance of the image, the smaller the Laplacian gradient variance, the smaller the image gradient change, the higher the blur degree, and the higher the score:

[0025]

[0026] where D(x) denotes the variance of x, denotes the Laplacian gradient operator;

[0027] Noise: measured by the local variance of the image, the larger the local variance, the more serious the image is affected by noise, the higher the score:

[0028]

[0029] where p i denotes a local region in img, m denotes the number of local regions divided in the image img:

[0030] Color degradation: measured by the saturation histogram entropy, the smaller the entropy value, the more serious the image color degradation, the higher the score:

[0031]

[0032] where H i denotes the i-th component in the saturation histogram;

[0033] Low resolution: measured by using FFT to analyze the proportion of high frequency components, the less high frequency information, the lower the image resolution, the higher the score:

[0034]

[0035] where FFT denotes the fast Fourier transform, X H denotes the high frequency region of X, X denotes the result of the fast Fourier transform of the image img;

[0036] Lighting problem: measured by the histogram distribution uniformity, the more serious the histogram accumulation at both ends, the more serious the image overexposure or underexposure problem, the higher the score:

[0037]

[0038] Sensor defects: measured by the proportion of bad points in the image, the higher the proportion of bad points, the more serious the sensor defects, the higher the score:

[0039]

[0040] Atmospheric degradation: measured by dark channel prior and global contrast, the higher the dark channel mean, the lower the global contrast, the more serious the image atmospheric degradation phenomenon, the higher the score:

[0041]

[0042] where, denotes the dark channel, std(x) denotes the standard deviation of x, and n denotes the number of pixels participating in the calculation of the mean of the dark channel;

[0043] Motion blur: the inter-frame motion amplitude mean calculated by the optical flow method is used for measurement. The larger the motion amplitude, the more serious the motion blur phenomenon, and the higher the score.

[0044] s 18 ∝|||flow||2, flow = Farneback(img)

[0045] Where Farneback represents the Farneback optical flow method.

[0046] The Janus-Pro-7B multi-modal large model is used to score different degradation features of the image, and the prompt words are constructed as follows:

[0047] In the urban rail transit engineering construction scene, the image <figure1>Degenerative features can be present <d>This degradation feature refers to <ex>Please analyze whether the image contains the degradation feature and give a score, the score range is an integer value in 0-10, 0 means that the image is completely free of the degradation feature, 10 means that the degradation feature in the image is very serious, only an integer value in 0-10 is answered, no explanation is required.

[0048] Replace the image img in the prompt word with <figure1>, degenerative features d replace <d>interpretation of its degradation feature d <ex>The final degradation score of the image obtained by the multimodal large model is: S2={s 21 s 22 , ..., s 28 }

[0049] Furthermore, the method for adding the degradation feature in S2 is as follows:

[0050] Blur: Use kernel size Gaussian blur kernel G σ Perform a convolution operation on h:

[0051]

[0052] In the formula, s1 represents the variable parameter used to control the blur level of the image, which takes a value between [0, 0.1] depending on the image situation, h represents an image extracted from the high-quality image set H, and h′ represents the image after adding degradation features to the image h;

[0053] Noise: Gaussian mixture noise z for h g Salt and pepper noise z s The ratio is controlled by s2:

[0054] h'=(1-0.3s²)·h+0.3s²·(z g +z s )

[0055] In the formula, s2 represents the Gaussian noise z used to control the noise. g Salt and pepper noise z s The mixing ratio coefficient has a value range of [0, 0.05].

[0056] Color degradation: Manually reduce color cast and saturation on h, with the proportion controlled by s3.

[0057]

[0058] Where Gray(x) represents the grayscale operation on x, h t s3 represents the color shift processing of h, and s3 represents the coefficient controlling the degree of color degradation, with a value range of [0, 0.5].

[0059] Low resolution: h is downsampled first and then upsampled, with the scaling ratio controlled by s4.

[0060] h'=Resize(Resize(h,scale),original),scale=1-0.05s4

[0061] where Resize(h, scale) denotes resizing h to scale, original denotes the original size of h, s4 denotes the scaling ratio, and the value range of s4 is [0, 10];

[0062] Lighting problem: dynamically generate the overexposed area h of h according to the score s5 o = h > 220 - 10s5 and the underexposed area h u = h < 35 + 10s5 and mix h with a proportion controlled by s5:

[0063] h' = h · (1 - 0.05s5·h u ) + 255 · 0.08s5·h o

[0064] In the formula, s5 denotes a variable parameter for controlling the proportion of h u and h o , the value range of s5 is [0, 10], h u denotes the underexposed area in the image h determined according to s5, and h o denotes the overexposed area in the image h determined according to s5.

[0065] Sensor defect: generate 10s6 bad points added to random positions on h, and s6 denotes a variable parameter for controlling the number of generated bad points, the value range of s6 is [0, 0.3];

[0066] Atmospheric degradation: mix the prepared haze and snow map z with h i , and the proportion is controlled by s7:

[0067] h' = (1 - 0.3s7) · h + 0.3s7·z i

[0068] In the formula, s7 denotes a variable parameter for adjusting the atmospheric degradation effect, and the value determines the mixing proportion of the image and the haze and snow map, the value range of s7 is [0.2, 0.6];

[0069] Motion blur: perform convolution operation on h using a linear motion kernel G θ，L controlled by s8:

[0070]

[0071] In the formula, s8 denotes a variable parameter for controlling the motion blur effect, which will affect the angle and length of the linear motion kernel, the value range of s8 is [0, 5], L denotes the length parameter of the linear motion kernel G θ，L , and θ denotes the angle parameter of the linear motion kernel G θ，L .

[0072] Furthermore, in S3, binary cross-entropy and L1 loss are used as loss functions for training, as detailed below:

[0073]

[0074] L=λ1L BCE +λ2L1

[0075] Among them, y i This represents the true label of the sample. The label s represents the output of the discriminator. i This represents the true degradation score sequence of the sample. Let m represent the degraded score sequence output by the discriminator, m represent the number of samples, and λ1 and λ2 represent the control losses L1 and L2, respectively. BCE The ratio is initially set to λ1 = λ2 = 0.5. As training progresses, λ1 gradually decreases linearly to 0.3, while λ2 increases linearly to 0.7.

[0076] Furthermore, the specific process of adding various degradation features to high-quality images to construct the dataset in S4 is as follows:

[0077] For each image in the high-quality image set H, a degradation feature is added to obtain the corresponding low-quality image; specifically, for a high-quality image h with an original rating of S = {s1, s2, ..., s8}, a degradation feature d is randomly selected. i According to its probability distribution N i A degradation score s′ is obtained randomly. i The original image is degraded to obtain the corresponding low-quality negative sample h', and the corresponding original score s is then processed. i Replace with s′ i Obtain the degradation score for h';

[0078] Select a degenerate feature that has not been added. j Repeat the above steps. For each high-quality image, add at least two degradation features to improve the generalization of the image restoration model. Finally, obtain the degradation score sequence S′={s′1,s′2,...,s′8} of the low-quality image. Each sample in the dataset contains the low-quality image h' as ​​input, its corresponding degradation score sequence S', and the original high-quality image h as the ground truth.

[0079] Furthermore, in S4, the degradation score embedding operation is completed through a fully connected network MLP. The specific process of embedding the degradation score into the high-dimensional latent space is as follows:

[0080] To complete the image inpainting by the degradation score prior guided model, the degradation score sequence needs to be embedded into a high-dimensional latent space; the embedding operation of the degradation score S' is realized by using a fully connected network (MLP), and the high-dimensional latent vector T_em is obtained after the MLP coding of the degradation score S';

[0081] The Resnet18 is used to extract the features of the low-quality image h', and a high-dimensional latent vector P_em is obtained; the weighted sum of the two is V_em = BP_em + (1-β)T_em, which encodes the degradation features of the low-quality image h', and β is the weight.

[0082] Further, the specific process of building the inpainting model based on the Transformer-UNet in S4 is as follows:

[0083] The left half of the Transformer input is the output of the previous layer and the degradation embedding V_em, the channel number gradually increases, and the image size gradually decreases; the right half of the discriminator Transformer input is the sum of the output of the previous layer and the output of the corresponding layer on the left side, and the degradation embedding V_em, the channel number gradually decreases, and the image size gradually increases; the overall Unet input is the low-quality image h' and the degradation embedding V_em, and the output is the predicted noise z' and the inpainted high-quality image h p = h'-z'.

[0084] Further, the specific method of training by using multiple loss functions in S4 is as follows:

[0085] Reconstruction loss: including L1 loss and SSIM loss, indicating that the inpainted high-quality image h p should be consistent with the original high-quality image h; the formula is as follows:

[0086]

[0087] Where, y i represents the pixel point in the original high-quality image h, represents the pixel point in the inpainted high-quality image h p , i represents the position of the pixel point in the image, μ x and μ γ are the pixel mean of the images h and h p , σ x 2 and σ γ 2 are the pixel variance of the images h and h p , σ xγ is the covariance of the images h and h p , C1 and C2 are stable constants;

[0088] Discrimination loss: binary cross-entropy loss, representing the high-quality image h repaired p Should be identified as high quality by the discriminator:

[0089]

[0090] Wherein, y i Indicates a high-quality label; Indicates the label output by the discriminator, and m indicates the number of samples.

[0091] Similarity loss: cosine similarity loss, representing that the two embeddings P_em and T_em should be basically consistent, and the formula is as follows:

[0092] L cos =1-cos(P,T)

[0093] Then the total loss function formula is as follows:

[0094] L=λ1L BCE +λ2L1+λ3L SSIM +λ4L cos

[0095] Wherein, λ1, λ2, λ3, λ4 are the weights of the corresponding loss function respectively.

[0096] As a second aspect of the application, a pre-screening system for video image quality degradation of rail transit engineering is also provided, comprising:

[0097] An image screening and scoring processing unit is configured to score the degradation characteristics of each image img in the image set IMG by combining traditional methods and multi-modal large models, and to obtain the degradation score of the image by weighting the two degradation scores of the image. The probability distribution of each degradation characteristic score in the image set IMG is calculated, and the high-quality image set H is screened according to the score.

[0098] A high-quality image discrimination dataset construction unit is configured to construct high-quality positive samples by data augmentation for the images in the high-quality image set H, and to construct low-quality negative samples by randomly selecting and adding single or fused degradation characteristics according to the degradation characteristic probability distribution. Each sample contains an image, a quality label, and a normalized degradation score sequence.

[0099] A built and trained image discriminator unit is configured to build a discriminator based on EfficientNetV2, which can judge the quality of the input image and output the degradation score sequence. The dataset is divided into a training set and a validation set according to the set proportion, and the quality label and the degradation score sequence of the sample are used as the true value. Binary cross-entropy and L1 loss are used as the loss function for training.

[0100] The image restoration model construction and training unit is used for constructing a data set by adding various degradation features to a high-quality image, performing embedding operation of the degradation score through a fully connected network MLP, embedding the degradation score into a high-dimensional hidden space, building a restoration model based on a Transformer-UNet, and training by using various loss functions, so as to realize high-quality image restoration.

[0101] Overall, compared with the prior art, the above technical solutions conceived by the present application can achieve the following beneficial effects:

[0102] 1. The pre-screening method for video image quality degradation of rail transit engineering of the present application combines traditional methods with multi-modal large models by constructing a multi-modal fusion degradation scoring system, scores various degradation features such as image blur and noise, and then performs weighted summation to obtain a comprehensive degradation score, and calculates the probability distribution of each degradation feature score, so as to screen high-quality images. This system effectively overcomes the limitations of single judgment criteria, comprehensively considers various degradation factors of images, improves the accuracy of low-quality image screening, provides a reliable data basis for subsequent processing, and ensures the accuracy of engineering analysis.

[0103] 2. The pre-screening method for video image quality degradation of rail transit engineering of the present application adds various degradation features to a high-quality image to construct a data set by designing a dynamic degradation synthesis method based on a physical model, and embeds the degradation score into a high-dimensional hidden space, thereby providing rich and high-quality data samples for model training. This method simulates the complex and variable image degradation conditions in actual engineering, enhances the adaptability and generalization ability of the model to different degradation scenarios, so that the model can more accurately identify and cope with various degradation problems when processing real image data, and ensures the reliability of the screening and restoration results.

[0104] 3. The pre-screening method for video image quality degradation of rail transit engineering of the present application realizes high-quality restoration of low-quality images by building a degradation condition guided Transformer-UNet restoration architecture and training by using various loss functions. This architecture fully utilizes the advantages of Transformer and UNet, combines degradation score information to guide the restoration process, so that the restored image is highly consistent with the original high-quality image in structure, texture and color, effectively solves the problem of poor restoration effect of traditional restoration methods, improves the usability of images, meets the strict requirements of urban rail transit engineering construction on image quality, and helps the smooth development of engineering management and decision-making work. BRIEF DESCRIPTION OF DRAWINGS

[0105] Figure 1 The flowchart of the pre-screening method for video image quality degradation of rail transit engineering of the present application of the present application embodiment;

[0106] Figure 2 A flowchart of an image screening process according to an embodiment of the present application;

[0107] Figure 3 A flowchart of construction of an image discrimination data set according to an embodiment of the present application;

[0108] Figure 4 A discriminator architecture diagram according to an embodiment of the present application;

[0109] Figure 5 A fully connected network architecture diagram according to an embodiment of the present application;

[0110] Figure 6 An architecture diagram of an image inpainting model according to an embodiment of the present application;

[0111] Figure 7 A system unit diagram according to an embodiment of the present application. DETAILED DESCRIPTION

[0112] In order to make the objectives, technical solutions and advantages of the present application clearer, further detailed descriptions will be given below in combination with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and should not be used to limit the present application. In addition, the technical features involved in each embodiment of the present application described below can be combined with each other as long as there is no conflict.

[0113] Embodiment 1

[0114] Please refer to Figure 1 The present embodiment 1 provides a pre-screening method for video image quality degradation of rail transit engineering, comprising:

[0115] S1. For each image img in the image set IMG, the degradation features of each image img are scored by a combination of traditional methods and multi-modal large models; the degradation scores of the images are weighted and summed to obtain the degradation score of the image; the probability distribution of each degradation feature score in the image set IMG is calculated; and high-quality image set H is screened according to the score;

[0116] S2. For the images in the high-quality image set H, construct high-quality positive samples by data augmentation; according to the degradation feature probability distribution, randomly select and add single or fused degradation features to construct low-quality negative samples; each sample contains an image, a quality label and a normalized degradation score sequence;

[0117] S3. Build a discriminator based on EfficientNetV2, which can judge the quality of the input image and output a degradation score sequence; divide the dataset into training set and validation set according to the set proportion, use the quality label and degradation score sequence of the sample as the true value, and use binary cross entropy and L1 loss as the loss function to train;

[0118] S4. Add multiple degradation features to high-quality images to construct a dataset, complete the embedding operation of the degradation score through the fully connected network MLP, embed the degradation score in the high-dimensional hidden space, build a repair model based on Transformer-UNet, and train it using multiple loss functions to achieve high-quality image repair.

[0119] This embodiment 1 expands on the above steps.

[0120] (1) Image screening

[0121] Let the image set IMG = {img1, img2,..., img k}, img is a city rail transit construction image. Please refer to Figure 2 , there is a common image degradation feature set D = {d1, d2,..., d8}, for example, d1 is blur, d2 is noise, d3 is color degradation, etc. For each image img in the image set IMG, use the traditional method and the multi-modal large model (MLLM) combined method to score the above degradation features, the higher the score, the more obvious the degradation feature in img.

[0122] 1.1 Traditional method scoring

[0123] Use gradient calculation, histogram, and other traditional methods to score each degradation feature of the image, with integer values in the range of 0-10, to obtain the traditional method degradation score sequence S1 = {s 11 , s 12 …, s 18} of the image. The interpretation, detection, and calculation method of each degradation feature is shown below, and the specific scoring standard can be determined manually through pre-test results.

[0124] Blur: blur refers to the phenomenon of unclear image edges and details caused by defocus, jitter, or motion, which can be measured by the Laplacian gradient variance of the image. The smaller the Laplacian gradient variance, the smaller the image gradient change, the higher the blur degree, and the higher the score.

[0125]

[0126] where D(x) represents the variance of x, represents the Laplacian gradient operator.

[0127] Noise: Noise refers to the phenomenon of random particles or abnormal pixels appearing in the image, which can be measured by the local variance of the image. The larger the local variance, the more serious the noise effect on the image, and the higher the score.

[0128]

[0129] where p i represents a local region in img, and m represents the number of local regions divided in the image img:

[0130] Color degradation: Color degradation refers to the phenomenon of color shift, fading or abnormal saturation, which can be measured by the saturation histogram entropy. The smaller the entropy value, the more serious the color degradation of the image, and the higher the score.

[0131]

[0132] where H i represents the i-th component in the saturation histogram.

[0133] Low resolution: Low resolution refers to the phenomenon of image detail loss and jagged edges, which can be measured by using FFT to analyze the proportion of high-frequency components. The less high-frequency information, the lower the image resolution, and the higher the score.

[0134]

[0135] where FFT represents Fast Fourier Transform, X H represents the high-frequency region of X, and X represents the result of Fast Fourier Transform on the image img;

[0136] Lighting problem: Lighting problem refers to the phenomenon of detail loss in bright areas (overexposure) or detail loss in dark areas (underexposure), which can be measured by the histogram distribution balance. The more serious the accumulation at both ends of the histogram, the more serious the overexposure or underexposure problem of the image, and the higher the score.

[0137]

[0138] Sensor defects: Sensor defects refer to the phenomenon of bad points (black / white pixel blocks) appearing at fixed positions, which can be measured by the proportion of bad points in the image. The higher the proportion of bad points, the more serious the sensor defects, and the higher the score.

[0139]

[0140] Atmospheric degradation: atmospheric degradation refers to the phenomenon of contrast reduction or color distortion caused by haze, rain and snow, which can be measured by dark channel prior and global contrast. The higher the average of dark channel, the lower the global contrast, the more serious the atmospheric degradation of the image, and the higher the score.

[0141]

[0142] wherein, represents the dark channel, std(x) represents the standard deviation of x, and n represents the number of pixels participating in the calculation of the average of the dark channel;

[0143] Motion blur: motion blur refers to the dynamic blur phenomenon caused by the movement of the camera or the object, which can be measured by calculating the average of the inter-frame motion amplitude by the optical flow method. The larger the motion amplitude, the more serious the motion blur phenomenon, and the higher the score. (If there is no adjacent frame, the score of this item is consistent with the score of (1))

[0144] s 18 ∝||flow|||2, flow = Farneback(img)

[0145] wherein, Farneback represents the Farneback optical flow method.

[0146] 1.2. MLLM score

[0147] Use Janus-Pro-7B multi-modal large model to score different degradation features of the image, and construct the prompt word as follows:

[0148] "In the urban rail transit engineering construction scene, the image <figure1>Degenerative features can be present <d>This degradation feature refers to <ex>Please analyze whether the image contains the degradation feature and give a score, the score range is an integer value in 0-10, 0 means that the image is completely free of the degradation feature, 10 means that the degradation feature in the image is very serious, only an integer value in 0-10 is answered, no explanation is required.

[0149] Replace the image img in the prompt word with <figure1>, degenerative features d replace <d>interpretation of its degradation feature d <ex>; finally get the multi-modal large model's degradation score of the image: S2 = {s 21 , s 22 ..., s 28}.

[0150] 1.3 Calculate the probability distribution

[0151] The degradation score of the image is obtained by weighted sum of two degradation scores of the image: S = aS1 + (1 - a)S2 = {s1, s2,..., s8}, where the weight a = 0.7 (need to be adjusted according to actual situation). The same operation is performed on all images in the image set IMG. For the degradation feature d i , the mean value μ i and the variance of the degradation score in IMG are calculated. Where N i represents the probability distribution of the degradation feature d i .

[0152]

[0153] 1.4. Screening high-quality images

[0154] If each item in the image score sequence is less than 2 (need to be adjusted according to actual situation), the image is identified as a high-quality image. Form the high-quality image set H = {h1, h2,..., h m}.

[0155] (2) Image discrimination

[0156] 2.1 Image degradation

[0157] In order to get the degraded image with the specified degradation score sequence S = {s1, s2,..., s8}, for each degradation feature, the image h is manually added with the degradation feature in the following way.

[0158] Blur: use a Gaussian blur kernel G σ with kernel size to perform convolution operation on h:

[0159]

[0160] In the formula, s1 represents a variable parameter for controlling the blurring degree of the image, which is selected from [0, 0.1] according to the image, h represents an image extracted from the high-quality image set H, and h' represents the image after the degradation feature addition processing of the image h;

[0161] Noise: mix Gaussian noise z g and salt and pepper noise z s , the proportion is controlled by s2:

[0162] h' = (1 - 0.3s2) h + 0.3s2(z g + z s )

[0163] In the formula, s2 represents a mixed proportion coefficient for controlling the Gaussian noise z g and the salt and pepper noise z s , and the value range is [0, 0.05];

[0164] Color degradation: manual color cast and saturation reduction operation is performed on h, and the proportion is controlled by s3:

[0165]

[0166] wherein Gray(x) represents a grayscale operation on x, h t represents a color cast processing of h, s3 represents a color degradation degree coefficient, and the value range is [0, 0.5];

[0167] Low resolution: h is first down-sampled and then up-sampled, and the scaling proportion is controlled by s4:

[0168] h' = Resize(Resize(h, scale), original), scale = 1 - 0.05s4

[0169] wherein Resize(h, scale) represents adjusting the size of h to scale, original represents the original size of h, s4 represents a scaling proportion, and the value range is [0, 10];

[0170] Illumination problem: the overexposure area h o = h > 220 - 10s5 and the underexposure area h u = h < 35 + 10s5 of h are dynamically generated according to the score s5, and h is mixed, and the proportion is controlled by s5:

[0171] h' = h (1 - 0.05s5 h u ) + 255 0.08s5 h o

[0172] In the formula, s5 represents a proportion for controlling h u and h o , the value range is [0, 10], h u represents an underexposure area in the image h determined according to s5, and h o represents an overexposure area in the image h determined according to s5;

[0173] Sensor defect: 10s6 bad points are generated and added to h at random positions, s6 represents the variable parameter for controlling the number of bad points, and the value range is [0, 0.3];

[0174] Atmospheric degradation: h is mixed with a pre-prepared haze, rain and snow map z i , and the proportion is controlled by s7:

[0175] h' = (1-0.3s7)·h+0.3s7·z i

[0176] In the formula, s7 represents a variable parameter for adjusting the atmospheric degradation effect, which determines the mixing proportion of the image and the haze, rain and snow map, and the value range is [0.2, 0.6];

[0177] Motion blur: a linear motion kernel G controlled by s8 is used θ，L to perform convolution operation on h:

[0178]

[0179] In the formula, s8 represents a variable parameter for controlling the motion blur effect, which will affect the angle and length of the linear motion kernel, and the value range is [0, 5], L represents the length parameter of the linear motion kernel G θ，L , and θ represents the angle parameter of the linear motion kernel G θ，L .

[0180] 2.2 Constructing the data set.

[0181] According to the Figure 3 steps, each image in the high-quality image set H is processed. Through appropriate rotation, scaling and other methods, high-quality positive samples are constructed through data enhancement, and corresponding low-quality negative samples are constructed by manually adding single degradation features or fusing degradation features.

[0182] When constructing low-quality negative samples, for a high-quality image h with original score S = {s1, s2,..., s8}, a degradation feature d i is randomly selected according to its probability distribution N i to obtain a degradation score s' i , and the original image is degraded to obtain the corresponding low-quality negative sample h' using the method in image degradation, and the corresponding original score s i is replaced by s' i to obtain the degradation score of h'.

[0183] When adding fused degradation features, no more than 4 degradation features are selected for the same processing. Through the above steps, the data set is constructed, and each sample contains an image, a high-quality / low-quality label and a normalized degradation score sequence.

[0184] The score is obtained according to the probability distribution, so as to make the model more suitable for the specific scene of urban rail transit; the score s' is obtained according to the score s and the probability distribution i The reason for processing high-quality images without directly using the low-quality images and their scores in image screening is that the scores in image screening are only general scores and are not consistent, and the score s' is obtained by using s i Processing high-quality images can obtain more consistent results, which is more conducive to the training and convergence of the model.

[0185] 2.3. Building a discriminator.

[0186] The discriminator is built based on EfficientNetV2, and the architecture thereof is shown in Figure 4 The discriminator can determine whether the input image is a low-quality image or a high-quality image and output the degradation score sequence thereof.

[0187] 2.4. Training the discriminator.

[0188] 70% of the samples in the data set are used as a training set, and 30% of the samples are used as a verification set. The high-quality / low-quality labels and the degradation score sequence in the samples are used as true values and are compared with the model output to train the discriminator. If the accuracy of the trained model in distinguishing high-quality / low-quality samples on the verification set reaches 95% or more and the average error of the degradation score is less than 1, it is considered that the discriminator is successfully trained. Binary cross-entropy and L1 loss are used as loss functions, as shown below:

[0189]

[0190] L = λ1L BCE + λ2L1

[0191] wherein y i represents the true label of the sample, represents the label output by the discriminator, s i represents the true degradation score sequence of the sample, represents the degradation score sequence output by the discriminator, and m represents the number of samples; λ1, λ2 represent the proportion of controlling the loss L1 and L BCE In a specific preferred embodiment, the initial values thereof are λ1 = λ2 = 0.5, λ1 gradually linearly decays to 0.3, and λ2 linearly increases to 0.7 as the training proceeds.

[0192] (3) Image inpainting

[0193] 3.1. Constructing a data set.

[0194] For each image in the high-quality image set H, a degradation feature is added to obtain the corresponding low-quality image; specifically, for a high-quality image h with an original rating of S = {s1, s2, ..., s8}, a degradation feature d is randomly selected. i According to its probability distribution N i A degradation score s′ is obtained randomly. i The original image is degraded to obtain the corresponding low-quality negative sample h', and the corresponding original score s is then processed. i Replace with s′ i Obtain the degradation score for h';

[0195] Select a degenerate feature that has not been added. j Repeat the above steps. For each high-quality image, add at least two degradation features to improve the generalization of the image restoration model. Finally, obtain the degradation score sequence S′={s′1,s′2,...,s′8} of the low-quality image. Each sample in the dataset contains the low-quality image h' as ​​input, its corresponding degradation score sequence S', and the original high-quality image h as the ground truth.

[0196] 3.2. Degenerate embedding.

[0197] To leverage prior degradation scores to guide the model for image inpainting, the degradation score sequence needs to be embedded into a high-dimensional latent space, such as... Figure 5 As shown:

[0198] The degradation score embedding operation is implemented using a fully connected network (MLP). The degradation score S' is encoded by the MLP to obtain a high-dimensional latent vector T_em.

[0199] Feature extraction is performed on the low-quality image h' based on ResNet18, and a high-dimensional latent vector P_em is obtained. The two are weighted and summed to obtain V_em = βP_em + (1-β)T_em. V_em encodes the degradation features of the low-quality image h', where β is the weight.

[0200] 3.3. Model building.

[0201] An image inpainting model is built based on Transformer-UNet, and its architecture is as follows: Figure 6 As shown. The left half of the Transformer takes the output of the previous layer and the degenerate embedding V_em as input, with the number of channels gradually increasing and the image size gradually decreasing. The right half of the discriminator Transformer takes the sum of the output of the previous layer and the output of the corresponding layer on the left, as well as the degenerate embedding V_em, with the number of channels gradually decreasing and the image size gradually increasing. The overall UNet takes the low-quality image h' and the degenerate embedding V_em as input, and outputs the predicted noise z' and the restored high-quality image h'. p =h′-z′.

[0202] 3.4. Loss function.

[0203] Reconstruction loss: including L1 loss and SSIM loss, represents the high-quality image h p restored should be consistent with the original high-quality image h. The formula is as follows:

[0204]

[0205] where y i represents the pixel point in the original high-quality image h, represents the pixel point in the high-quality image h p restored, i represents the position of the pixel point in the image, μ x and μ γ are the pixel mean of the images h and h p , σ x 2 and σ γ 2 are the pixel variance of the images h and h p , σ xγ is the covariance of the images h and h p , C1 and C2 are stable constants, and in some preferred embodiments, the values of the two parameters are 6.5 and 58.5, respectively.

[0206] Discrimination loss: binary cross-entropy loss, represents that the high-quality image h p restored should be recognized as high-quality by the discriminator:

[0207]

[0208] where y i represents the high-quality label; represents the label output by the discriminator, and m represents the number of samples.

[0209] Similarity loss: cosine similarity loss, represents that the two embeddings P_em and T_em should be basically consistent, and the formula is as follows:

[0210] L cos = 1 - cos(P, T)

[0211] Therefore, the total loss function formula is as follows:

[0212] L = λ1L BCE + λ2L1+ λ3L SSIM + λ4L cos

[0213] where λ1, λ2, λ3, and λ4 are the weights of the corresponding loss functions, respectively.

[0214] 3.5. Model training.

[0215] 70% of the samples in the data set are used as the training set, and 30% of the samples are used as the validation set. If the reconstructed image of the trained model on the validation set is compared with the original image, the average PSNR index is greater than 20, the average SSIM index is greater than 0.9, and 95% of the repaired images are identified as high quality by the discriminator, then the image repair model is considered to be successfully trained.

[0216] Embodiment 2

[0217] Please refer to Figure 7 The embodiment 2 provides a pre-screening system for video image quality degradation of rail transit engineering, comprising:

[0218] An image screening and scoring processing unit is configured to score the degradation characteristics of each image img in the image set IMG by a combination of a traditional method and a multi-modal large model; to obtain the degradation score of the image by weighted summation of the two degradation scores of the image; to obtain the probability distribution of each degradation characteristic score in the image set IMG by calculation; and to screen out a high-quality image set H according to the score.

[0219] A high-quality image discrimination data set construction unit is configured to construct high-quality positive samples by data enhancement for the images in the high-quality image set H; to construct low-quality negative samples by randomly selecting and adding single or fused degradation characteristics according to the degradation characteristic probability distribution; and to include the image, the quality label, and the normalized degradation score sequence in each sample.

[0220] A built and trained image discriminator unit is configured to build a discriminator based on EfficientNetV2, which can judge the quality and output the degradation score sequence according to the input image; to divide the data set into a training set and a validation set according to the set proportion; to use the quality label and the degradation score sequence of the sample as the true value; and to use binary cross-entropy and L1 loss as the loss function for training.

[0221] An image repair model construction and training unit is configured to construct a data set by adding various degradation characteristics to high-quality images, to complete embedding operation of the degradation score by a fully connected network MLP, to embed the degradation score in a high-dimensional hidden space, to build a repair model based on Transformer-UNet, and to train by using various loss functions to realize high-quality image repair.

[0222] Embodiment 3

[0223] The embodiment 3 also provides a computer readable storage medium having a computer program stored thereon, and the computer program is executable by a processor to implement any step of the video image data cleaning method for urban rail transit engineering.

[0224] The computer readable storage medium can include a U disk, a mobile hard disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk or an optical disk, and various media capable of storing program codes.

[0225] For the computer readable storage medium provided in the present application, refer to the above method embodiments, which will not be repeated herein.

[0226] Those skilled in the art can easily understand that the above description is only a preferred embodiment of the present application, and is not intended to limit the present application. Any modification, equivalent replacement and improvement made within the spirit and principle of the present application shall be included in the protection scope of the present application.< / ex> < / d> < / ex> < / d> < / ex> < / d> < / ex> < / d>

Claims

1. A pre-screening method for video image quality degradation in rail transit engineering, characterized in that... ,include: S1. For each image img in the image set IMG, the degradation features of each image img are scored using a combination of traditional methods and multimodal large model; the two degradation scores of the image are weighted and summed to obtain the degradation score of the image; the probability distribution of the degradation feature score in the image set IMG is calculated; and a high-quality image set H is selected based on the scores. S2. For images in the high-quality image set H, construct high-quality positive samples through data augmentation; construct low-quality negative samples by randomly selecting and adding single or fused degradation features according to the degradation feature probability distribution; each sample contains an image, a quality label, and a normalized degradation score sequence. S3. A discriminator is built based on EfficientNetV2. Its architecture can judge the quality of the input image and output the degradation score sequence. The dataset is divided into training set and validation set according to a set ratio. The quality label and degradation score sequence of the sample are used as the ground truth. Binary cross-entropy and L1 loss are used as loss functions for training. S4. Add various degradation features to high-quality images to construct a dataset, complete the degradation score embedding operation through a fully connected network MLP, embed the degradation score into a high-dimensional latent space, build a restoration model based on Transformer-UNet, and train it using various loss functions to achieve high-quality image restoration. The specific method for training using multiple loss functions in S4 is as follows: Reconstruction loss: includes L1 loss and SSIM loss, representing the high-quality image h after restoration. p It should be consistent with the original high-quality image h; the formula is as follows: ， ， in, This represents the pixels in the original high-quality image h. h represents the high-quality image that has been repaired. p The pixels in Indicates the position of a pixel in the image. and For images h and h p The average pixel value, and For images h and h p pixel variance For images h and h p covariance, and It is the stability constant; Discriminant loss: Binary cross-entropy loss, representing the high-quality image h after restoration. p The discriminator should classify it as high quality: ， Where y represents a high-quality tag; Similarity loss: Cosine similarity loss, representing the consistency between the two embeddings P_em and T_em, is calculated as follows: ， The total loss function formula is as follows: ， in, These are the weights of their respective loss functions.

2. The pre-screening method for video image quality degradation in rail transit engineering according to claim 1, characterized in that, The degradation features in S1 include: Blur: refers to the phenomenon where the edges and details of an image are not clear due to defocus, shaking, or motion; Noise: refers to the phenomenon of random particles or abnormal pixels in an image; Color degradation: refers to the phenomenon of color shift, fading, or abnormal saturation; Low resolution: refers to the loss of image details and the appearance of jagged edges; Lighting issues: refers to the loss of detail in either bright or dark areas; Sensor defect: refers to the phenomenon of dead pixels appearing at fixed locations; Atmospheric degradation: refers to the phenomenon of decreased contrast or color distortion caused by smog, rain, and snow; Motion blur: refers to the phenomenon of dynamic blur caused by the movement of a camera or object.

3. The pre-screening method for video image quality degradation in rail transit engineering according to claim 1, characterized in that, The specific method for scoring the degradation features of each image (img) in S1, which combines traditional methods with a multimodal large model, is as follows: Let the image degradation feature set be... ; By using traditional methods, including gradient calculation and histograms, to score each degradation feature of the image, with scores ranging from 0 to 10 integer values, a traditional method degradation score sequence for the image is obtained. The detection and calculation methods for each degradation feature are as follows: Blur: Measured by the Laplacian gradient variance of the image. The smaller the Laplacian gradient variance, the smaller the gradient change in the image, the higher the degree of blur, and the higher the score. ， Where D(x) represents the variance of x, Represents the Laplacian gradient operator; Noise: Measured by the local variance of the image. The larger the local variance, the more severely the image is affected by noise, and the higher the score. ， Where, p i express A local area: Color degradation: Measured by the entropy of the saturation histogram; the smaller the entropy value, the more severe the color degradation and the higher the score. ， Among them, H i This represents the i-th component in the saturation histogram; Low resolution: Measured by the proportion of high-frequency components analyzed using FFT. The less high-frequency information, the lower the image resolution, and the higher the score. ， Where FFT represents the FFT transform, X H Represents the high-frequency region of X; Lighting issues: Measured by histogram distribution uniformity. The more severe the clustering at both ends of the histogram, the more serious the overexposure or underexposure problem, and the higher the score. ， Sensor defects: Measured by the proportion of bad pixels in the image. A higher proportion of bad pixels indicates a more serious sensor defect and a higher score. ， Atmospheric degradation: Measured by dark channel prior and global contrast, a higher dark channel mean and lower global contrast indicate more severe atmospheric degradation and a higher score. ， in, This represents the dark channel, and std(x) represents the standard deviation of x. Motion blur: Measured by the average motion amplitude between frames calculated using optical flow. A larger motion amplitude indicates more severe motion blur and a higher score. ， Wherein, Farneback represents the Farneback optical flow method; The Janus-Pro-7B multimodal large model was used to score different degradation features of the image, and the following prompt words were constructed: "In the context of urban rail transit engineering construction, images" <figure1>There may be degenerative characteristics in it. <d>This degradation characteristic refers to <ex> Please analyze whether the image contains the degradation feature and give a score. The score range is an integer value between 0 and 10. 0 means that the image does not contain the degradation feature at all, and 10 means that the degradation feature in the image is very serious. Just answer with an integer between 0 and 10, no explanation is required.< / ex> < / d> Image Replace the prompt words <figure1>Degeneracy feature d replacement <d>The explanation and replacement of its degenerate feature d <ex>Finally, the degradation score of the image is obtained by the multimodal large model: .< / ex> < / d> 4. The pre-screening method for video image quality degradation in rail transit engineering according to claim 1, characterized in that, The method for adding the degenerate feature in S2 is as follows: Blur: Use kernel size Gaussian blur kernel Perform a convolution operation on h: ， Noise: Gaussian mixture noise z for h g Salt and pepper noise z s The ratio is controlled by s2: ， Color degradation: Manually reduce color cast and saturation on h, with the proportion controlled by s3. ， Where Gray(x) represents the grayscale operation on x, h t Indicates the color shift processing of h; Low resolution: h is downsampled first and then upsampled, with the scaling ratio controlled by s4. ， Where Resize(x,scale) means adjusting the size of x to scale, and original means the original size of h; Lighting issues: Dynamically generate overexposed areas (h) based on score s5. and underexposed areas And mix with h, the proportion controlled by s5: ， Sensor defect: Generate 6 bad pixels (10s) and add them to random locations on h: Atmospheric degradation: mixing pre-prepared haze, rain, and snow textures with h. i The ratio is controlled by S7: ， Motion blurring: using a linear motion kernel controlled by s8 Perform a convolution operation on h: 。 5. The pre-screening method for video image quality degradation in rail transit engineering according to claim 1, characterized in that, S3 uses binary cross-entropy and L1 loss as loss functions for training, as detailed below: ， ， ， Where y represents the true label of the sample. This represents the label output by the discriminator. This represents the true degradation score sequence of the sample. This represents the degraded score sequence output by the discriminator, where m represents the number of samples.

6. The pre-screening method for video image quality degradation in rail transit engineering according to claim 1, characterized in that, The specific process of adding multiple degradation features to high-quality images to construct the dataset in S4 is as follows: Adding degradation features to each image in the high-quality image set H yields the corresponding low-quality image; specifically, for an image with an original rating of... A high-quality image h, with a randomly selected degradation feature d i According to its probability distribution N i A degradation score is obtained randomly. The original image is degraded to obtain the corresponding low-quality negative samples. and the corresponding original score Replace with get Degradation score; Select a degenerate feature that has not been added. j Repeat the above steps, adding at least two degradation features for each high-quality image to improve the generalization of the image inpainting model, ultimately obtaining a degradation score sequence for low-quality images. Each sample in the dataset contains a low-quality image that served as input. Its corresponding degradation score sequence And the original high-quality image as the truth value. .

7. The pre-screening method for video image quality degradation in rail transit engineering according to claim 1, characterized in that, In step S4, the degradation score is embedded using a fully connected network MLP. The specific process of embedding the degradation score into the high-dimensional latent space is as follows: To guide the image inpainting model using prior degradation scores, the degradation score sequence needs to be embedded into a high-dimensional latent space. A fully connected machine-level programming (MLP) network is used to perform the embedding operation of the degradation scores. After MLP encoding, a high-dimensional latent vector T_em is obtained; Based on ResNet18 for low-quality images Feature extraction is performed, resulting in a high-dimensional latent vector P_em; the weighted sum of the two yields... V_em encodes low-quality images. The degenerative characteristics, As weight.

8. The pre-screening method for video image quality degradation in rail transit engineering according to claim 1, characterized in that, The specific process of building the repair model based on Transformer-UNet in S4 is as follows: The left half of the Transformer takes the output of the previous layer and the degenerate embedding V_em as input, with the number of channels gradually increasing and the image size gradually decreasing. The right half of the discriminator Transformer takes the sum of the output of the previous layer and the output of the corresponding layer on the left, as well as the degenerate embedding V_em, with the number of channels gradually decreasing and the image size gradually increasing. The overall UNet takes a low-quality image h' and a degenerate embedding V_em as input, and outputs prediction noise. High-quality restored images .

9. A pre-screening system for video image quality degradation in rail transit engineering, characterized in that, include: The image filtering and scoring unit is used to score the degradation features of each image (img) in the image set (IMG) using a combination of traditional methods and multimodal large model; the two degradation scores are weighted and summed to obtain the image degradation score; the probability distribution of each degradation feature score in the image set (IMG) is calculated; and a high-quality image set (H) is selected based on the scores. Construct an image discrimination dataset unit to build high-quality positive samples from images in the high-quality image set H through data augmentation; construct low-quality negative samples by randomly selecting and adding single or fused degradation features according to the degradation feature probability distribution; each sample contains an image, a quality label, and a normalized degradation score sequence. An image discriminator unit was built and trained to build a discriminator based on EfficientNetV2. Its architecture can judge the quality of the input image and output a degradation score sequence. The dataset was divided into a training set and a validation set according to a set ratio. The quality label and degradation score sequence of the sample were used as the ground truth, and binary cross-entropy and L1 loss were used as loss functions for training. The image restoration model construction and training unit is used to add various degradation features to high-quality images to build a dataset. The degradation score is embedded into a high-dimensional latent space through a fully connected network MLP. The restoration model is built based on Transformer-UNet and trained using various loss functions to achieve high-quality image restoration. The specific method for training the image restoration model using multiple loss functions in the image restoration model construction and training unit is as follows: Reconstruction loss: includes L1 loss and SSIM loss, representing the high-quality image h after restoration. p It should be consistent with the original high-quality image h; the formula is as follows: ， ， in, This represents the pixels in the original high-quality image h. h represents the high-quality image that has been repaired. p The pixels in Indicates the position of a pixel in the image. and For images h and h p The average pixel value, and For images h and h p pixel variance For images h and h p covariance, and It is the stability constant; Discriminant loss: Binary cross-entropy loss, representing the high-quality image h after restoration. p The discriminator should classify it as high quality: ， Where y represents a high-quality tag; Similarity loss: Cosine similarity loss, representing the consistency between the two embeddings P_em and T_em, is calculated as follows: ， The total loss function formula is as follows: ， in, These are the weights of their respective loss functions.

Images