Haze visibility detection method, device, equipment, vehicle and medium
By using continuous multi-frame image frames to train the haze detection model, the problem of single-frame image recognition is solved, the accuracy of haze visibility recognition and the efficiency of the model are improved, and the accurate prediction of haze visibility is achieved.
Patent Information
- Application Number
- CN202411026446.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-30
- Publication Date
- 2024-11-19
- Estimated Expiration
- 2044-07-30
AI Technical Summary
The prior art is prone to interference when identifying haze information in a single frame image, resulting in large errors in the haze visibility identification results and cannot be accurately identified.
By training the model to use continuous multi-frame image frames as samples, continuous changes in time and space are captured, interference during single-frame image modeling is avoided, and the accuracy of the haze detection model is improved.
It improves the accuracy of haze visibility identification results, realizes accurate prediction of haze visibility, reduces redundancy of computing resources and memory usage of models, and improves processing performance and deployability.
Smart Images

Figure CN118552884B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of data processing technology, and in particular to a haze visibility detection method, device, equipment, vehicle and medium. Background Art
[0002] In the related art, when identifying haze visibility, the haze visibility can often be determined by identifying a single-frame image through a model; the inventors found that it is easy to be disturbed when identifying haze information in a single-frame image, and when disturbed, the model cannot accurately perform recognition processing, which will result in a large error in the identified haze visibility and the inability to accurately identify the haze visibility. Summary of the invention
[0003] The present invention aims at the technical problem that the visibility of haze cannot be accurately identified in the prior art, and provides a haze visibility detection method, device, equipment, vehicle and medium.
[0004] The present invention provides a haze visibility detection method, comprising:
[0005] The haze video to be detected is identified by a haze detection model to obtain a haze visibility identification result corresponding to the haze video to be detected; the haze detection model is trained based on a plurality of haze detection samples; each of the haze detection samples includes a plurality of continuous image frames in the same haze video segment.
[0006] The present invention also provides a haze visibility detection device, comprising:
[0007] The recognition module is used to recognize the haze video to be detected through a haze detection model to obtain a haze visibility recognition result corresponding to the haze video to be detected; the haze detection model is trained based on multiple haze detection samples; each of the haze detection samples includes multiple consecutive image frames in the same haze video segment.
[0008] The present invention also provides a computer device, comprising a memory, a processor, and computer-readable instructions stored in the memory and executable on the processor, wherein the processor implements the above-mentioned haze visibility detection method when executing the computer-readable instructions.
[0009] The present invention also provides a vehicle, comprising a controller, wherein the controller is used to execute the above-mentioned haze visibility detection method.
[0010] The present invention also provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer-readable instructions, and when the computer-readable instructions are executed by a processor, the above-mentioned haze visibility detection method is implemented.
[0011] In the haze visibility detection method, device, equipment, vehicle and medium provided by the present invention, the haze visibility detection method comprises: identifying a haze video to be detected by a haze detection model to obtain a haze visibility recognition result corresponding to the haze video to be detected; the haze detection model is trained according to a plurality of haze detection samples; each of the haze detection samples includes a plurality of continuous image frames in the same haze video segment. Since the haze detection samples used to train the haze detection model include a plurality of continuous image frames in the same haze video segment, the haze detection model is trained according to the haze detection samples, and the continuous changes in time and space of the plurality of continuous image frames in the haze detection sample can be captured. Since the changes between the plurality of image frames are continuous, the modeling is carried out in an orderly manner according to the above-mentioned continuous changes, and the situation that a single frame image is easily disturbed when modeling is carried out as a sample can be avoided, thereby improving the accuracy of the haze detection model finally trained, thereby improving the accuracy of the haze visibility recognition result obtained by detecting the haze video to be detected by the haze detection model, and realizing the accurate prediction of haze visibility. BRIEF DESCRIPTION OF THE DRAWINGS
[0012] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.
[0013] Figure 1 is a flow chart of a haze visibility detection method according to an embodiment of the present invention;
[0014] Figure 2 is a flow chart of a haze visibility detection method according to another embodiment of the present invention;
[0015] Figure 3 is a flow chart of step S20 of the haze visibility detection method in one embodiment of the present invention;
[0016] Figure 4 is a flow chart of step S30 of the haze visibility detection method in one embodiment of the present invention;
[0017] Figure 5 is a flow chart of step S301 of the haze visibility detection method in one embodiment of the present invention;
[0018] Figure 6 is a principle block diagram of a haze visibility detection device in one embodiment of the present invention;
[0019] Figure 7 is a schematic diagram of a computer device in one embodiment of the present invention. DETAILED DESCRIPTION
[0020] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0021] The present invention provides a method for detecting haze visibility. In one embodiment, Figure 1 As shown, the haze visibility detection method includes the following steps S10:
[0022] S10, identifying the haze video to be detected by the haze detection model, and obtaining the haze visibility recognition result corresponding to the haze video to be detected; the haze detection model is trained according to multiple haze detection samples; each of the haze detection samples includes multiple continuous image frames in the same haze video segment. Among them, the haze video segment refers to a video segment cut out from the historical haze video, and the duration of the haze video segment can be set according to the demand, for example, all haze video segments can be the same preset duration. However, in some embodiments, different haze video segments can also be cut into different durations, which is not limited in the present invention. The haze detection sample includes multiple continuous image frames extracted from the haze video segment. In this embodiment, the continuous multiple image frames here can refer to all the image frames in the haze video segment, or can refer to a part of the image frames extracted from all the image frames in time sequence. In this embodiment, the continuity of multiple image frames in the same haze detection sample means that the multiple image frames are arranged from the beginning to the end in time sequence.
[0023] Among them, the haze video to be detected can be video data shot in real time, or it can be video data shot in advance and stored in a database. Before the haze detection model recognizes the haze video to be detected, it can refer to the above-mentioned haze detection sample processing method to pre-process the haze video to be detected (the haze video to be detected can refer to the haze detection sample processing method in the previous or later text for pre-processing, which is not limited here). For example, the haze video to be detected can be cut into video segments of preset duration, and a preset number of video frames are extracted from the video segments in chronological order. After all video frames are enhanced, part or all of the video frames corresponding to the same video segment are input into the haze detection model as an identification content for identification, and then the haze visibility identification result corresponding to the video segment of the haze video to be detected is obtained.
[0024] In one embodiment, after a haze detection model is obtained by training a plurality of haze detection samples, the haze detection model can be used to identify the haze video to be detected, and the obtained haze visibility recognition result includes the probability of the haze detection model predicting the haze visibility type in the haze video to be detected. That is, the haze visibility recognition result includes the visibility type probabilities corresponding to different haze visibility types. For example, when the haze visibility categories include high visibility (above 200 meters), relatively high visibility (100 to 200 meters), general visibility (50 to 100 meters) and weak visibility (within 50 meters), the haze detection model will determine the visibility type probabilities corresponding to the above four haze visibility types, and then output all visibility type probabilities and associated visibility type probabilities as haze visibility recognition results. Among them, the haze visibility type with the highest probability in the haze visibility recognition result is the haze visibility type of the haze video (video segment) to be detected predicted by the haze detection model.
[0025] Furthermore, the haze video to be detected is a haze video taken in real time by a camera installed on a vehicle. That is, when the haze video to be detected is a haze video taken by a camera installed on a vehicle, the haze visibility detection method can detect the haze visibility type corresponding to the haze in the environment where the vehicle is located in real time.
[0026] In the above embodiment of the present invention, since the haze detection sample used to train the haze detection model includes multiple consecutive image frames in the same haze video clip, the haze detection model is trained according to the haze detection sample, and the continuous changes of the multiple consecutive image frames in the haze detection sample in time and space can be captured. Since the changes between the multiple image frames are continuous, modeling is carried out in an orderly manner according to the above continuous changes, which can avoid the situation where a single frame image is easily interfered when used as a sample for modeling, thereby improving the accuracy of the haze detection model finally trained, and further improving the accuracy of the haze visibility recognition result obtained by detecting the haze video to be detected through the haze detection model, thereby achieving accurate prediction of haze visibility.
[0027] In one embodiment, if Figure 2 As shown, before the haze detection model is used to identify the haze video to be detected, the following steps S20-S30 are included:
[0028] S20, obtaining a training sample set, wherein the training sample set includes a plurality of haze detection samples. Each of the haze detection samples includes a plurality of continuous image frames in the same haze video clip; wherein the haze video clip refers to a video clip cut out from a historical haze video, and the duration of the haze video clip can be set according to demand, for example, all haze video clips can be of the same preset duration. However, in some embodiments, different haze video clips can also be cut into different durations, which is not limited in the present invention. The training sample set includes a plurality of haze detection samples, and the haze detection samples include a plurality of continuous image frames extracted from the haze video clip. In this embodiment, the continuous plurality of image frames here may refer to all image frames in the haze video clip, or may refer to a portion of image frames extracted from all image frames in time sequence. In this embodiment, the continuity of the plurality of image frames in the same haze detection sample refers to the plurality of image frames being arranged from first to last in time sequence.
[0029] In one embodiment, if Figure 3 As shown, before step S20, that is, before obtaining the training sample set, the following steps are included:
[0030] S201, cutting the historical smog video into smog video segments of preset duration; that is, the historical smog video may refer to the smog video continuously shot and stored by a camera device before the current time point on the city and urban block expressway or other roads. The historical smog video can be pre-stored in a database such as the OpenCV library, so that it can be retrieved from the database at any time when the historical smog video needs to be cut. The preset duration can be set according to demand, such as setting the preset duration to 1 second. In this step, multiple segments of historical smog videos can be clipped to obtain multiple smog video segments.
[0031] S202, extracting a preset number of image frames from each of the haze video segments in chronological order; that is, since the image frames in the same haze video segment are dense and continuous, there is a large amount of redundant information. Therefore, in order to avoid processing too much repeated data, which causes the haze detection samples finally trained to be overfitted, in this embodiment, the haze video segments are subjected to frame extraction processing in chronological order, so that the frame rate of the haze video segments corresponding to all the image frames obtained after the frame extraction processing is reduced (for example, the frame rate of the haze video segments before the frame extraction is 10Hz, and the frame rate of the haze video segments corresponding to the image frames after the frame extraction is 5Hz, that is, one frame is extracted from every two image frames). In this way, while retaining the temporal sequence of the continuous image frames, redundant data can also be removed.
[0032] S203, after performing enhancement processing on all the image frames, determining the image frames corresponding to the same haze video clip as a training sample, and determining a haze detection sample from all the training samples.
[0033] Understandably, enhancing each image frame can effectively increase the diversity of data, thereby improving the robustness of the final haze detection model. In this embodiment, after all image frames are enhanced one by one, all image frames corresponding to the haze video clip of the preset duration can be used as a training sample as the input of the model. It should be noted that each training sample needs to be associated with a sample label (such as the first sample label and the second sample label mentioned later), and the sample label is used to characterize the haze visibility category in the haze video clip corresponding to the training sample, and the haze visibility category includes but is not limited to high visibility (above 200 meters), high visibility (100 to 200 meters), general visibility (50 to 100 meters) and weak visibility (within 50 meters).
[0034] Specifically, all training samples can form a sample library, and then, the training sample set, verification sample set and test sample set can be determined from the sample library respectively; understandably, all training samples in the sample library can be randomly divided into training sample sets, verification sample sets and test sample sets according to a preset ratio. Specifically, the ratio between the training sample set, the verification sample set and the test sample set can be 6:2:2 or 8:1:1, or other ratios set according to needs, which are not limited here. Understandably, the training samples in the above-mentioned training sample set are haze detection samples, the training samples in the verification sample set are haze verification samples, and the training samples in the test sample set are haze test samples.
[0035] In one embodiment, in step S203, the step of performing enhancement processing on all the image frames includes:
[0036] All the image frames are scaled according to a preset size; specifically, each image frame can be scaled to a preset pixel size (e.g., 224 pixels) so that multiple consecutive image frames in the same haze detection sample can be synchronously processed simultaneously through the initial network model (or other models such as the haze detection model) in the future.
[0037] A target occlusion area is randomly selected on the image frame, and the target occlusion area is occluded. That is, in order to enhance the learning ability and robustness of the initial network model (or other models such as the haze detection model) for the data in the image frame, the randomly selected target occlusion area in the image frame is occluded. For example, a 30×30 rectangular box is randomly selected on each image frame as the target occlusion area, and then the element value corresponding to the target occlusion area is set to 0, thereby effectively increasing the diversity of the data, enhancing the image frame data, avoiding overfitting, and improving the robustness of the model to the data.
[0038] S30, synchronously inputting multiple consecutive image frames corresponding to the same haze video clip in the haze detection sample into the initial network model, so as to train the initial network model according to the haze detection sample to obtain the haze detection model. That is, in this embodiment, the haze detection samples in the training sample set can be input into the initial network model one by one, and then each haze detection sample is processed by the initial network model, and then the initial network model is iteratively trained to finally obtain the haze detection model.
[0039] Understandably, in the above training process, the initial network model can synchronously extract features and process in parallel all image frames in the same haze detection sample, thereby realizing parameter sharing between different image frames, so as to associate haze feature data between different image frames, and then capture the continuous changes of haze feature data in image frames in time and space (wherein, time change refers to the change in the time sequence of multiple consecutive image frames, and spatial change refers to the change of haze feature data on consecutive image frames over time, such as haze position and texture changes, etc.), and then orderly modeling according to the above continuous changes can avoid the situation that a single frame image is easily disturbed when modeling as a sample, thereby improving the accuracy of the haze detection model finally trained. Wherein, the above parameter sharing refers to the simultaneous input of multiple image frames corresponding to the same haze detection sample in the initial network model, so an initial network model can process and calculate multiple image frames at the same time, which can realize parameter sharing. Compared with the model training scheme that only inputs and processes one frame of image at a time, the above-mentioned parameter sharing setting of the present invention can realize reverse derivative calculations on multiple image frames at the same time to optimize the model, which will not produce bias in the model optimization results (when the model only optimizes one frame of image frames, the optimization results may be biased).
[0040] In the above embodiment of the present invention, the haze detection sample used to train the haze detection model includes multiple consecutive image frames in the same haze video clip. In this way, the initial network model is trained according to the haze detection sample, and the continuous changes of the multiple consecutive image frames in the haze detection sample in time and space can be captured. Since the changes between the consecutive multiple image frames are continuous, modeling is carried out in an orderly manner according to the above continuous changes, which can avoid the situation where a single frame image is easily interfered when used as a sample for modeling, thereby improving the accuracy of the haze detection model finally trained, and further improving the accuracy of the haze visibility recognition result detected by the haze detection model, and realizing accurate prediction of haze visibility. At the same time, since a haze detection sample includes multiple continuous image frames, the parameters in the multiple continuous image frames can be shared in the process of processing the haze detection sample (the continuous changes in time and space of the multiple continuous image frames in the haze detection sample can be shared). In this way, repeated calculations and storage can be reduced, and the utilization rate of computing resources is improved; and since a haze detection model in the present invention can synchronously process multiple image frames in the same haze detection sample, the number of models used to process image frames is reduced, thereby reducing the memory space occupied by the model. While ensuring the processing performance of the model, the model is lightweight, and the haze detection model also has higher efficiency and deployability.
[0041] In one embodiment, after step S30, that is, after training the initial network model according to the haze detection sample to obtain the haze detection model, the method further includes:
[0042] The haze detection model is verified by the verification sample set to obtain the verified haze detection model; the verification sample set includes multiple haze verification samples, and each of the haze verification samples includes multiple verification images in the same haze video clip. Understandably, after the haze detection model is obtained through training, it is preliminarily determined that the haze detection samples meet the training requirements. However, in order to further determine whether the haze detection model does meet the training requirements (the training requirements may refer to the accuracy or loss value, such as the accuracy of the training sample set reaches 99% or the loss value is reduced to 0.0001), the haze detection model needs to be further verified by the verification sample set. Since the haze verification samples included in the verification sample set are actually training samples determined in the same way as the haze detection samples, the specific process of verifying the haze detection model by the verification sample set to obtain the verified haze detection model can refer to the process of obtaining the haze detection model by training the initial network model according to the haze detection samples in step S30, and the haze detection model can be iteratively trained, which will not be repeated here. It can be understood that after continuing to iteratively train the haze detection model through the verification sample set, if the recognition accuracy of the haze detection model after the iterative training for haze visibility (that is, the accuracy of the haze recognition result) is higher than that of the initial haze detection model obtained in step 20, then the haze detection model after iterative training through the verification sample set is the verified haze detection model; if the recognition accuracy of the haze detection model after the iterative training for haze visibility is not higher than that of the initial haze detection model obtained in step 20, then the initial haze detection model is the verified haze detection model.
[0043] In one embodiment, after obtaining the verified haze detection model, the method further includes:
[0044] The verified haze detection model is tested by using a test sample set to obtain a tested haze detection model; the test sample set includes a plurality of haze test samples, and each of the haze test samples includes a plurality of test image frames in a same haze video clip.
[0045] That is, in this embodiment, the verified haze detection model is finally evaluated and tested according to the test sample set. Since the haze test samples included in the test sample set are actually training samples determined in the same way as the haze detection samples, the specific process of testing the haze detection model through the test sample set to obtain the tested haze detection model can refer to the process of training the initial network model according to the haze detection samples in step S30 to obtain the haze detection model, and the verified haze detection model can be iteratively trained, which will not be repeated here. It can be understood that after the verified haze detection model is iteratively trained through the test sample set, if the recognition accuracy of the haze visibility of the iteratively trained haze detection model (that is, the accuracy of the haze recognition result) is higher than that of the verified haze detection model, at this time, the haze detection model iteratively trained through the test sample set is the tested haze detection model; if the recognition accuracy of the haze visibility of the iteratively trained haze detection model is not higher than that of the verified haze detection model, at this time, the verified haze detection model is the tested haze detection model.
[0046] In one embodiment, if Figure 4 As shown, in the step S30, the initial network model is trained according to the haze detection sample to obtain a haze detection model, including:
[0047] S301, inputting the haze detection sample into the initial network model, and identifying the haze identification result corresponding to the haze detection sample through the initial network model; the haze detection sample is associated with a first sample label; wherein the first sample label is used to characterize the haze visibility category in the haze video clip corresponding to the haze detection sample, and the haze visibility category includes but is not limited to high visibility (above 200 meters), relatively high visibility (100 to 200 meters), general visibility (50 to 100 meters) and weak visibility (within 50 meters) and other categories.
[0048] S302, iteratively train the initial network model according to the haze recognition result and the first sample label to obtain a trained haze detection model. In one embodiment, an end-to-end initial network model can be built using the PyTorch framework. In the training stage of identifying the haze recognition result corresponding to the haze detection sample through the initial network model, it is only necessary to input the haze detection sample into the initial network model, and then optimize the model parameters of the initial network model through the back propagation algorithm according to the haze recognition result and the first sample label, thereby obtaining the final haze detection model.
[0049] In one embodiment, after step S301, that is, iteratively training the initial network model according to the haze recognition result and the first sample label to obtain a trained haze detection model includes:
[0050] When it is confirmed that the loss function value corresponding to the initial network model stops decreasing according to the haze recognition result and the first sample label, the learning rate of the initial network model is updated and reduced according to the target ratio; that is, during the training process of the initial network model, the loss function value corresponding to the initial network model is used to measure the difference between the predicted value of the initial network model (that is, the haze recognition result) and the actual value (the first sample label). In this embodiment, when the loss function value stops decreasing, it means that the initial network model is close to convergence. At this time, the learning rate is reduced according to the target ratio, and the adjustment of the model parameters can be further refined when the initial network model is close to convergence. It can be understood that the target ratio includes but is not limited to one tenth. That is, when the target ratio is one tenth, updating and reducing the learning rate of the initial network model according to the target ratio means attenuating the learning rate to one tenth of the original learning rate. It can be understood that when it is confirmed according to the haze recognition result and the first sample label that the loss function value corresponding to the initial network model stops decreasing, the model parameters of the initial network model are updated, and the initial network model after the updated model parameters is iteratively trained through the next haze detection sample, until it is confirmed that the loss function value corresponding to the initial network model after the updated model parameters stops decreasing, the learning rate of the initial network model is reduced according to the target ratio update. The above process can be repeated according to the actual training status, and will not be repeated here.
[0051] The updated initial network model is used to identify a new haze recognition result corresponding to the next haze detection sample; the next haze detection sample is associated with a second sample label; wherein the second sample label is used to characterize the haze visibility category in the haze video clip corresponding to the next haze detection sample. That is, since the initial network model has not yet completed convergence, the next haze detection sample needs to be input into the initial network model after the learning rate is updated, and a new haze recognition result continues to be obtained, so as to further iteratively train the initial network model after the learning rate is updated.
[0052] When the current learning rate is confirmed to be reduced to the preset learning rate threshold according to the new haze identification result and the second sample label, the current learning rate is kept unchanged, and when the loss function value no longer decreases, the updated initial network model is determined as the haze detection model. Among them, the preset learning rate threshold can be set according to the demand. For example, when the initial learning rate of the initial network model is 0.001, the preset learning rate threshold can be 0.000001. That is, when the current learning rate drops to 0.000001, further adjustment of the current learning rate will be stopped and kept unchanged. At this time, the updated initial network model will continue to be further iteratively trained according to the next haze detection sample until the loss function value no longer decreases, the training is stopped, and the current latest initial network model is determined as the haze detection model. In the above embodiment of the present invention, the learning rate attenuation and attenuation early stopping strategy are combined to further improve the model performance, thereby obtaining a haze detection model that can accurately and real-time judge the haze visibility type of the environment corresponding to the haze detection sample.
[0053] In one embodiment, the initial network model includes a feature extraction network and a Transformer (a self-attention mechanism structure) network connected to the feature extraction. In this embodiment, the feature extraction network can effectively extract haze feature data in the image frame, and then the haze feature data is input into the Transformer network. Compared with the solution of directly inputting the image frame into the Transformer network, this embodiment can better focus on the haze feature data corresponding to the haze in the image frame, making the initial network model processing process more robust.
[0054] Furthermore, if Figure 5 As shown, in the step S301, identifying the haze recognition result corresponding to the haze detection sample by using the initial network model includes the following steps S3011-S3012:
[0055] S3011, synchronously determine the haze feature blocks of all the image frames in the haze detection sample through the feature extraction network, and input all the haze feature blocks into the Transformer network according to the corresponding time sequence coding; understandably, the above haze feature blocks are generated based on the haze feature data extracted from the image frame, and the haze feature data includes semantic information and visual features, and the visual features may refer to the haze texture features in the image frame and other image information used to characterize the shape and state of the haze; semantic features refer to the semantic information used to describe the corresponding visual features. The time sequence coding has been determined when the haze detection sample is generated, and the time sequence coding characterizes the time sequence of each image frame in the haze video segment corresponding to the haze detection sample.
[0056] In this embodiment, all the haze feature blocks are input into the Transformer network according to the corresponding time series coding. In this way, a model lightweight strategy of shared parameters is adopted when processing continuous image frames to ensure that different image frames are trained and learned at the same time, and the learning features (that is, haze feature data or haze feature blocks) between different image frames can be associated according to the time series coding, so as to better capture the continuity and variability of the environmental information in the haze video clip corresponding to the image frame, thereby improving the accuracy and robustness of the haze recognition result output by the haze detection model, and also reducing the demand for storage space and computing resources, improving efficiency and deployability.
[0057] In one embodiment, the feature extraction network includes a modified ResNet50 (deep residual network) network with the global pooling layer and the fully connected layer removed, and a 1×1 convolution kernel connected between the last structural block of the modified ResNet50 network and the Transformer network. That is, in this embodiment, the modified ResNet50 network removes the global pooling layer and the fully connected layer to facilitate access to the Transformer network. The modified ResNet50 network is used to synchronously extract the haze feature data of all frames of the image frame in the haze detection sample, and then compress the haze feature data through the 1×1 convolution kernel to obtain the haze feature block corresponding to each image frame, and then all the haze feature blocks are input into the Transformer network according to the corresponding time series coding for further processing.
[0058] Furthermore, the step S3011, i.e., synchronously determining the haze feature blocks of all the image frames in the haze detection sample through the feature extraction network, and inputting all the haze feature blocks into the Transformer network according to the corresponding time sequence coding, includes:
[0059] The modified ResNet50 network is used to synchronously extract the haze feature data corresponding to all the image frames in the haze detection sample; in this embodiment, the modified ResNet50 network is used to synchronously extract the haze feature data of all the image frames in the haze detection sample. The haze feature data includes semantic information and visual features. The visual features may refer to image information such as haze texture features in the image frame used to characterize the shape and state of the haze; the semantic features refer to the semantic information used to describe the corresponding visual features.
[0060] Each of the haze feature data is compressed into a haze feature block of a preset size by the 1×1 convolution kernel, and all the haze feature blocks are input into the Transformer network according to the corresponding time series coding. The 1×1 convolution kernel is used to compress the haze feature data to obtain a haze feature block of a preset size corresponding to each image frame, wherein the preset size can be set according to demand. For example, the size of the haze feature data extracted by the modified ResNet50 network is 512×7×7, wherein 512 is the dimension and 7×7 is the size corresponding to the feature map in the haze feature data. At this time, the 1×1 convolution kernel is used to compress the feature and compress it into a haze feature block of 3×7×7 (preset size), that is, the dimension of the previous haze feature data is compressed from 512 channels to three channels. In this way, the number of channels corresponding to all image frames is adjusted, and the information of different channels is integrated, which is helpful to extract higher-level feature representations, realize further compression and feature extraction, and reduce the number of parameters to be processed while ensuring model performance. Afterwards, the haze feature blocks of preset size can be combined with the time sequence codes determined in the haze detection samples and sequentially input into the encoder of the Transformer network. It can be understood that in the above embodiment, the haze feature blocks corresponding to different image frames are input into the encoder as segmentation blocks according to the time sequence codes, that is, the segmentation blocks corresponding to different image frames are embedded with independent and fixed time sequence codes, so that the time features corresponding to the time sequence codes of continuous image frames are combined with the spatial changes of the image frames, so as to effectively capture the changes in spatial and temporal information in continuous image frames, and extract the contextual relationship between continuous different image frames, thereby significantly improving the accuracy and robustness of the haze detection model finally trained.
[0061] S3012, the Transformer network is instructed to determine and output the haze recognition result corresponding to the haze detection sample according to all the haze feature blocks. That is, in this embodiment, the deeper haze feature blocks extracted by the feature extraction network are used as the input of the encoder of the Transformer network, so that the feature representation of the deeper haze feature blocks can provide more comprehensive visual information, thereby enhancing the accuracy of the haze detection model finally trained for haze visibility detection and improving the image processing performance.
[0062] In one embodiment, the Transformer network includes an encoder and a fully connected layer connected to the encoder. The encoder of the Transformer network performs weighted similarity calculation on different segmented blocks (haze feature blocks) and finally outputs a probability value (haze recognition result) through the fully connected layer.
[0063] Furthermore, the step S3012, i.e., instructing the Transformer network to determine and output a haze recognition result corresponding to the haze detection sample according to all the haze feature blocks, includes:
[0064] The encoder performs weighted similarity calculation on all the haze feature blocks to determine the visibility type probability corresponding to the haze detection sample, generates a haze recognition result based on the visibility type probability and outputs it through the fully connected layer. That is, the encoder performs weighted similarity calculation on the haze feature blocks to obtain the visibility type probability corresponding to each haze visibility type. When the haze visibility categories include high visibility (above 200 meters), relatively high visibility (100 to 200 meters), general visibility (50 to 100 meters) and weak visibility (within 50 meters), the visibility type probabilities corresponding to the above four haze visibility types will be determined one by one, and then all visibility type probabilities and associated visibility type probabilities are output as haze recognition results. At this time, the haze visibility type of the haze detection sample predicted by the initial network model is the haze visibility type corresponding to the highest visibility type probability in the haze recognition result.
[0065] It should be understood that the order of execution of the steps in the above embodiment does not necessarily mean the order of execution. The order of execution of each process should be determined by its function and internal logic, and does not constitute any limitation on the implementation process of the embodiment of the present invention.
[0066] In one embodiment, if Figure 6 As shown, a haze visibility detection device is provided, and the haze visibility detection device corresponds to the haze visibility detection method in the above embodiment. The haze visibility detection device comprises:
[0067] The recognition module 11 is used to recognize the haze video to be detected through a haze detection model to obtain a haze visibility recognition result corresponding to the haze video to be detected; the haze detection model is trained based on multiple haze detection samples; each of the haze detection samples includes multiple consecutive image frames in the same haze video segment.
[0068] In the device of the above embodiment of the present invention, since the haze detection sample used to train the haze detection model includes multiple consecutive image frames in the same haze video clip, the haze detection model is trained according to the haze detection sample, and the continuous changes of the multiple consecutive image frames in the haze detection sample in time and space can be captured. Since the changes between the multiple image frames are continuous, modeling is carried out in an orderly manner according to the above continuous changes, which can avoid the situation where a single frame image is easily interfered when used as a sample for modeling, thereby improving the accuracy of the haze detection model finally trained, and further improving the accuracy of the haze visibility recognition result obtained by detecting the haze video to be detected through the haze detection model, thereby achieving accurate prediction of haze visibility.
[0069] For the specific definition of the haze visibility detection device, please refer to the definition of the haze visibility detection method above, which will not be repeated here. Each module in the above haze visibility detection device can be implemented in whole or in part by software, hardware and a combination thereof. The above modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.
[0070] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Figure 7 As shown. The computer device includes a processor, a memory, a network interface and a database connected via a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a readable storage medium and an internal memory. The readable storage medium stores an operating system, computer-readable instructions and a database. The internal memory provides an environment for the operation of the operating system and computer-readable instructions in the readable storage medium. When the computer-readable instructions are executed by the processor, a haze visibility detection method is implemented. The readable storage medium provided in this embodiment includes a non-volatile readable storage medium and a volatile readable storage medium.
[0071] In one embodiment, a computer device is provided, including a memory, a processor, and computer-readable instructions stored in the memory and executable on the processor, and the processor implements the steps of the above-mentioned haze visibility detection method when executing the computer-readable instructions.
[0072] In one embodiment, a vehicle is provided, comprising a controller, wherein the controller is configured to execute the above-mentioned haze visibility detection method.
[0073] For the specific definition of the controller, please refer to the definition of the haze visibility detection method above, which will not be repeated here. Each module in the above controller can be implemented in whole or in part by software, hardware and a combination thereof. The above modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.
[0074] In the vehicle of the above embodiment of the present invention, since the haze detection samples used to train the haze detection model include multiple consecutive image frames in the same haze video clip, the haze detection model is trained according to the haze detection samples, and the continuous changes of the multiple consecutive image frames in the haze detection samples in time and space can be captured. Since the changes between the multiple image frames are continuous, modeling is carried out in an orderly manner according to the above continuous changes, which can avoid the situation where a single frame image is easily interfered when used as a sample for modeling, thereby improving the accuracy of the haze detection model finally trained, and further improving the accuracy of the haze visibility recognition result obtained by detecting the haze video to be detected through the haze detection model, thereby achieving accurate prediction of haze visibility.
[0075] In one embodiment, a computer-readable storage medium is provided, on which computer-readable instructions are stored. When the computer-readable instructions are executed by a processor, the steps of the above-mentioned haze visibility detection method are implemented.
[0076] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment method can be implemented by instructing the relevant hardware through computer-readable instructions, and the computer-readable instructions can be stored in a readable storage medium, and the readable storage medium includes a non-volatile readable storage medium and a volatile readable storage medium. When the computer-readable instructions are executed, they may include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided by the present invention may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), memory bus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.
[0077] Those skilled in the art can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units or modules is used as an example. In actual applications, the above-mentioned functions can be assigned to different functional units or modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.
[0078] The embodiments described above are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included in the protection scope of the present invention.
Claims
1. A method for detecting haze visibility, characterized in that: include: The haze video to be detected is identified by using a haze detection model to obtain a haze visibility identification result corresponding to the haze video to be detected; The haze detection model is trained based on multiple haze detection samples; each of the haze detection samples includes multiple consecutive image frames in the same haze video clip; the multiple consecutive image frames corresponding to the same haze video clip in each of the haze detection samples are synchronously input into the initial network model, so that the initial network model synchronously performs feature extraction and parallel processing on all image frames in the same haze detection sample, thereby associating haze feature data between different image frames, and obtaining a haze recognition result of the haze detection sample; the haze detection model completes iterative training of the initial network model based on the haze recognition results corresponding to each of the haze detection samples.
2. The haze visibility detection method according to claim 1, characterized in that: Before the haze detection model is used to identify the haze video to be detected, the following steps are included: Acquire a training sample set, where the training sample set includes a plurality of haze detection samples; The continuous multiple image frames corresponding to the same haze video clip in the haze detection sample are synchronously input into the initial network model, so as to train the initial network model according to the haze detection sample to obtain the haze detection model.
3. The haze visibility detection method according to claim 2, characterized in that: Before obtaining the training sample set, the following steps are included: Cut historical smog videos into smog video clips of preset length; Extracting a preset number of image frames from each of the haze video clips in chronological order; After the enhancement processing is performed on all the image frames, the image frames corresponding to the same haze video clip are determined as a training sample, and a haze detection sample is determined from all the training samples.
4. The haze visibility detection method according to claim 3, characterized in that: The step of performing enhancement processing on all the image frames comprises: Scaling all the image frames according to a preset size; A target occlusion area is randomly selected on the image frame, and the target occlusion area is occluded.
5. The haze visibility detection method according to any one of claims 2 to 4, characterized in that: The training of the initial network model according to the haze detection sample to obtain a haze detection model includes: Identifying a haze recognition result corresponding to the haze detection sample by using the initial network model; associating the haze detection sample with a first sample label; The initial network model is iteratively trained according to the haze recognition result and the first sample label to obtain a trained haze detection model.
6. The haze visibility detection method according to claim 5, characterized in that: The initial network model includes a feature extraction network and a Transformer network connected to the feature extraction.
7. The haze visibility detection method according to claim 6, characterized in that: The identifying the haze recognition result corresponding to the haze detection sample by using the initial network model includes: Synchronously determining haze feature blocks of all the image frames in the haze detection sample through the feature extraction network, and inputting all the haze feature blocks into the Transformer network according to corresponding time sequence codes; The Transformer network is instructed to determine and output a haze recognition result corresponding to the haze detection sample according to all the haze feature blocks.
8. The haze visibility detection method according to claim 7, characterized in that: The feature extraction network includes a modified ResNet50 network with the global pooling layer and the fully connected layer removed, and a 1×1 convolution kernel connected between the last structural block of the modified ResNet50 network and the Transformer network.
9. The haze visibility detection method according to claim 8, characterized in that: The step of synchronously determining the haze feature blocks of all the image frames in the haze detection sample through the feature extraction network, and inputting all the haze feature blocks into the Transformer network according to the corresponding time sequence coding, comprises: Synchronously extracting haze feature data corresponding to each of the image frames in all frames in the haze detection sample through the modified ResNet50 network; Each of the haze feature data is compressed into the haze feature block of a preset size through the 1×1 convolution kernel, and all the haze feature blocks are input into the Transformer network according to corresponding time series coding.
10. The haze visibility detection method according to claim 7, characterized in that: The Transformer network includes an encoder and a fully connected layer connected to the encoder.
11. The haze visibility detection method according to claim 10, characterized in that: The step of causing the Transformer network to determine and output a haze recognition result corresponding to the haze detection sample according to all the haze feature blocks includes: The encoder performs weighted similarity calculation on all the haze feature blocks to determine the visibility type probability corresponding to the haze detection sample, and generates a haze recognition result according to the visibility type probability and outputs it through the fully connected layer.
12. The haze visibility detection method according to claim 5, characterized in that: The iterative training of the initial network model according to the haze recognition result and the first sample label to obtain a trained haze detection model includes: When it is confirmed according to the haze recognition result and the first sample label that the loss function value corresponding to the initial network model stops decreasing, updating and reducing the learning rate of the initial network model according to the target ratio; Identify a new haze recognition result corresponding to a next haze detection sample by using the updated initial network model; the next haze detection sample is associated with a second sample label; When it is confirmed that the current learning rate is reduced to a preset learning rate threshold according to the new haze identification result and the second sample label, the current learning rate is kept unchanged, and when the loss function value no longer decreases, the updated initial network model is determined as the haze detection model.
13. The haze visibility detection method according to claim 2, characterized in that: After the initial network model is trained according to the haze detection sample to obtain the haze detection model, the method further includes: The haze detection model is verified by using a verification sample set to obtain a verified haze detection model; the verification sample set includes a plurality of haze verification samples, and each of the haze verification samples includes a plurality of verification image frames in the same haze video clip.
14. The haze visibility detection method according to claim 13, characterized in that: After obtaining the verified haze detection model, the method further includes: The verified haze detection model is tested by using a test sample set to obtain a tested haze detection model; the test sample set includes a plurality of haze test samples, and each of the haze test samples includes a plurality of test image frames in a same haze video clip.
15. The haze visibility detection method according to claim 1, characterized in that: The smog video to be detected is a smog video taken in real time by a camera installed on a vehicle.
16. A haze visibility detection device, characterized in that: include: A recognition module, used for recognizing the haze video to be detected by using a haze detection model, and obtaining a haze visibility recognition result corresponding to the haze video to be detected; The haze detection model is trained based on multiple haze detection samples; each of the haze detection samples includes multiple consecutive image frames in the same haze video clip; the multiple consecutive image frames corresponding to the same haze video clip in each of the haze detection samples are synchronously input into the initial network model, so that the initial network model synchronously performs feature extraction and parallel processing on all image frames in the same haze detection sample, thereby associating haze feature data between different image frames, and obtaining a haze recognition result of the haze detection sample; the haze detection model completes iterative training of the initial network model based on the haze recognition results corresponding to each of the haze detection samples.
17. A computer device comprising a memory, a processor, and computer-readable instructions stored in the memory and executable on the processor, characterized in that: When the processor executes the computer-readable instructions, the haze visibility detection method according to any one of claims 1 to 15 is implemented.
18. A vehicle, characterized in that: It comprises a controller, wherein the controller is used to execute the haze visibility detection method according to any one of claims 1 to 15.
19. A computer-readable storage medium storing computer-readable instructions, characterized in that: When the computer-readable instructions are executed by a processor, the haze visibility detection method according to any one of claims 1 to 15 is implemented.
