Visibility analysis method and system, electronic equipment and storage medium

Abstract

The invention provides a visibility analysis method and system, electronic equipment and a storage medium, and relates to the field of visibility analysis. The visibility analysis method comprises the following steps: extracting a moving target in a target image according to two adjacent frames of target images; extracting a target area image from the target image based on the moving target; wherein the target area image comprises an image of a moving target; extracting image visibility characteristics from the target area image; wherein the image visibility characteristic represents the visibility corresponding to the target image; and determining a target visibility grade corresponding to the target image according to the image visibility characteristics. By using the visibility analysis method provided by the embodiment of the invention, the visibility under different weather conditions can be accurately analyzed, so that a reliable guarantee is provided for traffic safety, production safety and public safety.

Description

Technical Field

[0001] The present application relates to the field of visibility analysis, and in particular, to a visibility analysis method, system, electronic device and storage medium. Background Art

[0002] Weather has a certain impact on human transportation, production and life. Abnormal weather conditions may lead to reduced visibility, slippery roads, flight delays and other problems, causing great harm and losses to society and people. Therefore, how to quickly and accurately detect visibility in foggy weather has become one of the hot issues in the fields of transportation, aviation, meteorology, etc.

[0003] At present, the visibility detection in foggy weather mainly adopts the method of measuring with meteorological instruments. However, the setting points of meteorological instruments are limited, and comprehensive detection cannot be achieved; and the visibility data information obtained by the method of measuring with meteorological instruments is often lagging. Therefore, the current method of visibility detection is not only costly, but also not very real-time. Summary of the invention

[0004] The purpose of the embodiments of the present application is to provide a visibility analysis method, system, electronic device and storage medium, which obtains a target image through a video acquired by a shooting device; extracts image visibility features from the image after preprocessing the target image; and finally, determines the target visibility level corresponding to the target image based on the image visibility features. The visibility analysis method provided by the present application can not only calculate the visibility level with high accuracy, but also can be widely used in road traffic, air transportation, industrial production and other fields, providing reliable protection for traffic safety, production safety and public safety.

[0005] In a first aspect, an embodiment of the present application provides a visibility analysis method, the method comprising: extracting a moving target in a target image based on target images of two adjacent frames; extracting a target area image from the target image based on the moving target; wherein the target area image includes an image of the moving target; extracting image visibility features from the target area image; wherein the image visibility features characterize the visibility corresponding to the target image; and determining a target visibility level corresponding to the target image based on the image visibility features.

[0006] In the above implementation process, the preprocessing operations in the embodiment of the present application include denoising, smoothing, enhancement and inter-frame difference. First, the image or video is denoised to remove noise and interference, thereby improving the accuracy of visibility analysis; wherein, denoising can use methods based on wavelet transform, total variation, etc. Further, if it is video data, the video needs to be smoothed to minimize interference factors such as noise, jitter and vibration in the video sequence; it should be noted that smoothing algorithms based on Gaussian blur, mean filtering, etc. can be used. Further, if it is video data, the video needs to be differenciated between frames, the changes between video frames are detected, and the key frames are found to reduce the time and amount of calculation for subsequent processing.

[0007] Optionally, in an embodiment of the present application, after determining the visibility level corresponding to the target image according to the image visibility characteristics, the method also includes: comparing the target visibility level with the instrument visibility level to obtain a visibility comparison result; judging whether the visibility level is within the visibility error range; if it is determined that the visibility level is within the visibility error range, taking the target visibility level as the final visibility level of the target image.

[0008] In the above implementation process, in the visibility analysis method provided in the embodiment of the present application, after the target visibility level is calculated, the target visibility level can be compared with the visibility level obtained by the visibility data measured by traditional instruments; if the comparison result obtained is within the visibility error range, then the target visibility level is directly used as the final visibility level.

[0009] Optionally, in an embodiment of the present application, if it is determined that the visibility level is not within the visibility error range, the target visibility level of the target image is recalculated based on the instrument visibility data and the known visibility image.

[0010] In the above implementation process, the visibility analysis method provided in the embodiment of the present application can not only analyze the visibility based on the image or video information, but also compare and correct the measurement results of traditional meteorological instruments, continuously train and optimize the computer vision algorithm to improve the correctness and accuracy of the algorithm, and obtain more reliable and practical visibility level division results.

[0011] Optionally, in an embodiment of the present application, the image visibility features include natural features, and the natural features include brightness features; extracting the image visibility features from the target area image includes: performing color space conversion on the target area image to obtain an HSV image; extracting the image brightness of the HSV image, and obtaining brightness features based on the image brightness.

[0012] In the above implementation process, visibility can be measured by the degree of visible light passing through the atmosphere, and the intensity of visible light is related to brightness; under low visibility conditions, due to the presence of atmospheric pollutants such as haze, the intensity of visible light will be weakened, thereby reducing the brightness in the image; therefore, the brightness of different areas in the video can reflect the visibility of the area. Therefore, the visibility analysis method provided in the embodiment of the present application can greatly improve the accuracy of visibility level analysis based on the brightness characteristics of the target area image.

[0013] Optionally, in an embodiment of the present application, the natural features also include: color features; extracting image visibility features from the target area image, including: obtaining the pixel distribution of the target area image based on the target area image; calculating the pixel mean of each channel based on the pixel distribution, and obtaining color features based on the pixel mean.

[0014] In the above implementation process, under weather conditions with poor visibility, due to the presence of atmospheric pollutants such as haze, the color will become darker and grayer. Therefore, the color of different areas in the image can be used as an indicator for evaluating visibility, and there is a certain correlation with visibility. It can be seen that the visibility analysis method provided in the embodiment of the present application can effectively extract the color features of the target area image, which helps to more accurately determine the target visibility level.

[0015] Optionally, in an embodiment of the present application, the natural features also include texture features; extracting image visibility features from the target area image, including: obtaining a pixel matrix of the target area image based on the target area image; converting the pixel matrix into a grayscale co-occurrence matrix, and obtaining texture features based on the grayscale co-occurrence matrix.

[0016] In the above implementation process, since one of the main factors affecting visibility is aerosol concentration, and the type and concentration of aerosols will affect the scattering of light, thereby affecting the detailed information of the texture. In low visibility conditions, due to the presence of scattered light, the texture details of the object will be blurred, so there is also a certain correlation between the texture information and visibility in different areas of the video. For example, when driving in foggy weather, the texture of distant buildings and roads will become blurred, while the texture nearby will become more obvious. Therefore, the embodiment of the present application can improve the accuracy of visibility level analysis by analyzing the texture information of different areas in the video.

[0017] Optionally, in an embodiment of the present application, the image visibility feature includes a motion feature; extracting the image visibility feature from the target area image includes: extracting two frames of images to be analyzed corresponding to the target area image from the target image; determining the motion pixel area corresponding to the moving target in the two frames of images to be analyzed respectively; calculating the motion feature based on the motion pixel area; wherein the motion feature includes the motion direction of the moving target and / or the motion speed of the moving target.

[0018] In the above implementation process, the visibility analysis method provided in the embodiment of the present application can extract relevant motion features from the target area image, calculate the speed, direction and other information of the moving objects in continuous image frames, and accurately identify the moving objects and motion directions in the image.

[0019] Optionally, in an embodiment of the present application, before extracting a moving target in a target image based on target images of two adjacent frames, the method also includes: dividing the target video into multiple continuous frame sequences; calculating pixel differences between continuous frames in the continuous frame sequence; determining a preset target number of pixel differences from the pixel differences, and determining a target frame based on the pixel differences; and selecting a target image from the target frame.

[0020] In the above implementation process, in the visibility analysis method provided in the embodiment of the present application, the video is divided into a series of continuous frame sequences; for each sequence, the pixel values ​​between adjacent frames are calculated. The calculated pixel differences are sorted, and the pixel differences with the largest preset target are extracted, and the image corresponding to the larger difference is identified as the target frame. A more effective analysis basis can be obtained by screening the target frame, so that the visibility analysis method provided in the embodiment of the present application is performed on more reliable image information, which can improve the accuracy of visibility level analysis.

[0021] In a second aspect, an embodiment of the present application provides a visibility analysis system, which includes: an image analysis module, an image visibility feature extraction module and a visibility analysis module; the image analysis module is used to extract a moving target in a target image based on target images of two adjacent frames; the image analysis module is also used to extract a target area image from the target image based on the moving target; wherein the target area image includes an image of the moving target; the image visibility feature extraction module is used to extract image visibility features from the target area image; wherein the image visibility features characterize the visibility corresponding to the target image; the visibility analysis module is used to determine a target visibility level corresponding to the target image based on the image visibility features.

[0022] In a third aspect, an embodiment of the present application provides an electronic device, comprising a memory and a processor, wherein the memory stores program instructions, and when the processor reads and runs the program instructions, it executes the steps in any of the above implementation methods.

[0023] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer program instructions, and when the computer program instructions are read and executed by a processor, the steps in any of the above-mentioned implementation methods are executed. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments of the present application will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0025] Figure 1 A visibility analysis flow chart provided for an embodiment of the present application;

[0026] Figure 2 A visibility comparison flow chart provided for an embodiment of the present application;

[0027] Figure 3 A schematic diagram of the DenseNet network structure provided in the embodiment of the present application;

[0028] Figure 4 A schematic diagram of the Dense Block structure provided in an embodiment of the present application;

[0029] Figure 5 A flow chart of brightness feature extraction provided in an embodiment of the present application;

[0030] Figure 6 A color feature extraction flow chart provided for an embodiment of the present application;

[0031] Figure 7 A flowchart of extracting texture features provided in an embodiment of the present application;

[0032] Figure 8 A motion feature extraction flow chart provided in an embodiment of the present application;

[0033] Fig. 9 A target frame screening flow chart provided for an embodiment of the present application;

[0034] Fig.10 A schematic diagram of a confusion matrix for a training set provided in an embodiment of the present application;

[0035] Fig.11 A schematic diagram of a confusion matrix of a test set provided in an embodiment of the present application;

[0036] Fig.12A schematic diagram of a module of a visibility analysis system provided in an embodiment of the present application;

[0037] Fig.13 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0038] The technical solutions in the embodiments of the present application will be described below in conjunction with the accompanying drawings in the embodiments of the present application. For example, the flowcharts and block diagrams in the accompanying drawings show the possible architectures, functions and operations of the systems, methods and computer program products according to multiple embodiments of the present invention. In this regard, each box in the flowchart or block diagram may represent a module, a program segment or a part of a code, and the module, a program segment or a part of a code contains one or more executable instructions for implementing the specified logical functions. It should also be noted that in some alternative implementations, the functions marked in the box may also occur in an order different from that marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they may sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or the flowchart, and the combination of boxes in the block diagram and / or the flowchart, can be implemented by a dedicated hardware-based system that performs a specified function or action, or can be implemented by a combination of dedicated hardware and computer instructions. In addition, the functional modules in the various embodiments of the present invention can be integrated together to form an independent part, or each module can exist separately, or two or more modules can be integrated to form an independent part.

[0039] Weather has a certain impact on human transportation, production and life, such as fog, rain, snow, haze, sandstorms, etc. Abnormal weather conditions may lead to reduced visibility, slippery roads, flight delays and other problems, causing great harm and losses to society and people. Therefore, how to quickly and accurately detect visibility in foggy weather has become one of the hot issues in the fields of transportation, aviation, meteorology, etc.

[0040] The applicant found in the research process that, at present, the visibility detection for foggy weather mainly adopts the method of measuring with meteorological instruments. However, this method has problems such as limited measurement points, low real-time performance, and high cost. At the same time, the data of meteorological instruments need to go through a series of complex processing and analysis before the visibility level can be judged, which has a certain lag for real-time traffic management and safety decision-making.

[0041] Based on this, the present application provides a visibility analysis method, system, electronic device and storage medium; the visibility analysis method is based on real-time video or image data to analyze the visibility level under different weather conditions in real time; thereby providing reliable protection for traffic safety, production safety and public safety.

[0042] Before introducing the specific content of this application, a brief introduction to the currently common visibility detection instruments is first given.

[0043] Traditional meteorological instruments mainly include visibility meters, laser rangefinders, photoelectric converters, etc. Visibility meters mainly illuminate the measurement area with an infrared light source, then receive the reflected light signal, and calculate the visibility value based on the intensity of the received light signal. Laser rangefinders use laser beams to scan the measurement area, then measure the time and distance the laser beam travels in the air, and calculate the visibility value based on the time and distance. Photoelectric converters irradiate the measurement area with an infrared light source, then receive the reflected light signal, convert it into an electrical signal for measurement, and calculate the visibility value based on the intensity of the electrical signal.

[0044] However, when using visibility meters, laser rangefinders and photoelectric converters, they need to be placed at specific measurement points, and cannot detect and analyze the visibility of a large area in real time; in addition, visibility meters, laser rangefinders and photoelectric converters are expensive and require professional personnel to operate and maintain; in severe weather conditions such as rain and snow, the measurement data of visibility meters, laser rangefinders and photoelectric converters may be disturbed, resulting in large errors. On the other hand, the above instruments require complex data processing and analysis to determine the visibility level, and there is a certain lag.

[0045] Please see Figure 1 , Figure 1 A visibility analysis flow chart provided in an embodiment of the present application; in an embodiment of the present application, the visibility analysis method includes the following steps:

[0046] Step S100: extracting a moving target in a target image according to target images in two adjacent frames.

[0047] In the above step S100, the moving target in the target image is extracted based on the target images of two adjacent frames; it should be noted that, in the embodiment of the present application, the target image is an image of visibility to be analyzed; it can be extracted from the video, or it can be two consecutive frames of images taken by a camera at the location where visibility analysis is required.

[0048] In some embodiments, extracting moving targets based on the target image graph can utilize the changes in pixels in the image sequence in the time domain and the correlation between adjacent frames to find the correspondence between the previous frame and the current frame, thereby calculating the motion information of the object between adjacent frames, thereby achieving the extraction of moving targets in the image.

[0049] Step S101: extracting a target area image from a target image based on a moving target.

[0050] In the above step S101, the area where the target image is located is extracted from the target image. Specifically, a rectangular area centered on the moving target can be determined according to a certain ratio based on the position of the moving target on the target image; it should be noted that the rectangular area can include background information excluding the moving target. Further, the rectangular area is extracted using an image segmentation method. It can be understood by those skilled in the art that the image segmentation algorithm can be based on clustering, edge detection, and other image segmentation algorithms.

[0051] In some embodiments, a difference operation is performed on two adjacent frames of images to obtain a difference image; if a pixel value in the difference image is greater than a set threshold, the pixel point is considered to be a target area image.

[0052] Step S102: extracting image visibility features from the target area image.

[0053] In the above step S102, after the target area image is extracted, the image visibility feature is extracted from the target area image; it should be noted that the image visibility feature is a feature quantity that can characterize the visibility of the target image.

[0054] Step S103: Determine the target visibility level corresponding to the target image according to the image visibility feature.

[0055] In the above step S103, after the image visibility feature is determined according to the target area image, the target visibility level corresponding to the target image is determined according to the visibility feature analysis.

[0056] In an optional implementation manner, before extracting the moving target in the target image according to the target images of two adjacent frames, the visibility analysis method further includes:

[0057] Use an image acquisition device to acquire video or images of the visibility to be analyzed.

[0058] It is important to select the target angle and target position for image or video acquisition, such as a major traffic route, to ensure that the captured image or video can reflect abnormal weather conditions. When shooting, it is important to avoid factors that affect video quality, such as camera shake and insufficient light. Whether shooting images or videos, data continuity must be ensured to facilitate post-processing.

[0059] After acquiring the image or video, preprocessing operations are performed on the data to be processed.

[0060] In the above process, the preprocessing operations in the embodiment of the present application include denoising, smoothing, enhancement and inter-frame difference. First, the image or video is denoised to remove noise and interference, thereby improving the accuracy of visibility analysis; wherein, denoising can use methods based on wavelet transform, total variation, etc. Further, if it is video data, the video needs to be smoothed to minimize interference factors such as noise, jitter and vibration in the video sequence; it should be noted that smoothing algorithms based on Gaussian blur, mean filtering, etc. can be used. Further, if it is video data, the video needs to be differencing between frames, detecting changes between video frames, and finding key frames to reduce the time and amount of calculation for subsequent processing.

[0061] pass Figure 1 It can be seen that the visibility analysis method provided in the embodiment of the present application obtains a target image based on the video acquired by the shooting device; after preprocessing the target image, the image visibility features are extracted from the image; finally, the target visibility level corresponding to the target image is determined based on the image visibility features. Using the visibility analysis method provided in the embodiment of the present application, image processing and analysis based on computer vision algorithms can improve the accuracy and stability of visibility level division; the detection and level division of fog visibility can be completed in a relatively short time, with good real-time performance, which is of great significance to the fields of road traffic, air transportation, etc.

[0062] Please see Figure 2 , Figure 2 A visibility comparison flow chart provided for an embodiment of the present application; in an optional implementation manner of the embodiment of the present application, after determining the visibility level corresponding to the target image according to the image visibility features, the visibility analysis method further includes the following steps:

[0063] Step S200: Compare the target visibility level with the instrument visibility level to obtain a visibility comparison result.

[0064] In the above step S200, after the visibility level corresponding to the target image is determined according to the image visibility feature, the target visibility level and the instrument visibility level are compared to obtain a visibility comparison result. It should be noted that the instrument visibility level in the above process may be a visibility level of the corresponding area of ​​the target image output by an instrument capable of measuring the visibility level.

[0065] Step S201: Determine whether the visibility level is within the visibility error range.

[0066] Step S202: If it is determined that the visibility level is within the visibility error range, the target visibility level is used as the final visibility level of the target image.

[0067] In the above steps S201-S202, the target visibility level is compared with the instrument visibility level to obtain a visibility level comparison result; based on the comparison result, it is determined whether the visibility level is within the visibility error range. For example, if the determined target visibility level and the instrument visibility level differ by less than one visibility level, the target visibility level is deemed to meet the requirements and is within the visibility error range.

[0068] In some possible implementations, the visibility analysis method further includes: obtaining an instrument visibility level. That is, obtaining a visibility level measured from a traditional visibility measuring instrument; such as data on visibility measured by an optical visibility meter, a microwave radiometer, a laser radar, etc. Calibrate the collected raw data to eliminate errors that may be introduced by the instrument itself and the transmission process. For example, for a temperature sensor, temperature calibration can be performed to eliminate the error between the temperature sensor and a standard thermometer.

[0069] Furthermore, the calibrated data is preprocessed to extract data features related to visibility. For example, in foggy weather, data features such as humidity in the air need to be extracted to determine the current visibility level.

[0070] Furthermore, the processed data is visualized for easy observation and analysis. For example, the data is displayed in the form of charts, maps, etc. to help users better understand the meteorological data. The processed data is stored in a database to ensure data management and maintenance; at the same time, the data integrity and reliability are guaranteed.

[0071] It should be noted that comparing the target visibility level with the instrument visibility level is only used when the data base is not large enough, or in the process of accuracy correction of the visibility analysis method provided in the present application; it should be understood that when the visibility analysis method provided in the embodiment of the present application can realize visibility level analysis with a higher accuracy, there is no need to compare the target visibility level with the instrument visibility level.

[0072] pass Figure 2 It can be seen that in the visibility analysis method provided in the embodiment of the present application, after the target visibility level is calculated, the target visibility level can be compared with the visibility level obtained by the visibility data measured by traditional instruments; if the comparison result obtained is within the visibility error range, then the target visibility level is directly used as the final visibility level.

[0073] In an optional embodiment, if it is determined that the visibility level is not within the visibility error range, the target visibility level of the target image is recalculated based on the instrument visibility data and the known visibility image.

[0074] For the comparison between the instrument visibility data and the target visibility level, a convolutional neural network (CNN) can be used for video image processing and feature extraction, and a classification model can be used to predict the fog level. When training the model, it is necessary to prepare sufficient data sets, and perform data preprocessing, data enhancement and other operations to increase the generalization ability and stability of the model. In order to improve the performance of the model, various techniques can be used to optimize the model, such as learning rate adjustment, regularization and other methods.

[0075] According to the comparison results in the above process, the visibility level of the video data can be divided. In general, the visibility level is divided according to different visibility ranges, which are generally determined by the standards set by the meteorological departments of various countries. Taking the China Meteorological Administration as an example, the visibility range and visibility level specified by it are as follows: visibility greater than 10 kilometers, level 1; visibility 5-10 kilometers, level 2; visibility 1-5 kilometers, level 3; visibility 0.5-1 kilometers, level 4; visibility 0.2-0.5 kilometers, level 5; visibility 0-0.2 kilometers, level 6.

[0076] According to different visual ranges, we can classify video data into different levels. For example, when the visual range of video data is between 5-10 kilometers, it is classified into level 2 and the data is marked as level 2 data.

[0077] The present application embodiment exemplarily proposes a visibility level classification model (DenseNet network), please refer to Figure 3 , Figure 3 Schematic diagram of the DenseNet network structure provided in the embodiment of the present application; the DenseNet network mainly includes two component modules: DenseBlock and transition layer.

[0078] Dense Block is a densely connected highway module that defines the connection relationship between input and output; transition layer is the part between two adjacent Dense Blocks and controls the number of channels. Each Dense Block’s 3*3 convolution is preceded by a 1*1 convolution operation, which is the bottleneck layer. The purpose is to reduce the number of input feature maps, which can reduce the dimension and reduce the amount of calculation, and can also fuse the features of each channel. In order to further compress parameters, a 1*1 convolution operation is added between every two Dense Blocks to reduce the dimension, which is the transition layer.

[0079] The number of layers of each Dense Block module is 5, that is, it has multiple BN+ReLu+(1x1)Conv+BN+ReLu+(3*3)Conv layers, and the network growth rate is 4. Simply put, the dimension of the feature map output by each layer is 4. Please refer to Figure 4 , Figure 4 A schematic diagram of the Dense Block structure provided in the embodiment of the present application; Figure 4 In the example, input X0 is passed through the H1 convolution layer to obtain output X1. X1 and X0 are concatenated on the channel axis to obtain the aggregated feature tensor, which is sent to the H2 convolution layer to obtain output X2. In the same way, X2 is aggregated with the feature information X1 and X0 of all previous layers and then sent to the next layer. This cycle continues until the output X4 of the last layer and the feature information {x i} i＝0,1,2,3 Aggregation is performed to obtain the final output of the module. Such a module based on Skip Connection dense connection is called Dense Block.

[0080] Each Dense Block module utilizes the information of all previous layers in the module, that is, each layer has a dense highway connection with the previous layer. The dense connection of DenseNet is that each layer concatenates the input of all previous layers (channel-wise concatenation), and then passes the output feature map to all subsequent layers. Assuming a network with L layers, the number of highway dense connections is L*(L+1) / 2. The dense connection method of highway increases the transmission of gradients, reuses features, and even reduces overfitting on small sample data. Among them, the feature fusion in the dense convolution block adopts the concat method of x1=H1([x0,x1,…,x l-1 ]), distinguishes between information added to the network and information retained, and does not hinder the flow of information.

[0081] Bottleneck layers are composed of BN+ReLU+1×1Conv+BN+ReLU+3×3Conv. The bottleneck layer of 1×1 convolution is used to reduce the amount of calculation in Dense Blocks. The output channel number of 1×1 convolution is 4k, which plays a role in reducing the feature dimension.

[0082] It can be seen that the visibility analysis method provided in the embodiment of the present application can not only analyze visibility based on image or video information, but also compare and correct the measurement results of traditional meteorological instruments, continuously train and optimize computer vision algorithms, so as to improve the correctness and accuracy of the algorithm, and obtain more reliable and practical visibility level classification results.

[0083] Please see Figure 5 , Figure 5 A brightness feature extraction flow chart provided for an embodiment of the present application; in an optional implementation manner of an embodiment of the present application, the image visibility features include natural features, and the natural features include brightness features.

[0084] Extracting image visibility features from the target area image includes the following steps:

[0085] Step S300: performing color space conversion on the target area image to obtain an HSV image.

[0086] In the above step S300, in order to obtain the brightness feature of the image, it is necessary to perform color space conversion on the target area image, and convert the target area image into the HSV color space to obtain an HSV image.

[0087] It should be noted that HSV (Hue, Saturation, Value) is a color space about the intuitive characteristics of color; the color parameters in this model are: hue (H), saturation (S), and value (V).

[0088] Step S301: extracting the image brightness of the HSV image, and obtaining brightness features based on the image brightness.

[0089] In the above step S301, after the target area image is converted into an HSV image, the brightness of the HSV image is extracted to obtain the corresponding image brightness; and the image brightness feature value is obtained based on the image brightness. It should be noted that the image brightness can be directly used as the image brightness feature value, or the mean value of the image brightness can be calculated as the image brightness feature.

[0090] pass Figure 5 It can be seen that visibility can be measured by the degree of visible light passing through the atmosphere, and the intensity of visible light is related to brightness; under low visibility conditions, due to the presence of atmospheric pollutants such as haze, the intensity of visible light will be weakened, thereby reducing the brightness in the image; therefore, the brightness of different areas in the video can reflect the visibility of the area. Therefore, the visibility analysis method provided in the embodiment of the present application can greatly improve the accuracy of visibility level analysis based on the brightness characteristics of the target area image.

[0091] Please see Figure 6 , Figure 6 A color feature extraction flow chart provided for an embodiment of the present application; in an optional implementation manner of an embodiment of the present application, the natural feature also includes: color feature.

[0092] Extracting image visibility features from the target area image may include the following steps:

[0093] Step S400: acquiring pixel distribution of the target area image according to the target area image.

[0094] Step S401: Calculate the pixel mean of each channel according to the pixel distribution, and obtain the color feature based on the pixel mean.

[0095] In the above steps S400 and S401, in order to extract the color features of the image, the pixel distribution of the target area image is obtained; further, the pixel mean of each color channel is calculated, and then the color features of the image are obtained based on the pixel mean. It should be noted that the pixel distribution of the image can be represented by a pixel matrix, which should still be a vector or matrix after calculation and transformation; in practical applications, for the convenience of calculation, the corresponding vector or matrix is ​​often simply transformed to obtain a unified format to obtain the above color features.

[0096] Furthermore, in order to target different types of air pollutants, color features may be extracted from color channels of different rows with different weights.

[0097] pass Figure 6 It can be seen that under weather conditions with poor visibility, due to the presence of atmospheric pollutants such as haze, the color will become darker and grayer. Therefore, the color of different areas in the image can be used as an indicator for evaluating visibility and has a certain correlation with visibility. It can be seen that the visibility analysis method provided in the embodiment of the present application can effectively extract the color features of the target area image, which helps to more accurately determine the target visibility level.

[0098] Please see Figure 7 , Figure 7 A flow chart for extracting texture features provided in an embodiment of the present application; in an optional implementation of an embodiment of the present application, the natural features also include texture features.

[0099] Extracting the image visibility feature from the target area image may further include the following steps:

[0100] Step S500: obtaining a pixel matrix of the target area image according to the target area image.

[0101] In the above step S500, since the image texture is represented by the grayscale distribution of the pixel and its surrounding spatial neighborhood, that is, local texture information, in order to obtain the texture features of the target image, it is necessary to first obtain the pixel matrix of the target area image.

[0102] It should be noted that texture features are a kind of global features, which reflect the visual characteristics of homogeneous phenomena in the image and embody the surface tissue structure arrangement properties of the object surface with slow or periodic changes.

[0103] Step S501: converting the pixel matrix into a gray level co-occurrence matrix, and obtaining texture features based on the gray level co-occurrence matrix.

[0104] In the above step S501, the obtained pixel matrix is ​​converted into a gray level co-occurrence matrix. Since the texture is formed by the repeated appearance of gray level distribution in the spatial position, there will be a certain gray level relationship between two pixels separated by a certain distance in the image space, that is, the spatial correlation characteristics of gray level in the image. Therefore, in order to obtain the texture features of the target image area, it is necessary to first convert the pixel matrix into a gray level co-occurrence matrix.

[0105] Furthermore, after obtaining the gray-level co-occurrence matrix, the roughness, contrast, orientation, linearity, regularity and coarseness of the image are calculated respectively to obtain the final texture features.

[0106] pass Figure 7 It can be seen that since one of the main factors affecting visibility is aerosol concentration, and the type and concentration of aerosols will affect the scattering of light, thereby affecting the detailed information of the texture. In low visibility conditions, due to the presence of scattered light, the texture details of the object will be blurred, so there is a certain correlation between the texture information of different areas in the video and visibility. For example, when driving in fog, the texture of distant buildings and roads will become blurred, while the texture nearby will become more obvious. Therefore, the embodiment of the present application can improve the accuracy of visibility level analysis by analyzing the texture information of different areas in the video.

[0107] It should be noted that the embodiments of the present application extract the brightness features, color features and texture features of the image respectively; illustratively, the visibility is characterized by calculating the brightness, color, texture and other features of the image; and each video frame is converted into a feature vector using commonly used computer vision feature extraction methods, such as histogram of directional gradients (HOG), local binary pattern (LBP) and color histogram. These feature vectors can reflect information such as brightness, color and texture of different regions in the video.

[0108] Please see Figure 8 , Figure 8A motion feature extraction flow chart provided for an embodiment of the present application; in an optional implementation manner of an embodiment of the present application, the image visibility feature includes a motion feature.

[0109] Extracting the image visibility feature from the target area image also includes the following steps:

[0110] Step S600: extracting two frames of images to be analyzed corresponding to the target area image from the target image.

[0111] Step S601: determining the moving pixel regions corresponding to the moving target in the two frames of images to be analyzed respectively.

[0112] In the above steps S600-S601, two frames of images to be analyzed corresponding to the target area image are extracted from the target image, and the moving pixel areas corresponding to the moving targets determined in the two frames of images to be analyzed are respectively analyzed.

[0113] Step S602: Calculate motion features based on the motion pixel area.

[0114] In the above step S602, motion features are calculated based on the motion pixel area, and the motion features include the motion direction and / or the motion speed of the motion target; illustratively, the motion features can be calculated based on the motion pixel area using methods such as optical flow method and Kalman filtering.

[0115] pass Figure 8 It can be seen that the visibility analysis method provided in the embodiment of the present application can extract relevant motion features from the target area image, calculate the speed, direction and other information of the moving objects in continuous image frames, and can accurately identify the moving objects and motion directions in the image.

[0116] Please see Fig. 9 , Fig. 9 A target frame screening flow chart provided for an embodiment of the present application; in an optional implementation manner of the present application, before extracting a moving target in a target image based on target images of two adjacent frames, the method further includes the following steps:

[0117] Step S700: Divide the target video into a plurality of continuous frame sequences.

[0118] Step S701: Calculate pixel differences between consecutive frames in a sequence of consecutive frames.

[0119] Step S702: determining a preset target number of pixel differences from the pixel differences, and determining a target frame according to the pixel differences.

[0120] Step S703: Select a target image from the target frame.

[0121] In the above steps S700-S703, the video is divided into a series of continuous frame sequences; for each sequence, the pixel values ​​between adjacent frames are calculated. The calculated pixel differences are sorted, and the pixel differences with the largest preset target values ​​are extracted, and the image corresponding to the larger difference is identified as the target frame.

[0122] Exemplarily, a video file or a video stream is read using a video reading library, such as OpenCV; further, the frame rate of the video is determined; each frame is obtained using the video reading library, and the video frames are extracted one by one by looping through each frame in the video stream; each frame image is stored as an image file, such as a JPEG, PNG or other format, thereby achieving the extraction of continuous video frames from the original video at a certain time interval.

[0123] Furthermore, for each video frame, it is cropped into a size of 800x800 pixels, and the image area to be retained is determined according to the content of the video frame and the object to be analyzed; each frame is cropped according to the determined position; each frame after cropping should have the same size and content area; and the frame rate of the cropped image is adjusted; the cropped video frame is saved as an image file for subsequent analysis.

[0124] In some possible implementations, the target frame in the above process can be further screened; for example, the pixel difference between the key frame and the adjacent frame is calculated to determine whether it is a real key frame. If the difference between the key frame and the adjacent frame is small, it can be regarded as an unnecessary key frame and deleted.

[0125] pass Fig. 9 It can be seen that in the visibility analysis method provided in the embodiment of the present application, the video is divided into a series of continuous frame sequences; for each sequence, the pixel values ​​between adjacent frames are calculated. The calculated pixel differences are sorted, and the pixel differences with the largest preset target are extracted, and the image corresponding to the larger difference is identified as the target frame. A more effective analysis basis can be obtained by screening the target frame, so that the visibility analysis method provided in the embodiment of the present application is performed on more reliable image information, which can improve the accuracy of visibility level analysis.

[0126] It can be seen from the above embodiments that the visibility analysis method provided in the embodiments of the present application does not require any contact detection equipment, does not require manual intervention, and has the advantages of automation and high efficiency; in order to ensure the accuracy of visibility level analysis, a corresponding correction method is also provided to improve accuracy; on the other hand, the visibility analysis method provided in the embodiments of the present application can complete the detection and level classification of heavy fog visibility in a relatively short time, is real-time, and is of great significance to the fields of road traffic, air transportation, etc.

[0127] Furthermore, an application process of the visibility analysis method provided in an embodiment of the present application is provided, corresponding to the use of the visibility level classification model constructed above.

[0128] Taking the weather conditions of a southern city as an example, data was acquired using the collected historical videos, and 2,400 samples of different visibility types were collected, including: 400 each of visibility level 1 data, visibility level 2 data, visibility level 3 data, visibility level 4 data, visibility level 5 data, and visibility level 6 data.

[0129] During preprocessing, the images of different visibility types are resized to 800*800.

[0130] The preprocessed visibility image dataset is divided into training set and test set in a ratio of 4:1.

[0131] When performing visibility recognition, the training data set is input into the recognition model based on the DenseNet network to obtain the trained visibility recognition model, and the test set is input into the trained visibility recognition model to obtain the recognition effect test result of the visibility recognition model.

[0132] When performing fog visibility recognition, visibility level 1 data, visibility level 2 data, visibility level 3 data, visibility level 4 data, visibility level 5 data, and visibility level 6 are divided into six categories, and their training data sets are input into the DenseNet convolutional neural network to obtain specific categories of visibility recognition. Among them, the DenseNet convolutional neural network consists of 1 convolution layer, 3 Dense Blocks, 2 Transition Blocks, 1 Classification Block layer, and a softmax layer. Each Dense Block module consists of multiple BN+Relu+Conv(1*1)+dropout+BN+Relu+Conv(3*3)+dropout structures; each Transition Block module consists of a BN+Relu+Conv(1*1)(filternum:θ)+dropout+Pooling(2*2) structure; the Classification Block module consists of a pooling layer and a linear layer. After the image is input, it first passes through 1 convolution layer, 1 Dense Block and Transition Block, then through the second Dense Block and Transition Block, and then through the third Dense Block, followed by ClassificationBlock, and finally the visibility recognition result is obtained through the softmax activation function.

[0133] The evaluation model should be based on the model accuracy A_CC, recall rate R, precision rate P, F value, recognition time T and confusion matrix. Among them, the model accuracy reflects the ratio of the correct results of the model recognition prediction to the total number of recognized images. The higher the correct result, the higher the recognition accuracy. The calculation formula is as follows: Among them, Acc is the model accuracy, TP (True positive) represents the positive samples predicted by the model as positive, TN (Truenegative) represents the negative samples predicted by the model as negative, FP (False negative) represents the positive samples predicted by the model as negative, and FN (False negative) represents the negative samples predicted by the model as positive. The confusion matrix is ​​shown in Table 1.

[0134] sample Predicted as positive sample Predicted as negative sample The actual positive sample TP FP Actually negative sample FN TN

[0135] Table 1

[0136] Use the DenseNet convolutional neural network for fog visibility recognition and train 50 rounds; see Fig.10 , Fig.10 A schematic diagram of a confusion matrix of a training set provided in an embodiment of the present application; Fig.10 It can be seen that the accuracy of the training set reaches 100%. Fig.11 , Fig.11 A schematic diagram of a test set confusion matrix provided in an embodiment of the present application; Fig.11 In the confusion matrix, the recognition accuracy of the first and fourth level visibility is 100%, and the recognition accuracy of the other visibility levels is also above 95%. After calculation, using the visibility analysis model provided in the embodiment of the present application, the comprehensive recognition accuracy of fog visibility can reach 97.5%, the comprehensive recall rate is 97.58%, and the F1 value also reaches 97.59%, which can meet the actual application needs of engineering.

[0137] It can be concluded from the above results that the visibility analysis method provided in the embodiment of the present application has good recognition performance. The classification of fog visibility levels based on visual analysis is realized, and the measurement results of traditional meteorological instruments are compared and corrected, and the computer vision algorithm is continuously trained and optimized to improve the correctness and accuracy of the algorithm, so as to obtain more reliable and practical visibility level classification results.

[0138] Please see Fig.12 , Fig.12A module schematic diagram of a visibility analysis system provided in an embodiment of the present application; A second aspect of the present application provides a visibility analysis system, the visibility analysis system 100 comprising: an image analysis module 110, an image visibility feature extraction module 120 and a visibility analysis module 130.

[0139] The image analysis module 110 is used to extract a moving target in a target image based on target images of two adjacent frames; the image analysis module is also used to extract a target area image from the target image based on the moving target; wherein the target area image includes an image of the moving target.

[0140] The image visibility feature extraction module 120 is used to extract image visibility features from the target area image; wherein the image visibility features represent the visibility corresponding to the target image.

[0141] The visibility analysis module 130 is used to determine the target visibility level corresponding to the target image according to the image visibility characteristics.

[0142] Using the visibility analysis system provided in the embodiment of the present application, it is possible to implement any implementation method of the visibility analysis method provided in the first aspect of the present application and achieve corresponding effects, which will not be repeated here.

[0143] See also Fig.13 , Fig.13 The electronic device 300 provided in the embodiment of the present application includes: a processor 301 and a memory 302, wherein the memory 302 stores machine-readable instructions executable by the processor 301, and when the machine-readable instructions are executed by the processor 301, the above method is executed.

[0144] Based on the same inventive concept, an embodiment of the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer program instructions, and when the computer program instructions are read and executed by a processor, the steps in any of the above implementation methods are executed.

[0145] The computer-readable storage medium may be a random access memory (RAM), a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or any other medium that can store program codes. The storage medium is used to store the program, and the processor executes the program after receiving the execution instruction. The method executed by the electronic terminal of the process definition disclosed in any embodiment of the present invention can be applied to the processor or implemented by the processor.

[0146] In the embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interfaces, and the indirect coupling or communication connection of the devices or units can be electrical, mechanical or other forms.

[0147] In addition, the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0148] Furthermore, the functional modules in the various embodiments of the present application may be integrated together to form an independent part, or each module may exist separately, or two or more modules may be integrated to form an independent part.

[0149] Alternatively, the present invention may be implemented in whole or in part by software, hardware, firmware, or any combination thereof. When implemented by software, the present invention may be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present invention is generated in whole or in part.

[0150] The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means.

[0151] In this article, relational terms such as first and second, etc. are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the statement "include..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0152] The above description is only an embodiment of the present application and is not intended to limit the protection scope of the present application. For those skilled in the art, the present application may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A visibility analysis method, characterized in that: The method comprises: According to the target images of two adjacent frames, the moving target in the target image is extracted; Based on the moving target, extracting a target area image from the target image; wherein the target area image includes an image of the moving target; Extracting image visibility features from the target area image; wherein the image visibility features represent the visibility corresponding to the target image; According to the image visibility feature, a target visibility level corresponding to the target image is determined.

2. The method according to claim 1, characterized in that After determining the visibility level corresponding to the target image according to the image visibility feature, the method further includes: Comparing the target visibility level with the instrument visibility level to obtain a visibility comparison result; Determining whether the visibility level is within a visibility error range; If it is determined that the visibility level is within the visibility error range, the target visibility level is taken as the final visibility level of the target image.

3. The method according to claim 2, characterized in that If it is determined that the visibility level is not within the visibility error range, the target visibility level of the target image is recalculated based on the instrument visibility data and the known visibility image.

4. The method according to claim 1, characterized in that: Wherein, the image visibility feature includes a natural feature, and the natural feature includes a brightness feature; Extracting the image visibility feature from the target area image includes: Performing color space conversion on the target area image to obtain an HSV image; The image brightness of the HSV image is extracted, and the brightness feature is obtained based on the image brightness.

5. The method according to claim 4, characterized in that in, The natural characteristics also include: color characteristics; Extracting the image visibility feature from the target area image includes: According to the target area image, obtaining pixel distribution of the target area image; According to the pixel distribution, a pixel mean of each channel is calculated, and the color feature is obtained based on the pixel mean.

6. The method according to claim 4, characterized in that The natural features also include texture features; Extracting the image visibility feature from the target area image includes: According to the target area image, obtaining a pixel matrix of the target area image; The pixel matrix is ​​converted into a gray level co-occurrence matrix, and the texture feature is obtained based on the gray level co-occurrence matrix.

7. The method according to claim 1, characterized in that in, The image visibility features include motion features; Extracting the image visibility feature from the target area image includes: Extracting two frames of images to be analyzed corresponding to the target area image from the target image; Determining the moving pixel regions corresponding to the moving target in the two frames of images to be analyzed respectively; The motion feature is calculated based on the motion pixel area; wherein the motion feature includes the motion direction of the motion target and / or the motion speed of the motion target.

8. The method according to claim 1, characterized in that Before extracting the moving target in the target image based on the target images of two adjacent frames, the method further includes: Divide the target video into multiple continuous frame sequences; Calculating pixel differences between consecutive frames in the sequence of consecutive frames; Determining a preset target number of pixel difference values ​​from the pixel difference values, and determining a target frame according to the pixel difference values; The target image is selected from the target frame.

9. A visibility analysis system, characterized in that: The visibility analysis system comprises: an image analysis module, an image visibility feature extraction module and a visibility analysis module; The image analysis module is used to extract the moving target in the target image based on the target images of two adjacent frames; The image analysis module is further used to extract a target area image from the target image based on the moving target; wherein the target area image includes an image of the moving target; The image visibility feature extraction module is used to extract image visibility features from the target area image; wherein the image visibility features represent the visibility corresponding to the target image; The visibility analysis module is used to determine the target visibility level corresponding to the target image according to the image visibility characteristics.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer program instructions, and when the computer program instructions are executed by a processor, the steps in the method according to any one of claims 1 to 8 are executed.

Images