A method and system for visual detection of foreign matter inside a smart tunnel, and a medium

Abstract

The present application relates to the technical field of tunnel foreign body detection, and specifically to a method, system, and medium for visually detecting foreign bodies inside a smart tunnel. The method comprises: determining the texture difference coefficient of each contour in each frame of the tunnel grayscale image based on the complexity and average distribution of the texture feature coefficients of all pixels on each contour; determining the light and shadow variation coefficient of each contour in each frame of the tunnel grayscale image based on the correlation of the histograms of all superpixel blocks within the contour of the same target between different frames of the tunnel grayscale image, and the degree of discreteness of the texture difference coefficients of the contour of the same target between different frames of the tunnel grayscale image, and detecting foreign bodies inside the tunnel in combination with a neural network model. The present application improves the accuracy of foreign body detection inside the tunnel by analyzing the texture differences between the inside and outside of the contour, as well as the degree to which the texture of the same target area between different frames is affected by light.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of tunnel foreign matter detection, in particular to a smart tunnel internal foreign matter visual detection method, system and medium. BACKGROUND

[0002] Highway tunnels, as an important part of modern transportation infrastructure, bear the task of connecting cities and road transportation, and play a key role in the long-term stable development of the national economy. The frequent appearance of foreign matters such as garbage, animals, and tilted road safety facilities inside the tunnel seriously threatens the safety of vehicles and pedestrians, and even affects the structural stability of the tunnel. Tunnel internal foreign matter detection is of great significance to maintaining urban road traffic safety.

[0003] The illumination environment inside the tunnel is complex, and some foreign matters are close to the tunnel background, which affects the identification and judgment of foreign matters. Moreover, the types of foreign matters inside the tunnel are diverse, such as animals, scattered garbage, and tilted road safety facilities. Tunnel foreign matter detection is mostly based on image recognition technology to locate and identify foreign matter targets. However, due to the diversification of the size of the foreign matter targets and the interference of light in the tunnel, the existing detection methods have false positives and false negatives, reducing the accuracy of tunnel internal foreign matter detection. SUMMARY

[0004] In a first aspect, the embodiments of the present application provide a smart tunnel internal foreign matter visual detection method, which comprises the following steps:

[0005] Obtain tunnel gray scale images at all collection times within a preset time period;

[0006] Extract all contours in each frame of tunnel gray scale image, and determine the gray scale change coefficient of each pixel point on each contour based on the change trend of the gray scale values of all pixel points in the neighborhood of each pixel point on the contour;

[0007] Divide the neighborhood of each pixel point on each contour into two sub-regions, and obtain the gray scale histogram of the two sub-regions. Fit the gray scale histogram of each sub-region to obtain a histogram curve. Determine the texture complexity coefficient of each sub-region of each pixel point on each contour based on the distribution of all peaks on each histogram curve and the discrete degree of the distance between all adjacent peaks. Combine the gray scale change coefficient to determine the texture feature coefficient of each pixel point on each contour in each frame of tunnel gray scale image;

[0008] Determine the texture difference coefficient of each contour in each frame of tunnel gray scale image based on the complexity and average distribution of the texture feature coefficients of all pixel points on each contour;

[0009] dividing each frame of the tunnel gray scale image into a plurality of super-pixel blocks, and obtaining a gray scale histogram of each super-pixel block, determining a light and shadow change coefficient of each contour in each frame of the tunnel gray scale image based on a correlation of the gray scale histograms of all super-pixel blocks within the contour of the same object between different frames of the tunnel gray scale images, and a discrete degree of a texture difference coefficient of the contour of the same object between different frames of the tunnel gray scale images;

[0010] based on the light and shadow change coefficient and each frame of the tunnel gray scale image, constructing each frame of a feature analysis image, and combining a neural network model to detect the foreign matter inside the tunnel.

[0011] Preferably, the method for determining the gray scale change coefficient of each pixel point on each contour is as follows:

[0012] In each frame of the tunnel gray scale image, all pixel points in a neighborhood of each pixel point on each contour are sorted from near to far according to the distance from the pixel point belonging to the neighborhood, and the gray scale values of all sorted pixel points are fitted to obtain a plurality of fitting curves. The mean value of the absolute values of the curvatures of all fitting points on the fitting curves is taken as the gray scale change coefficient of each pixel point on each contour in each frame of the tunnel gray scale image.

[0013] Preferably, the method for determining the texture complexity coefficient of each sub-region of each pixel point on each contour is as follows:

[0014] In each frame of the tunnel gray scale image, the average level of all peak values on the histogram curve of each sub-region of each pixel point on each contour is analyzed. The texture complexity coefficient of each sub-region of each pixel point on each contour is the result of fusing the discrete degree of the distance between all adjacent peak values on the histogram curve of each sub-region of each pixel point on each contour and the average level.

[0015] Preferably, the expression of the texture feature coefficient of each pixel point on each contour in each frame of the tunnel gray scale image is as follows: ; in the formula, represents the texture feature coefficient of pixel point k on contour j in the i-th frame of the tunnel gray scale image; represents the gray scale change coefficient of pixel point k on contour j in the i-th frame of the tunnel gray scale image; represents the difference of the texture complexity coefficients of the two sub-regions of pixel point k on contour j in the i-th frame of the tunnel gray scale image.

[0016] Preferably, the expression of the texture difference coefficient of each contour in each frame of the tunnel gray scale image is as follows: ; in the formula, represents the texture difference coefficient of contour j in the i-th frame of the tunnel gray scale image; represents the fractal dimension of contour j in the i-th frame of the tunnel gray scale image; The mean value of the texture feature coefficient of all pixel points on the contour j in the i-th frame tunnel gray image.

[0017] Preferably, the method for determining the light and shadow change coefficient of each contour in each frame tunnel gray image is as follows:

[0018] The contours of the same target in different frames of tunnel gray images are assigned the same serial number, and the expression of the light and shadow change coefficient of the contour j in the i-th frame tunnel gray image is as follows: ; in the formula, The dispersion degree of the texture difference coefficient of all contours j in the i-th frame tunnel gray image and the previous continuous preset number of frames of tunnel gray images is represented by The Pearson correlation coefficient of the histogram of all super-pixel blocks of the contour j in the i-th frame tunnel gray image and the histogram of all super-pixel blocks of the contour j in the n-th frame tunnel gray image is represented by The previous preset number of continuous frames of tunnel gray images of the i-th frame tunnel gray image is represented by The preset constant greater than 0 is represented by

[0019] Preferably, the construction process of each frame feature analysis image is as follows:

[0020] The light and shadow change coefficient of each contour in each frame tunnel gray image is taken as the pixel value of all pixel points on the corresponding contour, and the pixel values of the remaining pixel points are set to 0, thereby obtaining each frame feature analysis image.

[0021] Preferably, the detection of the foreign matter inside the tunnel comprises:

[0022] Each frame tunnel gray image is taken as the input of the neural network, and each frame feature extraction image is outputted, and each frame feature analysis image and the feature extraction image are taken as the input of the neural network, and the foreign matter detection result is outputted.

[0023] In a second aspect, an embodiment of the present application provides a visual detection system for foreign matter inside a smart tunnel, and the system comprises:

[0024] A tunnel data acquisition module is configured to acquire tunnel gray images at all acquisition time points within a preset time period.

[0025] A tunnel foreign matter analysis module is configured to extract all contours in each frame tunnel gray image, and determine the gray change coefficient of each pixel point on each contour based on the variation trend of the gray values of all pixel points in the neighborhood of each pixel point.

[0026] ​The neighborhood of each pixel point on the contour is divided into two sub-regions, and the gray level histogram of the two sub-regions is obtained, the gray level histogram of each sub-region is fitted to obtain a histogram curve, the texture complexity coefficient of each sub-region of each pixel point on each contour is determined based on the distribution of all peak values on each histogram curve and the discrete degree of the distance between all adjacent peak values, and the texture feature coefficient of each pixel point on each contour in each frame of tunnel gray scale image is determined in combination with the gray level change coefficient;

[0027] The texture difference coefficient of each contour in each frame of tunnel gray scale image is determined based on the complexity degree and average distribution of the texture feature coefficients of all pixel points on each contour.

[0028] Each frame of tunnel gray scale image is divided into a plurality of super-pixel blocks, and the gray level histogram of each super-pixel block is obtained, each contour corresponds to a target, the light and shadow change coefficient of each contour in each frame of tunnel gray scale image is determined based on the correlation of the gray level histograms of all super-pixel blocks in the contour of the same target between different frames of tunnel gray scale image, and the discrete degree of the texture difference coefficient of the contour of the same target between different frames of tunnel gray scale image.

[0029] The tunnel foreign matter detection module is configured to construct a feature analysis graph based on the light and shadow change coefficient and each frame of tunnel gray scale image, and detect foreign matters in the tunnel in combination with a neural network model.

[0030] In a third aspect, an intelligent tunnel internal foreign matter visual detection medium is also provided, which includes a memory, a processor, and a computer program stored in the memory and running on the processor, and the processor implements the steps of the intelligent tunnel internal foreign matter visual detection method when executing the computer program.

[0031] As can be seen from the above embodiments, the intelligent tunnel internal foreign matter visual detection method provided by the embodiments has at least the following beneficial effects:

[0032] The application can distinguish foreign matter from tunnel ground under the influence of light and exclude the interference of tunnel ground on the foreign matter identification process by extracting all contours in each frame of tunnel grayscale image and analyzing the change trend of the grayscale values of all pixel points in the neighborhood of each pixel point on each contour to determine the grayscale change coefficient of each pixel point on each contour; the application can more accurately determine the region where foreign matter is located by analyzing the texture difference near the contour based on the complexity and average distribution of the texture feature coefficients of all pixel points on each contour to determine the texture difference coefficient of each contour in each frame of tunnel grayscale image; the application can enhance the distinguishability of foreign matter and non-foreign matter and improve the accuracy of foreign matter detection in the tunnel by determining the light and shadow change coefficient of each contour in each frame of tunnel grayscale image based on the correlation of the grayscale histograms of all super-pixel blocks in the contour of the same target between different frames of tunnel grayscale image and the discrete degree of the texture difference coefficients of the contour of the same target between different frames of tunnel grayscale image and combining the neural network model to detect foreign matter in the tunnel; the application can reduce the interference of light on the foreign matter identification process and improve the accuracy of foreign matter detection in the tunnel by analyzing the difference in texture between foreign matter and non-foreign matter. BRIEF DESCRIPTION OF DRAWINGS

[0033] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, and the advantages thereof, a brief introduction will be given to the drawings needed in the embodiments or prior art description. Obviously, the drawings in the following description only show some embodiments of the present application, and for those skilled in the art, other drawings can be obtained without creative effort based on these drawings.

[0034] Figure 1 A step flow chart of a visual detection method for foreign matter in a smart tunnel according to an embodiment of the present application is shown in the following figure:

[0035] Figure 2 A schematic diagram of tunnel contour extraction according to an embodiment of the present application is shown in the following figure:

[0036] Figure 3 A schematic diagram of grayscale change coefficient extraction process according to an embodiment of the present application is shown in the following figure:

[0037] Figure 4 A schematic diagram of texture complexity coefficient extraction process according to an embodiment of the present application is shown in the following figure:

[0038] Figure 5 A schematic diagram of light and shadow change coefficient extraction process according to an embodiment of the present application is shown in the following figure:

[0039] Figure 6 A block diagram of a foreign matter visual detection system inside a smart tunnel is provided for an embodiment of the present application;

[0040] Figure 7 A structural schematic diagram of a computer device is provided for an embodiment of the present application. DETAILED DESCRIPTION

[0041] In order to further illustrate the technical means and effects adopted by the present application to achieve the predetermined purpose, the following describes in detail the specific implementation, structure, features and effects of the method, system and medium for visual detection of foreign matter inside a smart tunnel according to the present application, in combination with the accompanying drawings and preferred embodiments. In the following description, different "one embodiment" or "another embodiment" do not necessarily refer to the same embodiment. In addition, the specific features, structures or characteristics in one or more embodiments can be combined in any suitable form.

[0042] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the present application belongs.

[0043] The specific scheme of the method, system and medium for visual detection of foreign matter inside a smart tunnel provided by the present application is described below in combination with the accompanying drawings.

[0044] Please refer to Figure 1 which shows a step flowchart of a method for visual detection of foreign matter inside a smart tunnel provided by an embodiment of the present application. The method includes the following steps:

[0045] Step S1: Obtain tunnel gray scale images at all collection time points within a preset time period.

[0046] A CMOS high-definition camera is installed at every interval inside the tunnel to capture video data of the tunnel road surface. The specific installation position interval of the camera can be determined according to the actual situation of the tunnel. The number of frames of the camera is set to f, video data within a preset time period t of any camera is obtained, tunnel images at all collection time points within the preset time period t are collected, the sampling interval is set to T, and all tunnel images are subjected to noise reduction processing. Finally, the preprocessed tunnel images are converted into gray scale images, denoted as tunnel gray scale images.

[0047] It should be noted that the values of the number of frames f, the preset time period t and the sampling interval T are all artificially set. In the present embodiment, the value of the number of frames f is 30 FPS, the value of the preset time period is 20 s, and the value of the sampling interval T is 1 s. The implementer can also set them according to the specific situation, and the present embodiment does not have special limitations.

[0048] It should be understood that there are many common image denoising algorithms, and the wavelet denoising algorithm is used in the embodiment to remove noise in the tunnel image. The implementer can also use other image denoising algorithms such as median filtering and Gaussian filtering to remove noise in the image. The selection of the image denoising algorithm is not particularly limited in the embodiment.

[0049] The wavelet denoising algorithm is a known technology, and the specific process of removing noise in the image will not be repeated.

[0050] Step S2: Based on the change trend of the gray values of all the pixel points in the neighborhood of each pixel point on each contour, the gray change coefficient of each pixel point on each contour is determined; the neighborhood of each pixel point on each contour is divided into two sub-regions, and the gray histogram of the two sub-regions is obtained, the gray histogram curve of each sub-region is fitted, based on the distribution of all peaks on each histogram curve and the discrete degree of the distance between all adjacent peaks, the texture complexity coefficient of each sub-region of each pixel point on each contour is determined, and the texture feature coefficient of each pixel point on each contour in each frame of tunnel gray image is determined in combination with the gray change coefficient.

[0051] The common foreign objects inside the tunnel can be divided into the following categories: animal class mainly including cats, dogs and the like, scattered garbage class such as fragments, bag-shaped objects and the like, and road safety facilities such as tilted crash barrels, guardrails and the like. These common foreign objects are usually irregular-shaped objects in the image, and their colors and textures are mostly different from the tunnel pavement. The tunnel pavement markings, such as lane boundary lines, guide arrows and the like, and the vehicles running in and out, have relatively regular contours and shapes.

[0052] Therefore, in order to identify the possible foreign objects in the tunnel, a contour extraction algorithm is used to extract all the contours in each frame of tunnel gray image. Each contour is connected and corresponds to a target region, and each object has only one contour.

[0053] It should be noted that there are many common contour extraction algorithms, and the findContours function in OpenCV is used in the embodiment to extract all the contours in each frame of tunnel gray image. The implementer can also use other contour extraction algorithms such as boundary tracking method or region growing method. The selection of the contour extraction algorithm is not particularly limited in the embodiment.

[0054] The findContours function in OpenCV is a known technology, and the specific process of extracting contours in the image will not be repeated. Preferably, the tunnel contour extraction schematic diagram provided in the embodiment is as shown in Figure 2 .

[0055] Further, since the color of the foreign matter is greatly different from the color of the tunnel surface, the texture feature coefficient of each pixel on each contour in each frame of the tunnel gray scale graph is determined by analyzing the color distribution difference and the texture distribution difference of the contour and the pixels around the contour, based on the trend of the gray scale values of all the pixels in the neighborhood of each pixel on each contour, and the difference between the gray scale values of the pixels in the left and right neighborhoods of each pixel on each contour, to identify the possible foreign matter in the tunnel, specifically:

[0056] S201: Determine the gray scale change coefficient of each pixel on each contour in each frame of the tunnel gray scale graph based on the trend of the gray scale values of all the pixels in the neighborhood of each pixel on each contour.

[0057] In each frame of the tunnel gray scale graph, the neighborhood W is divided with each pixel on each contour as the center, all the pixels in the neighborhood of each pixel on each contour are sorted according to the distance from the pixel in the neighborhood, and the gray scale values of all the sorted pixels are fitted to obtain the fitting curve of each pixel on each contour. The mean of the absolute values of the curvatures of all the fitting points on each fitting curve is taken as the gray scale change coefficient of each pixel on each contour in each frame of the tunnel gray scale graph.

[0058] It should be noted that the value of the neighborhood W is artificially set, and in the embodiment, the value of the neighborhood W is 224, and the implementer can also set it according to the specific situation, which is not specially limited in the embodiment.

[0059] In addition, it should be understood that there are many commonly used curve fitting algorithms, and in the embodiment, the least square method is used to obtain the fitting curve, and the implementer can also use other curve fitting methods such as polynomial fitting method, and the selection of the curve fitting method is not specially limited in the embodiment.

[0060] Among them, the least square method is a known technology, and the specific process of fitting data to obtain the fitting curve will not be repeated.

[0061] Preferably, the gray scale change coefficient extraction process provided in the embodiment is as shown in Figure 3 .

[0062] S202: Divide the neighborhood of each pixel on each contour into two sub-regions, obtain the gray scale histogram of the two sub-regions, fit the gray scale histogram of each sub-region to obtain the histogram curve, determine the texture complexity coefficient of each sub-region of each pixel on each contour based on the distribution of all the peaks on each histogram curve and the discrete degree of the distance between all the adjacent peaks, and determine the texture feature coefficient of each pixel on each contour in each frame of the tunnel gray scale graph in combination with the gray scale change coefficient.

[0063] The greater the gray scale variation coefficient, the greater the bending degree of the contour, which means that the shape of the contour is more complex, and the contour edge can have more unevenness. The unevenness can cause the texture features on both sides of the contour to be more abundant and varied, and the target region is more likely to belong to the foreign matter. Generally, the contour near the region where the foreign matter is located has complex and diverse textures, while the textures near the contour outside the region where the foreign matter is located are relatively simple. Therefore, the texture features on both sides of the contour are further extracted, the distribution of the gray scale values of the pixel points on both sides of the contour and the distribution difference are analyzed, and the region where the foreign matter is located is identified. Specifically:

[0064] (1) In each frame of the tunnel gray scale image, the neighborhood of each pixel point on each contour is divided into two sub-regions by the continuous contour, the gray scale histograms of all pixel points in the two sub-regions are calculated respectively, and the two gray scale histograms are fitted respectively to obtain two histogram curves.

[0065] It should be noted that the process of obtaining the gray scale histogram is a known technology, and the specific obtaining process will not be described again.

[0066] In particular, when the neighborhood of the pixel point is located at the edge of the contour, the missing pixel is filled by using a boundary filling algorithm.

[0067] It should be noted that there are many commonly used boundary filling algorithms. In the present embodiment, the missing pixel is filled by using a mirror filling algorithm. The implementer can also use other boundary filling algorithms such as constant filling. The selection of the boundary filling algorithm is not specially limited in the present embodiment.

[0068] It should be noted that the mirror filling method is a known technology, and the specific filling process of the missing pixel will not be described again.

[0069] (2) If the texture of a sub-region on one side of the contour is complex, the distribution of the gray scale values in the gray scale range is relatively uniform, there are multiple peak values in the histogram curve, and the distances between the peak values are relatively balanced. The gray scale values of the sub-window with relatively simple texture are concentrated in a certain interval, the number of peak values of the histogram curve is small, and the amplitude is high.

[0070] Therefore, by analyzing the distribution of the gray scale values of the pixel points in the sub-region on either side of the contour, the texture complexity coefficient of each sub-region of each pixel point on each contour in each frame of the tunnel gray scale image is determined, so as to distinguish the region where the foreign matter is located and the region where the foreign matter is not located in the tunnel. Specifically:

[0071] In each frame of the tunnel gray scale image, the average level of all peak values on the histogram curve of each sub-region of each pixel point on each contour is analyzed, and the texture complexity coefficient of each sub-region of each pixel point on each contour is the result of the dispersion degree of the distances between all adjacent peak values on the histogram curve of each sub-region of each pixel point on each contour and the average level.

[0072] It should be noted that there are many ways to measure the average level of a group of data. In the present embodiment, the average level of all peaks on the histogram curve is measured by calculating the average value of all peaks on the histogram curve. Other methods such as geometric mean can also be used to measure the average level of data. The present embodiment does not make special restrictions on the selection of the method for measuring the average level of data.

[0073] It should be noted that there are many ways to measure the dispersion of a group of data. In the present embodiment, the dispersion of the distance between all adjacent peaks on the histogram curve is measured by calculating the standard deviation of the distance between all adjacent peaks on the histogram curve. Other methods such as variance or dispersion coefficient can also be used to measure the dispersion of data. The present embodiment does not make special restrictions on the selection of the method for measuring the dispersion of data.

[0074] It should be understood that fusion refers to the result of combining two or more indicators by positive fusion, i.e., combining two or more indicators by addition or multiplication, etc., in order to obtain a comprehensive indicator, so as to more comprehensively and accurately evaluate a phenomenon or problem. This fusion method is not limited to simple arithmetic operations, but can also include more complex statistical models and analysis methods, which can be selected by the implementer according to the specific circumstances, and the present embodiment does not make special restrictions.

[0075] Preferably, the texture complexity coefficient of each sub-region of each pixel point on each contour in the present embodiment is the product of the dispersion and the average level of the distance between all adjacent peaks on the histogram curve of each sub-region of each pixel point on each contour. In actual application, as another implementation, the texture complexity coefficient of each sub-region of each pixel point on each contour is the exponential function value with the sum of the dispersion and the average level of the distance between all adjacent peaks on the histogram curve of each sub-region of each pixel point on each contour as the independent variable.

[0076] According to the texture complexity coefficient of each sub-region of each pixel point on each contour, if the sub-region is a foreign object region, the more complex the texture in the sub-region is, the more peaks on the histogram curve of the sub-region are, and the more uneven the distribution of these peaks is, i.e., the average level of all peaks is greater, the dispersion between adjacent peaks is greater, and the texture complexity coefficient is greater. Conversely, if the sub-region is not a foreign object region, the simpler the texture in the sub-region is, the fewer peaks on the histogram curve of the sub-region are, and the more concentrated the distribution of these peaks is, i.e., the average level of all peaks is smaller, the dispersion between adjacent peaks is smaller, and the texture complexity coefficient is smaller.

[0077] (3) Further, based on the grayscale variation coefficient and the texture complexity coefficient, the texture feature coefficient of each pixel point on each contour in each frame of the tunnel grayscale image is determined, specifically:

[0078] Texture feature coefficient of pixel k on contour j in the i-th frame tunnel grayscale image The expression is: Where, Represents the grayscale variation coefficient of pixel k on contour j in the i-th frame tunnel grayscale image; Represents the difference in texture complexity coefficients between the two sub-regions of pixel k on contour j in the i-th frame tunnel grayscale image.

[0079] It should be noted that there are many methods to measure the differences between data. In this embodiment, the difference between the texture complexity coefficients of the sub-regions is measured by calculating the absolute value of the difference between the texture complexity coefficients of the two sub-regions. The implementer may also use other methods to measure the difference, such as ratio. This embodiment does not impose any special restrictions on the selection of methods to measure the differences between data.

[0080] Furthermore, according to the texture feature coefficient of each pixel on each contour in each frame of the tunnel grayscale image, it can be understood that the larger the grayscale variation coefficient of the pixel on the contour, the greater the contour curvature near the corresponding pixel, which means that the shape of the contour where the pixel is located is more complex and more likely to be the edge of a foreign object; the larger the difference in texture complexity coefficients between the two sub-regions of the pixel on the contour, the greater the texture difference on both sides of the contour, and the more likely it is to be the region where the foreign object is located; therefore, if the grayscale variation coefficient of the pixel on the contour is larger, the difference in texture complexity coefficients between the two sub-regions is larger, and the texture difference coefficient of the pixel on the contour is larger, then the corresponding contour is more likely to be part of a foreign object; conversely, the smaller the grayscale variation coefficient of the pixel on the contour, the smaller the contour curvature near the corresponding pixel, which means that the shape of the contour where the pixel is located is simpler and less likely to be the edge of a foreign object; the smaller the difference in texture complexity coefficients between the two sub-regions of the pixel on the contour, the smaller the texture difference on both sides of the contour, and the less likely it is to be the region where the foreign object is located; therefore, if the grayscale variation coefficient of the pixel on the contour is smaller, the difference in texture complexity coefficients between the two sub-regions is smaller, and the texture difference coefficient of the pixel on the contour is smaller, then the corresponding contour is less likely to be part of a foreign object.

[0081] Preferably, the texture complexity coefficient extraction process diagram provided in this embodiment is as follows: Figure 4 shown.

[0082] Step S3: Determine the texture difference coefficient of each contour based on the complexity and average distribution of the texture feature coefficients of all pixels on each contour.

[0083] Generally, the texture features near consecutive pixels on the contour of a foreign object vary relatively greatly. Therefore, by analyzing the changes in the texture feature coefficients of all pixels on the contour, we can identify whether there is a foreign object in the tunnel. Specifically:

[0084] (1) In each frame of the tunnel grayscale image, calculate the fractal dimension of the texture feature coefficients of all pixels on each contour.

[0085] It should be noted that there are many methods for calculating the fractal dimension of a set of data. In this embodiment, the texture feature coefficients of all pixels on each contour are used as the input of the Higuchi algorithm to obtain the fractal dimension of each contour. In actual application, as other implementation methods, the implementer may also adopt other methods such as the Hurst exponent method or the box counting method to obtain the fractal dimension. This embodiment does not impose any special restrictions on the method for obtaining the fractal dimension.

[0086] The Higuchi algorithm is a well-known technology, and the specific process of calculating the fractal dimension is not described in detail here.

[0087] (2) Further, based on the fractal dimension and the texture feature coefficient, the texture difference coefficient of each contour in each frame of the tunnel grayscale image is determined, specifically:

[0088] Texture difference coefficient of contour j in the grayscale image of the tunnel in frame i The expression is: Where, represents the fractal dimension of contour j in the grayscale image of the tunnel in frame i; Represents the mean value of the texture feature coefficients of all pixels on contour j in the i-th frame tunnel grayscale image.

[0089] Furthermore, according to the texture difference coefficient of each contour in the grayscale image of each frame of the tunnel, it can be understood that if the fractal dimension of the contour is larger, the degree of texture color change near the continuous pixel points on the corresponding contour is greater; if the texture feature coefficient of the contour is larger, the more irregular the foreign body contour is and the more obvious the color difference is; therefore, if the fractal dimension is larger and the texture feature coefficient of the contour is larger, it means that the contour is more likely to belong to the foreign body part; conversely, if the fractal dimension of the contour is smaller, the degree of texture color change near the continuous pixel points on the corresponding contour is smaller; if the texture feature coefficient of the contour is smaller, the more irregular the foreign body contour is and the less obvious the color difference is; therefore, if the fractal dimension is smaller and the texture feature coefficient of the contour is smaller, it means that the contour is less likely to belong to the foreign body part.

[0090] Step S4: Based on the correlation of the histograms of all superpixel blocks of the same target in the tunnel grayscale images of different frames and the discrete degree of the texture difference coefficient of the same target in different frames, the light and shadow variation coefficient of each contour in the tunnel grayscale images of each frame is determined.

[0091] Due to the influence of light inside the tunnel, the color and texture of the object inside the tunnel under different light may be different. Especially during the driving process, the light shining on the foreign object will cause the light and shadow and texture of the target area of the foreign object to change significantly, while the surface of the vehicle is smooth, so the light and shadow and texture of the vehicle target change is not obvious.

[0092] The light and shadow features near the contour of the same target area between different frames may be different, and therefore, the change of light and shadow between different frames is further analyzed to exclude the interference of light and shadow features on foreign object detection, specifically:

[0093] (1) The position of the same target in different frames is obtained by using an optical flow estimation algorithm, and the contours of the same target in different frames are set with the same serial number.

[0094] It should be noted that there are many commonly used optical flow estimation algorithms, and in this embodiment, the Lucas-Kanade optical flow method is used to obtain the position of the same target in different frames. In actual application, as other implementation manners, the implementer can also use the Horn-Schunck optical flow method or the Farneback optical flow method to obtain the position of the same target in different images, and the selection of the optical flow estimation algorithm is not specially limited in this embodiment.

[0095] (2) Further, since the texture of the central part of the same target area between different frames will also be affected by the change of light, in order to more accurately obtain the light and shadow features of the image inside the target area, this embodiment divides each frame of tunnel grayscale image into a plurality of superpixel blocks, further determines whether the tunnel grayscale contains foreign object features by analyzing the distribution of the grayscale values of the local area pixels inside the contour, and thus more accurately detects whether there is a foreign object in the tunnel.

[0096] It should be noted that there are many commonly used image segmentation algorithms, and in this embodiment, the SLIC superpixel segmentation algorithm is used to segment the internal area of the contour, and the implementer can also use region growing or clustering-based image segmentation algorithm to segment the image, and the selection of the image segmentation algorithm is not specially limited in this embodiment.

[0097] The SLIC superpixel segmentation algorithm is a known technology in the field of image segmentation, and the specific process of segmenting the image will not be repeated.

[0098] (3) Further, by analyzing the correlation of the histogram of all superpixel blocks inside the contour of the same target between different frames of tunnel grayscale images and the discrete degree of the texture difference coefficient of the contour of the same target between different frames of tunnel grayscale images, the light and shadow change coefficient of each contour in each frame of tunnel grayscale image is determined, specifically:

[0099] The light and shade change coefficient of the contour j in the i-th frame tunnel gray scale image The expression of the light and shade change coefficient is: ; wherein, represents the dispersion degree of the texture difference coefficients of all contours j in the i-th frame tunnel gray scale image and the previous preset number of continuous frame tunnel gray scale images; represents the Pearson correlation coefficient of the histograms of all super-pixel blocks of the contour j in the i-th frame tunnel gray scale image and the contour j in the n-th frame tunnel gray scale image; represents the previous preset number of continuous frame tunnel gray scale images of the i-th frame tunnel gray scale image; represents a preset constant greater than 0, which is used to prevent the denominator from being 0, wherein The value of the preset number is artificially set, and in the embodiment, the preset number is 10. The value of the preset number is 0.01, and the implementer can set it by himself according to the specific situation on the premise of ensuring that the denominator is not 0 and not excessively affecting the calculation result, and the embodiment does not make special restrictions.

[0100] It should be noted that, The value of the preset number is artificially set, and in the embodiment, the preset number is 10. The value of the preset number is 20, and the implementer can also set it by himself according to the specific situation, and the embodiment does not make special restrictions. In particular, for the tunnel gray scale image without the previous preset number of continuous frame tunnel gray scale images, the corresponding tunnel gray scale image is not analyzed.

[0101] It should be understood that there are many methods for measuring the dispersion degree of a group of data, and in the embodiment, the dispersion degree of the texture difference coefficients is measured by calculating the standard deviation of the texture difference coefficients of the contour of the same target between different frame tunnel gray scale images, and the implementer can also use other methods for measuring the dispersion degree, such as variance or dispersion coefficient, and the embodiment does not make special restrictions on the selection of the method for measuring the dispersion degree of data.

[0102] In addition, it should be understood that there are many methods for calculating the correlation coefficient between two variables, and in the embodiment, the Pearson correlation coefficient is used to measure the correlation between the histograms of all super-pixel blocks in the contour of the same target between different frame tunnel gray scale images, and the implementer can also use other methods for measuring the correlation between variables, such as Kendall rank correlation coefficient or Spearman rank correlation coefficient, and the embodiment does not make special restrictions on the selection of the method for calculating the correlation coefficient between variables.

[0103] The calculation process of the Pearson correlation coefficient is a known technology, and the specific calculation process is not described again.

[0104] Further, according to the light and shadow change coefficient of each contour in each frame tunnel grayscale image, it can be understood that the greater the discrete degree of the texture difference coefficient of the contour of the same target between different frame tunnel grayscale images, the smaller the absolute value of the correlation coefficient of the histogram of all superpixel blocks in the contour of the same target between different frame tunnel grayscale images, the greater the light and shadow change degree inside the corresponding contour, that is, the greater the light and shadow change coefficient of the corresponding contour, which means that it is more likely that the light and shadow change is caused by the light shining on the foreign matter; on the contrary, the smaller the discrete degree of the texture difference coefficient of the contour of the same target between different frame tunnel grayscale images, the greater the absolute value of the correlation coefficient of the histogram of all superpixel blocks in the contour of the same target between different frame tunnel grayscale images, the smaller the light and shadow change degree inside the corresponding contour, that is, the smaller the light and shadow change coefficient of the corresponding contour, which means that it is less likely that the light and shadow change is caused by the light shining on the foreign matter.

[0105] Preferably, the light and shadow change coefficient extraction process provided by the embodiment is shown in the schematic diagram as Figure 5

[0106] Step S5: Based on the light and shadow change coefficient and each frame tunnel grayscale image, each frame feature analysis image is constructed, and a neural network model is combined to detect the foreign matter inside the tunnel.

[0107] The light and shadow change coefficient of each contour in each frame tunnel grayscale image is taken as the pixel value of all pixel points on the corresponding contour, and the pixel value of the remaining pixel points in the tunnel grayscale image is set to 0, to obtain each frame feature analysis image.

[0108] Further, each frame tunnel grayscale image is taken as the input of the YOLOv7 model, the loss function is set as the SloU loss function, the BiFPN pyramid structure is adopted, and each frame feature extraction image extracted by the backbone network in the YOLOv7 model is output.

[0109] Each frame feature extraction image and feature analysis image are taken as the input of the neck network in the YOLOv7 model, different scale feature information is obtained, the head network in the YOLOv7 model is combined to fuse and detect the feature information under different scales, and finally the foreign matter detection result is output.

[0110] Among them, the YOLOv7 model, the SloU loss function and the BiFPN pyramid structure are all known technologies, and the specific principles will not be repeated.

[0111] At this point, by analyzing the texture difference of the inside and outside of the contour and the influence degree of the texture of the same target region between different frames, the embodiment weakens the interference of light and foreign matter scale diversification on foreign matter detection, avoids the defects of false detection and missed detection in the foreign matter detection process, and improves the precision of foreign matter detection inside the tunnel.

[0112] ​Based on the same inventive concept as the above method, the embodiment of the present application also provides a foreign matter visual detection system in a smart tunnel, comprising:

[0113] A tunnel data acquisition module is configured to acquire tunnel gray scale images at all acquisition time points within a preset time length.

[0114] A tunnel foreign matter analysis module is configured to extract all contours in each frame of tunnel gray scale image, determine a gray scale change coefficient of each pixel point on each contour based on a change trend of gray scale values of all pixel points in a neighborhood of each pixel point on each contour.

[0115] The neighborhood of each pixel point on each contour is divided into two sub-regions, and gray scale histograms of the two sub-regions are acquired, a gray scale histogram curve is obtained by fitting the gray scale histogram of each sub-region, a texture complexity coefficient of each sub-region of each pixel point on each contour is determined based on a distribution of all peaks on each gray scale histogram curve and a discrete degree of distances between all adjacent peaks, and a texture feature coefficient of each pixel point on each contour in each frame of tunnel gray scale image is determined in combination with the gray scale change coefficient.

[0116] A texture difference coefficient of each contour in each frame of tunnel gray scale image is determined based on a complexity degree and an average distribution of the texture feature coefficients of all pixel points on each contour.

[0117] Each frame of tunnel gray scale image is divided into a plurality of super-pixel blocks, and gray scale histograms of the super-pixel blocks are acquired, each contour corresponds to a target, a light and shadow change coefficient of each contour in each frame of tunnel gray scale image is determined based on a correlation of the gray scale histograms of all super-pixel blocks in the contour of the same target between different frames of tunnel gray scale image and a discrete degree of the texture difference coefficients of the contours of the same target between different frames of tunnel gray scale image.

[0118] A tunnel foreign matter detection module is configured to construct each frame of feature analysis image based on the light and shadow change coefficient and each frame of tunnel gray scale image, and detect foreign matters in the tunnel in combination with a neural network model.

[0119] The embodiment of the present application provides a block diagram of a foreign matter visual detection system in a smart tunnel, as shown in Figure 6 .

[0120] Based on the same inventive concept as the above method, the embodiment of the present application also provides a foreign matter visual detection medium in a smart tunnel, comprising a memory, a processor and a computer program stored in the memory and running on the processor, and the processor implements the steps of any one of the methods in the above-mentioned foreign matter visual detection method in the smart tunnel when executing the computer program.

[0121] Figure 7A structural schematic diagram of a computer device is provided for an embodiment of the present application. As shown in the figure, the computer device of the embodiment includes at least one processor (only one processor is shown in the figure), a memory, and a computer program stored in the memory and executable on the at least one processor, and the processor implements the steps of any one of the methods in the above-mentioned method for visual detection of foreign matter inside a smart tunnel when executing the computer program. It should be noted that the above-mentioned sequence of the embodiments is only for description, and does not represent the advantages and disadvantages of the embodiments. Moreover, the above-mentioned description is for a specific embodiment of the present application. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multi-task processing and parallel processing are possible or can be advantageous. Figure 7 Figure 7 The above-mentioned sequence of the embodiments is only for description, and does not represent the advantages and disadvantages of the embodiments. Moreover, the above-mentioned description is for a specific embodiment of the present application. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multi-task processing and parallel processing are possible or can be advantageous.

[0122] Each of the embodiments in the present specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other. Each embodiment focuses on the difference from other embodiments.

[0123] The above-mentioned sequence of the embodiments is only for description, and does not represent the advantages and disadvantages of the embodiments. Moreover, the above-mentioned description is for a specific embodiment of the present application. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multi-task processing and parallel processing are possible or can be advantageous.​

Claims

1. A method for visually detecting foreign objects inside a smart tunnel, characterized in that: The method comprises the following steps: Obtain the tunnel grayscale image at all acquisition moments within a preset time period; Extract all contours in each frame of the tunnel grayscale image, and determine the grayscale variation coefficient of each pixel on each contour based on the grayscale value variation trend of all pixels in the neighborhood of each pixel on each contour; The neighborhood of each pixel on each contour is divided into two sub-regions, and the grayscale histograms of the two sub-regions are obtained. The grayscale histogram of each sub-region is fitted to obtain a histogram curve. Based on the distribution of all peaks on each histogram curve and the degree of dispersion of the distances between all adjacent peaks, the texture complexity coefficient of each sub-region of each pixel on each contour is determined. Combined with the grayscale variation coefficient, the texture feature coefficient of each pixel on each contour in each frame of the tunnel grayscale image is determined; Based on the complexity and average distribution of the texture feature coefficients of all pixels on each contour, the texture difference coefficient of each contour in each frame of the tunnel grayscale image is determined; Each tunnel grayscale image is divided into multiple superpixel blocks, and the grayscale histogram of each superpixel block is obtained. Each contour corresponds to a target. Based on the correlation of the grayscale histograms of all superpixel blocks within the contour of the same target between different tunnel grayscale images, and the degree of discreteness of the texture difference coefficient of the contour of the same target between different tunnel grayscale images, the light and shadow variation coefficient of each contour in each tunnel grayscale image is determined; Based on the light and shadow variation coefficient and the grayscale image of each frame of the tunnel, a feature analysis graph of each frame is constructed, and foreign objects inside the tunnel are detected in combination with a neural network model; The expression of the texture difference coefficient of each contour in the tunnel grayscale image of each frame is: Where, represents the texture difference coefficient of contour j in the grayscale image of the tunnel in frame i; represents the fractal dimension of contour j in the grayscale image of the tunnel in frame i; represents the mean value of the texture feature coefficients of all pixels on the contour j in the i-th frame tunnel grayscale image; The method for determining the light and shadow variation coefficient of each contour in each frame of the tunnel grayscale image is as follows: The contours of the same target in different frames of tunnel grayscale images are set to the same serial number, and the light and shadow change coefficient of contour j in the i-th frame of tunnel grayscale image is The expression is: Where, Indicates the degree of dispersion of the texture difference coefficients of all contours j in the tunnel grayscale image of the preceding preset number of frames; N represents the Pearson correlation coefficient between the histogram of all superpixel blocks of contour j in the i-th frame tunnel grayscale image and the histogram of all superpixel blocks of contour j in the n-th frame tunnel grayscale image; i represents a preset number of consecutive frames of tunnel grayscale images before the i-th frame of tunnel grayscale image; ε represents a preset constant greater than 0.

2. The method for visually detecting foreign objects inside a smart tunnel according to claim 1, wherein: The method for determining the grayscale variation coefficient of each pixel point on each contour is: In each frame of the tunnel grayscale image, all pixels in the neighborhood of each pixel point on each contour are sorted from near to far according to their distance from the pixel point in the neighborhood, and the grayscale values ​​of all sorted pixels are fitted to obtain fitting curves. The average of the absolute values ​​of the curvatures of all fitting points on each fitting curve is used as the grayscale variation coefficient of each pixel point on each contour in each frame of the tunnel grayscale image.

3. The method for visually detecting foreign objects in a smart tunnel according to claim 1, wherein: The method for determining the texture complexity coefficient of each sub-region of each pixel point on each contour is as follows: In each frame of the tunnel grayscale image, the average level of all peaks on the histogram curve of each sub-region of each pixel point on each contour is analyzed. The texture complexity coefficient of each sub-region of each pixel point on each contour is the discrete degree of the distance between all adjacent peaks on the histogram curve of each sub-region of each pixel point on each contour, and the result of fusion with the average level.

4. The method for visually detecting foreign objects in a smart tunnel according to claim 1, wherein: The expression of the texture feature coefficient of each pixel point on each contour in each frame of the tunnel grayscale image is: Where, Represents the texture feature coefficient of pixel k on contour j in the grayscale image of the i-th frame tunnel; Represents the grayscale variation coefficient of pixel k on contour j in the i-th frame tunnel grayscale image; Represents the difference in texture complexity coefficients between the two sub-regions of pixel k on contour j in the i-th frame tunnel grayscale image.

5. The method for visually detecting foreign objects inside a smart tunnel according to claim 1, wherein: The construction process of each frame feature analysis graph is as follows: The light and shadow variation coefficient of each contour in the tunnel grayscale image of each frame is used as the pixel value of all pixels on the corresponding contour, and the pixel values ​​of the remaining pixels are set to 0 to obtain the feature analysis graph of each frame.

6. The method for visually detecting foreign objects in a smart tunnel according to claim 1, wherein: The detecting of foreign matter inside the tunnel includes: Each frame of the tunnel grayscale image is used as the input of the neural network, and the feature extraction image of each frame is output. The feature analysis image and feature extraction image of each frame are used as the input of the neural network, and the foreign body detection result is output.

7. A foreign body visual detection system inside a smart tunnel, implementing the foreign body visual detection method inside a smart tunnel as claimed in claim 1, characterized in that: The system comprises: Tunnel data acquisition module, used to obtain tunnel grayscale images at all acquisition moments within a preset time period; The tunnel foreign body analysis module is used to extract all contours in each frame of the tunnel grayscale image and determine the grayscale variation coefficient of each pixel on each contour based on the grayscale value variation trend of all pixels in the neighborhood of each pixel on each contour; The neighborhood of each pixel on each contour is divided into two sub-regions, and the grayscale histograms of the two sub-regions are obtained. The grayscale histogram of each sub-region is fitted to obtain a histogram curve. Based on the distribution of all peaks on each histogram curve and the degree of dispersion of the distances between all adjacent peaks, the texture complexity coefficient of each sub-region of each pixel on each contour is determined. Combined with the grayscale variation coefficient, the texture feature coefficient of each pixel on each contour in each frame of the tunnel grayscale image is determined; Based on the complexity and average distribution of the texture feature coefficients of all pixels on each contour, the texture difference coefficient of each contour in each frame of the tunnel grayscale image is determined; Each tunnel grayscale image is divided into multiple superpixel blocks, and the grayscale histogram of each superpixel block is obtained. Each contour corresponds to a target. Based on the correlation of the grayscale histograms of all superpixel blocks within the contour of the same target between different tunnel grayscale images, and the degree of discreteness of the texture difference coefficient of the contour of the same target between different tunnel grayscale images, the light and shadow variation coefficient of each contour in each tunnel grayscale image is determined; The tunnel foreign object detection module is used to construct a feature analysis diagram for each frame based on the light and shadow variation coefficient and the grayscale image of each frame of the tunnel, and detect foreign objects inside the tunnel in combination with a neural network model.

8. A foreign body visual detection medium inside a smart tunnel, comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that: When the processor executes the computer program, the steps of the method for visually detecting foreign objects inside a smart tunnel as described in any one of claims 1 to 6 are implemented.

Images