Visual detection method and system for foreign matters in intelligent tunnel and medium

Abstract

The invention relates to the technical field of tunnel foreign matter detection, in particular to a visual detection method and system for foreign matters in an intelligent tunnel and a medium, and the method comprises the steps: determining a texture difference coefficient of each contour in each frame of tunnel grey-scale map based on the complexity and average distribution condition of texture feature coefficients of all pixel points on each contour; determining a shadow change coefficient of each contour in each frame of tunnel grey-scale map based on the correlation of histograms of all super-pixel blocks in the contour of the same target between different frames of tunnel grey-scale maps and the dispersion degree of texture difference coefficients of the contour of the same target between different frames of tunnel grey-scale maps; and foreign matters in the tunnel are detected in combination with the neural network model. By analyzing the texture difference between the inner side and the outer side of the contour and the influence degree of the light on the texture of the same target area between different frames, the accuracy of tunnel internal foreign matter detection is improved.

Description

Technical Field

[0001] The present application relates to the technical field of tunnel foreign body detection, and specifically to a method, system and medium for visually detecting foreign bodies inside a smart tunnel. Background Art

[0002] As an important part of modern transportation infrastructure, highway tunnels carry tasks such as connecting cities and road transportation, and play a key role in the long-term stable development of the national economy. Foreign objects such as garbage, animals, and tilted road safety facilities frequently appear inside tunnels, seriously threatening the safety of vehicles and pedestrians, and even affecting the structural stability of tunnels. Foreign object detection inside tunnels is of great significance to maintaining urban road traffic safety.

[0003] The lighting environment inside the tunnel is complex. Some foreign objects are close to the tunnel background, which affects the identification and judgment of foreign objects. In addition, there are many types of foreign objects in the tunnel, such as animals, scattered garbage, and tilted road safety facilities. Most of the foreign object detection in tunnels is based on image recognition technology to locate and identify foreign objects. However, due to the diversity of foreign object targets in the tunnel and the interference of light, the existing detection methods have false detections and missed detections, which reduces the accuracy of foreign object detection in the tunnel. Summary of the invention

[0004] In a first aspect, an embodiment of the present application provides a method for visually detecting foreign objects inside a smart tunnel, the method comprising the following steps: Obtain the tunnel grayscale image at all acquisition moments within a preset time period; Extract all contours in each frame of the tunnel grayscale image, and determine the grayscale change coefficient of each pixel on each contour based on the change trend of the grayscale values ​​of all pixels in the neighborhood of each pixel on each contour; The neighborhood of each pixel point on each contour is divided into two sub-regions, and the grayscale histograms of the two sub-regions are obtained, and the grayscale histogram of each sub-region is fitted to obtain a histogram curve, and the texture complexity coefficient of each sub-region of each pixel point on each contour is determined based on the distribution of all peaks on each histogram curve and the discrete degree of the distance between all adjacent peaks, and the texture feature coefficient of each pixel point on each contour in each frame of the tunnel grayscale image is determined in combination with the grayscale variation coefficient; Based on the complexity and average distribution of the texture feature coefficients of all pixels on each contour, the texture difference coefficient of each contour in each frame of the tunnel grayscale image is determined; Divide each frame of the tunnel grayscale image into multiple superpixel blocks, and obtain the grayscale histogram of each superpixel block. Each contour corresponds to a target. Based on the correlation of the grayscale histograms of all superpixel blocks in the contour of the same target between different frame tunnel grayscale images, and the discrete degree of the texture difference coefficient of the contour of the same target between different frame tunnel grayscale images, determine the light and shadow change coefficient of each contour in each frame tunnel grayscale image; Based on the light and shadow variation coefficient and the grayscale image of each frame of the tunnel, a feature analysis image of each frame is constructed, and foreign objects inside the tunnel are detected in combination with a neural network model.

[0005] Preferably, the grayscale variation coefficient of each pixel point on each contour is determined by: In each frame of the tunnel grayscale image, all the pixels in the neighborhood of each pixel on each contour are sorted from near to far according to the distance to the pixel in the neighborhood, and the grayscale values ​​of all the sorted pixels are fitted to obtain various fitting curves. The average of the absolute values ​​of the curvatures of all the fitting points on each fitting curve is used as the grayscale variation coefficient of each pixel on each contour in each frame of the tunnel grayscale image.

[0006] Preferably, the method for determining the texture complexity coefficient of each sub-region of each pixel point on each contour is: In each frame of the tunnel grayscale image, the average level of all peaks on the histogram curve of each sub-region of each pixel point on each contour is analyzed, and the texture complexity coefficient of each sub-region of each pixel point on each contour is the discrete degree of the distance between all adjacent peaks on the histogram curve of each sub-region of each pixel point on each contour, and the result of fusion with the average level.

[0007] Preferably, the expression of the texture feature coefficient of each pixel point on each contour in each frame of the tunnel grayscale image is: ; In the formula, Represents the texture feature coefficient of pixel k on contour j in the i-th frame tunnel grayscale image; Represents the grayscale variation coefficient of pixel k on contour j in the i-th frame tunnel grayscale image; Represents the difference in texture complexity coefficients between two sub-regions of pixel k on contour j in the i-th frame tunnel grayscale image.

[0008] Preferably, the expression of the texture difference coefficient of each contour in each frame of the tunnel grayscale image is: ; In the formula, represents the texture difference coefficient of contour j in the i-th frame tunnel grayscale image; represents the fractal dimension of contour j in the i-th frame tunnel grayscale image; Represents the mean value of the texture feature coefficients of all pixels on contour j in the i-th frame tunnel grayscale image.

[0009] Preferably, the method for determining the light and shadow variation coefficient of each contour in each frame of the tunnel grayscale image is: The contours of the same target in different frame tunnel grayscale images are set to the same sequence number, and the light and shadow change coefficient of contour j in the i-th frame tunnel grayscale image is The expression is: ; In the formula, Indicates the discrete degree of the texture difference coefficients of all contours j in the tunnel grayscale image of the previous preset number of consecutive frames of the i-th frame; represents the Pearson correlation coefficient between the histogram of all superpixel blocks of contour j in the i-th frame tunnel grayscale image and the histogram of all superpixel blocks of contour j in the n-th frame tunnel grayscale image; Indicates a preset number of continuous frames of tunnel grayscale images before the i-th frame of tunnel grayscale image; Indicates a preset constant greater than 0.

[0010] Preferably, the construction process of each frame feature analysis graph is: The light and shadow variation coefficient of each contour in each frame of the tunnel grayscale image is used as the pixel value of all pixels on the corresponding contour, and the pixel values ​​of the remaining pixels are set to 0 to obtain the feature analysis graph of each frame.

[0011] Preferably, the detecting of foreign matter inside the tunnel includes: Each frame of the tunnel grayscale image is used as the input of the neural network, and each frame of the feature extraction image is output. The feature analysis image and the feature extraction image of each frame are used as the input of the neural network, and the foreign body detection result is output.

[0012] In a second aspect, an embodiment of the present application provides a foreign body visual detection system inside a smart tunnel, the system comprising: The tunnel data acquisition module is used to obtain the tunnel grayscale image at all acquisition moments within a preset time period; The tunnel foreign body analysis module is used to extract all contours in each frame of the tunnel grayscale image, and determine the grayscale change coefficient of each pixel on each contour based on the change trend of the grayscale values ​​of all pixels in the neighborhood of each pixel on each contour; The neighborhood of each pixel point on each contour is divided into two sub-regions, and the grayscale histograms of the two sub-regions are obtained, and the grayscale histogram of each sub-region is fitted to obtain a histogram curve, and the texture complexity coefficient of each sub-region of each pixel point on each contour is determined based on the distribution of all peaks on each histogram curve and the discrete degree of the distance between all adjacent peaks, and the texture feature coefficient of each pixel point on each contour in each frame of the tunnel grayscale image is determined in combination with the grayscale variation coefficient; Based on the complexity and average distribution of the texture feature coefficients of all pixels on each contour, the texture difference coefficient of each contour in each frame of the tunnel grayscale image is determined; Divide each frame of the tunnel grayscale image into multiple superpixel blocks, and obtain the grayscale histogram of each superpixel block. Each contour corresponds to a target. Based on the correlation of the grayscale histograms of all superpixel blocks in the contour of the same target between different frame tunnel grayscale images, and the discrete degree of the texture difference coefficient of the contour of the same target between different frame tunnel grayscale images, determine the light and shadow change coefficient of each contour in each frame tunnel grayscale image; The tunnel foreign body detection module is used to construct a feature analysis diagram for each frame based on the light and shadow variation coefficient and the tunnel grayscale image of each frame, and detect foreign bodies inside the tunnel in combination with a neural network model.

[0013] In a third aspect, an embodiment of the present application also provides a medium for visual detection of foreign objects inside a smart tunnel, comprising a memory, a processor, and a computer program stored in the memory and running on the processor, wherein the processor implements the steps of the above-mentioned method for visual detection of foreign objects inside a smart tunnel when executing the computer program.

[0014] It can be seen from the above embodiments that the foreign body visual detection method inside a smart tunnel provided by the embodiments of the present application has at least the following beneficial effects: The present application determines the grayscale variation coefficient of each pixel on each contour by extracting all contours in each frame of the tunnel grayscale image and analyzing the variation trend of the grayscale values ​​of all pixels in the neighborhood of each pixel on each contour; the beneficial effect is that foreign objects can be distinguished from the tunnel floor under the influence of light, eliminating the interference of the tunnel floor on the foreign object identification process; the present application determines the texture feature coefficient of each pixel on each contour in each frame of the tunnel grayscale image by analyzing the distribution of all peaks on each histogram curve and the discrete degree of the distance between all adjacent peaks, and combining the grayscale variation coefficient; based on the complexity of the texture feature coefficients of all pixels on each contour and average distribution, determine the texture difference coefficient of each contour in each frame of the tunnel grayscale image, and its beneficial effect is that by analyzing the texture difference near the contour, the area where the foreign matter is located can be more accurately determined; this application is based on the correlation of the grayscale histograms of all superpixel blocks in the contour of the same target between different frames of tunnel grayscale images, and the discrete degree of the texture difference coefficient of the contour of the same target between different frames of tunnel grayscale images, to determine the light and shadow change coefficient of each contour in each frame of the tunnel grayscale image, and combine the neural network model to detect foreign matter inside the tunnel, and its beneficial effect is to enhance the distinguishability of foreign matter and non-foreign matter, and improve the accuracy of foreign matter detection inside the tunnel. This application reduces the interference of light on the foreign matter identification process by analyzing the difference in texture between foreign matter and non-foreign matter, and improves the accuracy of foreign matter detection inside the tunnel. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] In order to more clearly illustrate the technical solutions and advantages in the embodiments of the present application or the prior art, the drawings required for use in the embodiments or the prior art descriptions are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0016] Figure 1 A flowchart of a method for visually detecting foreign objects in a smart tunnel provided by one embodiment of the present application; Figure 2 A schematic diagram of tunnel contour extraction provided in one embodiment of the present application; Figure 3 A schematic diagram of a grayscale variation coefficient extraction process provided by an embodiment of the present application; Figure 4 A schematic diagram of a texture complexity coefficient extraction process provided by an embodiment of the present application; Figure 5 A schematic diagram of a light and shadow variation coefficient extraction process provided by an embodiment of the present application; Figure 6 A block diagram of a foreign body visual detection system inside a smart tunnel provided in one embodiment of the present application; Figure 7 A schematic diagram of the structure of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0017] In order to further explain the technical means and effects adopted by the present application to achieve the predetermined invention purpose, the following, in combination with the accompanying drawings and preferred embodiments, describes in detail the specific implementation method, structure, features and effects of a foreign body visual detection method, system and medium inside a smart tunnel proposed in the present application. In the following description, different "one embodiment" or "another embodiment" does not necessarily refer to the same embodiment. In addition, specific features, structures or characteristics in one or more embodiments may be combined in any suitable form.

[0018] Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs.

[0019] The following is a detailed description of a method, system and medium for visually detecting foreign objects inside a smart tunnel provided by the present application in conjunction with the accompanying drawings.

[0020] See also Figure 1, which shows a flowchart of a method for visually detecting foreign objects in a smart tunnel provided by an embodiment of the present application, the method comprising the following steps: Step S1: Obtain the tunnel grayscale images of all acquisition moments within a preset time period.

[0021] A CMOS high-definition camera is installed at a certain distance inside the tunnel to shoot the tunnel road surface to obtain video data. The specific installation position interval of the camera can be determined according to the actual situation of the tunnel. The number of camera frames is set to f, and the video data within the preset time length t in any camera is obtained. The tunnel images of all acquisition moments within the preset time length t are collected, and the sampling interval is set to T. All tunnel images are subjected to noise reduction processing. Finally, the pre-processed tunnel image is converted into a grayscale image, which is recorded as the tunnel grayscale image.

[0022] It should be noted that the values ​​of the shooting frame number f, the preset time length t and the sampling interval T are all manually set. In this embodiment, the shooting frame number f is 30FPS, the preset time length is 20s, and the sampling interval T is 1s. The implementer can also set them according to the specific situation. This embodiment does not impose any special restrictions.

[0023] In addition, it should be understood that there are many commonly used image denoising algorithms. In this embodiment, a wavelet denoising algorithm is used to remove noise in the tunnel image. The implementer may also use other image denoising algorithms such as median filtering and Gaussian filtering to remove noise in the image. This embodiment does not impose any special restrictions on the selection of image denoising algorithms.

[0024] The wavelet denoising algorithm is a well-known technology, and the specific process of removing noise from an image is not described in detail.

[0025] Step S2: based on the change trend of the grayscale values ​​of all pixels in the neighborhood of each pixel on each contour, determine the grayscale variation coefficient of each pixel on each contour; divide the neighborhood of each pixel on each contour into two sub-regions, and obtain the grayscale histograms of the two sub-regions, fit the grayscale histogram of each sub-region to obtain a histogram curve, and determine the texture complexity coefficient of each sub-region of each pixel on each contour based on the distribution of all peaks on each histogram curve and the discrete degree of the distance between all adjacent peaks, and determine the texture feature coefficient of each pixel on each contour in each frame of the tunnel grayscale image in combination with the grayscale variation coefficient.

[0026] Common foreign objects in tunnels can be divided into the following categories: animals, such as cats and dogs; scattered garbage, such as debris and bag-like objects; and road safety facilities, such as tilted crash buckets and guardrails. These common foreign objects usually appear as irregularly shaped objects in the image, and their colors and textures are mostly different from those of the tunnel road surface. However, the road markings in the tunnel, such as lane dividing lines, guide arrows, etc., as well as the vehicles passing by, have relatively regular outlines and shapes.

[0027] Therefore, in order to identify possible foreign objects in the tunnel, a contour extraction algorithm is used to extract all contours in each frame of the tunnel grayscale image. Among them, each contour is connected, corresponding to a target area, and each object has only one contour.

[0028] It should be noted that there are many commonly used contour extraction algorithms. In this embodiment, the findContours function in OpenCV is used to extract all contours in the grayscale image of each frame of the tunnel. The implementer may also use other contour extraction algorithms such as boundary tracking method or region growing method. This embodiment does not impose any special restrictions on the selection of contour extraction algorithm.

[0029] Among them, the findContours function in OpenCV is a well-known technology, and the specific process of extracting the contour in the image is not repeated here. Preferably, the tunnel contour extraction schematic diagram provided in this embodiment is as follows Figure 2 shown.

[0030] Furthermore, since the color of foreign objects is quite different from the color of the tunnel pavement, the color distribution difference and texture distribution difference of the grayscale values ​​of the contour and its surrounding pixels are analyzed, and based on the change trend of the grayscale values ​​of all pixels in the neighborhood of each pixel on each contour, and the difference in the grayscale values ​​of the pixels between the left and right neighborhoods of each pixel on each contour, the texture feature coefficient of each pixel on each contour in each frame of the tunnel grayscale image is determined to identify possible foreign objects in the tunnel. Specifically: S201: Determine the grayscale variation coefficient of each pixel on each contour in each frame of the tunnel grayscale image based on the variation trend of the grayscale values ​​of all pixels in the neighborhood of each pixel on each contour.

[0031] In each frame of the tunnel grayscale image, a neighborhood W is divided with each pixel on each contour as the center, and all the pixels in the neighborhood of each pixel on each contour are sorted from near to far according to the distance to the pixel in the neighborhood, and the grayscale values ​​of all the sorted pixels are fitted to obtain a fitting curve for each pixel on each contour, and the average of the absolute values ​​of the curvatures of all the fitting points on each fitting curve is used as the grayscale variation coefficient of each pixel on each contour in each frame of the tunnel grayscale image.

[0032] It should be noted that the value of the neighborhood W is artificially set. In this embodiment, the value of the neighborhood W is 224. The implementer can also set it by himself according to the specific situation. This embodiment does not impose any special restrictions.

[0033] In addition, it should be understood that there are many commonly used curve fitting algorithms. In this embodiment, the least squares method is used to obtain the fitting curve. The implementer can also use other curve fitting methods such as polynomial fitting method. This embodiment does not impose any special restrictions on the selection of curve fitting method.

[0034] Among them, the least square method is a well-known technology, and the specific process of fitting the data to obtain the fitting curve will not be repeated here.

[0035] Preferably, the grayscale variation coefficient extraction process schematic diagram provided in this embodiment is as follows Figure 3 shown.

[0036] S202: Divide the neighborhood of each pixel point on each contour into two sub-regions, obtain the grayscale histograms of the two sub-regions, fit the grayscale histogram of each sub-region to obtain a histogram curve, determine the texture complexity coefficient of each sub-region of each pixel point on each contour based on the distribution of all peaks on each histogram curve and the discrete degree of the distance between all adjacent peaks, and determine the texture feature coefficient of each pixel point on each contour in each frame of the tunnel grayscale image in combination with the grayscale variation coefficient.

[0037] The larger the grayscale variation coefficient, the greater the degree of bending of the contour, which means that the shape of the contour is more complex, and the contour edge may have more bumps and unevenness. These bumps and unevenness may cause the texture features on both sides of the contour to be richer and more varied, and the target area is more likely to belong to a foreign body. Usually, the contour of the area where the foreign body is located has complex and diverse textures, while the texture near the outer contour of the non-foreign body area is relatively simple. Therefore, the texture features on both sides of the contour are further extracted, and the distribution and distribution differences of the grayscale values ​​of the pixels on both sides of the contour are analyzed to identify the area where the foreign body is located. Specifically: (1) In each frame of the tunnel grayscale image, the neighborhood of each pixel on each contour is divided into two sub-regions by the continuous contour. The grayscale histograms of all pixels in the two sub-regions are calculated respectively, and the two grayscale histograms are fitted respectively to obtain two histogram curves.

[0038] The grayscale histogram acquisition process is a well-known technology, and the specific acquisition process will not be described in detail.

[0039] In particular, when the neighborhood of a pixel is located at the edge of a contour, a boundary filling algorithm is used to fill in the missing pixels.

[0040] It should be noted that there are many commonly used boundary filling algorithms. In this embodiment, a mirror filling algorithm is used to fill the missing pixels. The implementer may also use other boundary filling algorithms such as constant filling. This embodiment does not impose any special restrictions on the selection of the boundary filling algorithm.

[0041] Among them, the mirror filling method is a well-known technology, and the specific filling process of the missing pixels will not be described in detail.

[0042] (2) If the texture of a sub-region on one side of the contour is complex, its grayscale value is relatively evenly distributed within the grayscale range, there will be multiple peaks in the histogram curve, and the distances between the peaks are relatively balanced. For a sub-window with a relatively simple texture, its grayscale value will be concentrated in a certain interval, and the histogram curve will have fewer peaks and higher amplitudes.

[0043] Therefore, by analyzing the distribution of the grayscale values ​​of the pixels in the sub-regions on either side of the contour, the texture complexity coefficient of any sub-region of each pixel on each contour in the grayscale image of each frame of the tunnel is determined, so as to distinguish the area where the foreign objects are located and the area where the foreign objects are not located in the tunnel. Specifically: In each frame of the tunnel grayscale image, the average level of all peaks on the histogram curve of each sub-region of each pixel point on each contour is analyzed, and the texture complexity coefficient of each sub-region of each pixel point on each contour is the discrete degree of the distance between all adjacent peaks on the histogram curve of each sub-region of each pixel point on each contour, and the result of fusion with the average level.

[0044] It should be noted that there are many methods to measure the average level of a set of data. In this embodiment, the average level of all peaks on the histogram curve is measured by calculating the mean of all peaks on the histogram curve. The implementer may also use other methods to measure the average level of data, such as the geometric mean. This embodiment does not impose any special restrictions on the selection of the method for measuring the average level of data.

[0045] In addition, it should be noted that there are many methods for measuring the degree of dispersion of a set of data. In this embodiment, the standard deviation of the distances between all adjacent peaks on the histogram curve is calculated to measure the degree of dispersion of the distances between all adjacent peaks. The implementer may also use other methods for measuring the degree of dispersion of data, such as variance or dispersion coefficient. This embodiment does not impose any special restrictions on the selection of methods for measuring the degree of dispersion of data.

[0046] It should be understood that fusion refers to the result of combining two or more indicators through positive fusion, that is, combining two or more indicators by adding or multiplying them together in order to obtain a comprehensive indicator, so as to more comprehensively and accurately evaluate a phenomenon or problem. This fusion method is not limited to simple arithmetic operations, but can also include more complex statistical models and analysis methods, which can be selected by the implementer according to the specific situation, and this embodiment does not impose any special restrictions.

[0047] Preferably, in the present embodiment, the texture complexity coefficient of each subregion of each pixel point on each contour is the product of the discreteness of the distances between all adjacent peaks on the histogram curve of each subregion of each pixel point on each contour and the average level; in actual application, as other implementation methods, the texture complexity coefficient of each subregion of each pixel point on each contour is an exponential function value with a natural constant as the base and the sum of the discreteness of the distances between all adjacent peaks on the histogram curve of each subregion of each pixel point on each contour and the average level as the independent variable.

[0048] According to the texture complexity coefficient of each sub-region of each pixel point on each contour, it can be understood that if the sub-region is the region where the foreign body is located, the more complex the texture in the sub-region, the more peaks there are on its histogram curve, and the distribution of these peaks will be more uneven, that is, the greater the average level of all peaks, the greater the discreteness between adjacent peaks, and the greater the texture complexity coefficient; conversely, if the sub-region is not the region where the foreign body is located, the simpler the texture in the sub-region, the fewer peaks there are on its histogram curve, and the distribution of these peaks will be more concentrated, that is, the smaller the average level of all peaks, the smaller the discreteness between adjacent peaks, and the smaller the texture complexity coefficient.

[0049] (3) Further, based on the grayscale variation coefficient and the texture complexity coefficient, the texture feature coefficient of each pixel point on each contour in each frame of the tunnel grayscale image is determined, specifically: Texture feature coefficient of pixel k on contour j in the i-th frame tunnel grayscale image The expression is: ; In the formula, Represents the grayscale variation coefficient of pixel k on contour j in the i-th frame tunnel grayscale image; Represents the difference in texture complexity coefficients between two sub-regions of pixel k on contour j in the i-th frame tunnel grayscale image.

[0050] It should be noted that there are many methods for measuring the differences between data. In this embodiment, the difference between the texture complexity coefficients of the sub-regions is measured by calculating the absolute value of the difference between the texture complexity coefficients of the two sub-regions. The implementer may also use other methods for measuring the difference, such as ratio. This embodiment does not impose any special restrictions on the selection of the method for measuring the differences between data.

[0051] Further, according to the texture feature coefficient of each pixel point on each contour in the grayscale image of each frame of the tunnel, it can be understood that the larger the grayscale variation coefficient of the pixel point on the contour, the greater the contour curvature near the corresponding pixel point, which means that the contour shape where the pixel point is located is more complex and more likely to be the edge of a foreign object; the larger the difference in texture complexity coefficients between the two sub-regions of the pixel point on the contour, the greater the texture difference between the two sides of the contour, and the more likely it is to be the area where the foreign object is located; therefore, if the grayscale variation coefficient of the pixel point on the contour is larger, the difference in texture complexity coefficients between the two sub-regions is larger, then the texture difference coefficient of the pixel point on the contour is larger, indicating that the corresponding contour is more likely to be a part of a foreign object; conversely, the smaller the grayscale variation coefficient of the pixel point on the contour, the smaller the contour curvature near the corresponding pixel point, which means that the contour shape where the pixel point is located is simpler and less likely to be the edge of a foreign object; the smaller the difference in texture complexity coefficients between the two sub-regions of the pixel point on the contour, the smaller the texture difference between the two sides of the contour, and the less likely it is to be the area where the foreign object is located; therefore, if the grayscale variation coefficient of the pixel point on the contour is smaller, the difference in texture complexity coefficients between the two sub-regions is smaller, then the texture difference coefficient of the pixel point on the contour is smaller, indicating that the corresponding contour is less likely to be a part of a foreign object.

[0052] Preferably, the texture complexity coefficient extraction process diagram provided in this embodiment is as follows: Figure 4 shown.

[0053] Step S3: Determine the texture difference coefficient of each contour based on the complexity and average distribution of the texture feature coefficients of all pixels on each contour.

[0054] Generally, the texture features near the continuous pixels on the contour of the foreign object change relatively greatly. Therefore, by analyzing the changes in the texture feature coefficients of all the pixels on the contour, it is possible to identify whether there are foreign objects in the tunnel. Specifically: (1) In each frame of the tunnel grayscale image, calculate the fractal dimension of the texture feature coefficients of all pixels on each contour.

[0055] It should be noted that there are many methods for calculating the fractal dimension of a set of data. In this embodiment, the fractal dimension of each contour is obtained by taking the texture feature coefficients of all pixels on each contour as the input of the Higuchi algorithm. In actual application, as other implementation methods, the implementer may also adopt other methods such as the Hurst exponent method or the box counting method to obtain the fractal dimension. This embodiment does not impose any special restrictions on the method for obtaining the fractal dimension.

[0056] Among them, the Higuchi algorithm is a well-known technology, and the specific process of calculating the fractal dimension is not repeated here.

[0057] (2) Further, based on the fractal dimension and the texture feature coefficient, the texture difference coefficient of each contour in each frame of the tunnel grayscale image is determined, specifically: Texture difference coefficient of contour j in the i-th frame tunnel grayscale image The expression is: ; In the formula, represents the fractal dimension of contour j in the i-th frame tunnel grayscale image; Represents the mean value of the texture feature coefficients of all pixels on contour j in the i-th frame tunnel grayscale image.

[0058] Furthermore, according to the texture difference coefficient of each contour in the grayscale image of each frame of the tunnel, it can be understood that if the fractal dimension of the contour is larger, it means that the degree of texture color change near the continuous pixel points on the corresponding contour is greater; if the texture feature coefficient of the contour is larger, it means that the contour of the foreign body is more irregular and the color difference is more obvious; therefore, if the fractal dimension is larger and the texture feature coefficient of the contour is larger, it means that the contour is more likely to belong to the foreign body part; conversely, if the fractal dimension of the contour is smaller, it means that the degree of texture color change near the continuous pixel points on the corresponding contour is smaller; if the texture feature coefficient of the contour is smaller, it means that the contour of the foreign body is more irregular and the color difference is less obvious; therefore, if the fractal dimension is smaller and the texture feature coefficient of the contour is smaller, it means that the contour is less likely to belong to the foreign body part.

[0059] Step S4: Based on the correlation of the histograms of all superpixel blocks of the same target in the tunnel grayscale images of different frames and the discrete degree of the texture difference coefficient of the same target in different frames, the light and shadow variation coefficient of each contour in the tunnel grayscale images of each frame is determined.

[0060] Affected by the light inside the tunnel, the color and texture of the surface of objects inside the tunnel may be different under different light. Especially when the vehicle is driving, the light shining on the foreign object will cause the light and shadow and texture of the foreign object target area to change significantly, while the vehicle surface is smooth, so the light and shadow and texture of the vehicle target will not change significantly.

[0061] The light and shadow features near the outer contour of the same target area between different frames may be different. Therefore, the light and shadow changes between different frames are further analyzed to eliminate the interference of light and shadow features on foreign body detection. Specifically: (1) Use the optical flow estimation algorithm to obtain the position of the same target in different frames, and set the same serial number for the contours of the same target in different frames.

[0062] It should be noted that there are many commonly used optical flow estimation algorithms. In this embodiment, the Lucas-Kanade optical flow method is used to obtain the position of the same target in different frames. In actual application, as other implementation methods, the implementer may also use the Horn-Schunck optical flow method or the Farneback optical flow method to obtain the position of the same target in different images. This embodiment does not impose any special restrictions on the selection of the optical flow estimation algorithm.

[0063] (2) Furthermore, since the texture of the central part of the same target area between different frames will also be affected by the change of light, in order to more accurately obtain the light and shadow characteristics of the internal image of the target area, this embodiment divides the grayscale image of the tunnel in each frame into multiple superpixel blocks, and further determines whether the grayscale of the tunnel contains foreign body features by analyzing the distribution of the grayscale values ​​of the pixels in the local area within the contour, thereby more accurately detecting whether there are foreign bodies in the tunnel.

[0064] It should be noted that there are many commonly used image segmentation algorithms. In this embodiment, the SLIC superpixel segmentation algorithm is used to segment the internal area of ​​the contour. The implementer can also use region growing or clustering-based image segmentation algorithm to segment the image. This embodiment does not impose any special restrictions on the selection of image segmentation algorithm.

[0065] The SLIC superpixel segmentation algorithm is a well-known technology in the field of image segmentation, and the specific process of image segmentation is not described in detail.

[0066] (3) Furthermore, by analyzing the correlation of the histograms of all superpixel blocks within the contour of the same target between different frame tunnel grayscale images, and the discrete degree of the texture difference coefficient of the contour of the same target between different frame tunnel grayscale images, the light and shadow variation coefficient of each contour in each frame tunnel grayscale image is determined, specifically: The light and shadow variation coefficient of contour j in the i-th frame tunnel grayscale image The expression is: ; In the formula, Indicates the discrete degree of the texture difference coefficients of all contours j in the tunnel grayscale image of the previous preset number of consecutive frames of the i-th frame; represents the Pearson correlation coefficient between the histogram of all superpixel blocks of contour j in the i-th frame tunnel grayscale image and the histogram of all superpixel blocks of contour j in the n-th frame tunnel grayscale image; Indicates a preset number of continuous frames of tunnel grayscale images before the i-th frame of tunnel grayscale image; Indicates a constant greater than 0 to prevent the denominator from being 0. The value of is set manually. The value of is 0.01. The implementer can set it according to the specific situation, provided that the denominator is not 0 and does not excessively affect the calculation result. This embodiment does not impose any special restrictions.

[0067] It should be noted that The value of is set manually. In this embodiment, the preset value The value of is 20, and the implementer can also set it according to the specific situation. This embodiment does not impose any special restrictions. In particular, for a tunnel grayscale image without a preset number of continuous frame tunnel grayscale images, the corresponding tunnel grayscale image is not analyzed.

[0068] It should be understood that there are many methods for measuring the degree of discreteness of a set of data. In this embodiment, the standard deviation of the texture difference coefficient of the contour of the same target between the grayscale images of different frames of the tunnel is calculated to measure the degree of discreteness of the texture difference coefficient. The implementer may also use other methods for measuring the degree of discreteness such as variance or dispersion coefficient. This embodiment does not impose any special restrictions on the selection of the method for measuring the degree of discreteness of data.

[0069] In addition, it should be understood that there are many methods for calculating the correlation coefficient between two variables. In this embodiment, the Pearson correlation coefficient is used to measure the correlation of the histograms of all superpixel blocks within the contour of the same target between the grayscale images of different frames of the tunnel. The implementer may also use other methods for measuring the correlation between variables, such as the Kendall rank correlation coefficient or the Spearman rank correlation coefficient. This embodiment does not impose any special restrictions on the selection of the method for calculating the correlation coefficient between variables.

[0070] The calculation process of the Pearson correlation coefficient is a well-known technique, and the specific calculation process will not be described in detail.

[0071] Furthermore, according to the light and shadow variation coefficient of each contour in each frame tunnel grayscale image, it can be understood that the greater the discrete degree of the texture difference coefficient of the contour of the same target between different frame tunnel grayscale images, the smaller the absolute value of the correlation coefficient of the histogram of all superpixel blocks in the contour of the same target between different frame tunnel grayscale images, indicating that the light and shadow variation degree inside the corresponding contour is greater, that is, the larger the light and shadow variation coefficient of the corresponding contour is, the more likely it is that the light and shadow variation is caused by the light shining on the foreign object; conversely, the smaller the discrete degree of the texture difference coefficient of the contour of the same target between different frame tunnel grayscale images, the larger the absolute value of the correlation coefficient of the histogram of all superpixel blocks in the contour of the same target between different frame tunnel grayscale images, indicating that the light and shadow variation degree inside the corresponding contour is smaller, that is, the smaller the light and shadow variation coefficient of the corresponding contour is, the less likely it is that the light and shadow variation is caused by the light shining on the foreign object.

[0072] Preferably, the schematic diagram of the light and shadow variation coefficient extraction process provided in this embodiment is as follows: Figure 5 shown.

[0073] Step S5: Based on the light and shadow variation coefficient and the grayscale image of each frame of the tunnel, a feature analysis image of each frame is constructed, and foreign objects inside the tunnel are detected in combination with a neural network model.

[0074] The light and shadow variation coefficient of each contour in each frame of the tunnel grayscale image is used as the pixel value of all pixels on the corresponding contour, and the pixel values ​​of the remaining pixels in the tunnel grayscale image are set to 0 to obtain the feature analysis graph of each frame.

[0075] Furthermore, the grayscale image of each tunnel frame is used as the input of the YOLOv7 model, the loss function is set to the SloU loss function, and the BiFPN pyramid structure is adopted to output the feature extraction images of each frame extracted by the backbone network in the YOLOv7 model.

[0076] The feature extraction graph and feature analysis graph of each frame are used as the input of the neck network in the YOLOv7 model to obtain feature information of different scales. The head network in the YOLOv7 model fuses and detects the feature information at different scales, and finally outputs the foreign body detection result.

[0077] Among them, the YOLOv7 model, SloU loss function and BiFPN pyramid structure are all well-known technologies, and the specific principles will not be repeated here.

[0078] At this point, this embodiment reduces the interference of illumination and foreign object scale diversity on foreign object detection by analyzing the texture differences on both sides of the contour and the degree to which the texture of the same target area between different frames is affected by light, thereby avoiding the defects of false detection and missed detection in the foreign object detection process and improving the accuracy of foreign object detection inside the tunnel.

[0079] Based on the same inventive concept as the above method, the embodiment of the present application also provides a foreign body visual detection system inside a smart tunnel, comprising: The tunnel data acquisition module is used to obtain the tunnel grayscale image at all acquisition moments within a preset time period; The tunnel foreign body analysis module is used to extract all contours in each frame of the tunnel grayscale image, and determine the grayscale change coefficient of each pixel on each contour based on the change trend of the grayscale values ​​of all pixels in the neighborhood of each pixel on each contour; The neighborhood of each pixel point on each contour is divided into two sub-regions, and the grayscale histograms of the two sub-regions are obtained, and the grayscale histogram of each sub-region is fitted to obtain a histogram curve, and the texture complexity coefficient of each sub-region of each pixel point on each contour is determined based on the distribution of all peaks on each histogram curve and the discrete degree of the distance between all adjacent peaks, and the texture feature coefficient of each pixel point on each contour in each frame of the tunnel grayscale image is determined in combination with the grayscale variation coefficient; Based on the complexity and average distribution of the texture feature coefficients of all pixels on each contour, the texture difference coefficient of each contour in each frame of the tunnel grayscale image is determined; Divide each frame of the tunnel grayscale image into multiple superpixel blocks, and obtain the grayscale histogram of each superpixel block. Each contour corresponds to a target. Based on the correlation of the grayscale histograms of all superpixel blocks in the contour of the same target between different frame tunnel grayscale images, and the discrete degree of the texture difference coefficient of the contour of the same target between different frame tunnel grayscale images, determine the light and shadow change coefficient of each contour in each frame tunnel grayscale image; The tunnel foreign body detection module is used to construct a feature analysis diagram for each frame based on the light and shadow variation coefficient and the tunnel grayscale image of each frame, and detect foreign bodies inside the tunnel in combination with a neural network model.

[0080] The embodiment of the present application provides a block diagram of a foreign body visual detection system inside a smart tunnel, such as Figure 6 shown.

[0081] Based on the same inventive concept as the above method, an embodiment of the present application also provides a visual detection medium for foreign objects inside a smart tunnel, including a memory, a processor, and a computer program stored in the memory and running on the processor. When the processor executes the computer program, the steps of any one of the above-mentioned methods for visual detection of foreign objects inside a smart tunnel are implemented.

[0082] Figure 7 This is a schematic diagram of the structure of a computer device provided in an embodiment of the present application. Figure 7 As shown, the computer device of this embodiment includes: at least one processor ( Figure 7 Only one is shown), a memory, and a computer program stored in the memory and executable on at least one processor, and when the processor executes the computer program, the steps of any one of the methods described in the above-mentioned method for visual detection of foreign objects inside a smart tunnel are implemented. It should be noted that the above-mentioned sequence of the embodiments of the present application is only for description and does not represent the advantages and disadvantages of the embodiments. And the above-mentioned specific embodiments of this specification are described. In addition, the processes depicted in the accompanying drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0083] The various embodiments in this specification are described in a progressive manner, and the same or similar parts between the various embodiments can be referenced to each other, and each embodiment focuses on the differences from other embodiments.

[0084] The above description is only a preferred embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the principles of the present application should be included in the protection scope of the present application.

Claims

1. A method for visually detecting foreign objects inside a smart tunnel, characterized in that: The method comprises the following steps: Obtain the tunnel grayscale image at all acquisition moments within a preset time period; Extract all contours in each frame of the tunnel grayscale image, and determine the grayscale change coefficient of each pixel on each contour based on the change trend of the grayscale values ​​of all pixels in the neighborhood of each pixel on each contour; The neighborhood of each pixel point on each contour is divided into two sub-regions, and the grayscale histograms of the two sub-regions are obtained, and the grayscale histogram of each sub-region is fitted to obtain a histogram curve, and the texture complexity coefficient of each sub-region of each pixel point on each contour is determined based on the distribution of all peaks on each histogram curve and the discrete degree of the distance between all adjacent peaks, and the texture feature coefficient of each pixel point on each contour in each frame of the tunnel grayscale image is determined in combination with the grayscale variation coefficient; Based on the complexity and average distribution of the texture feature coefficients of all pixels on each contour, the texture difference coefficient of each contour in each frame of the tunnel grayscale image is determined; Divide each frame of the tunnel grayscale image into multiple superpixel blocks, and obtain the grayscale histogram of each superpixel block. Each contour corresponds to a target. Based on the correlation of the grayscale histograms of all superpixel blocks in the contour of the same target between different frame tunnel grayscale images, and the discrete degree of the texture difference coefficient of the contour of the same target between different frame tunnel grayscale images, determine the light and shadow change coefficient of each contour in each frame tunnel grayscale image; Based on the light and shadow variation coefficient and the grayscale image of each frame of the tunnel, a feature analysis image of each frame is constructed, and foreign objects inside the tunnel are detected in combination with a neural network model.

2. A method for visually detecting foreign objects inside a smart tunnel as claimed in claim 1, characterized in that: The method for determining the grayscale variation coefficient of each pixel point on each contour is: In each frame of the tunnel grayscale image, all the pixels in the neighborhood of each pixel on each contour are sorted from near to far according to the distance to the pixel in the neighborhood, and the grayscale values ​​of all the sorted pixels are fitted to obtain various fitting curves. The average of the absolute values ​​of the curvatures of all the fitting points on each fitting curve is used as the grayscale variation coefficient of each pixel on each contour in each frame of the tunnel grayscale image.

3. A method for visually detecting foreign objects inside a smart tunnel as claimed in claim 1, characterized in that: The method for determining the texture complexity coefficient of each sub-region of each pixel point on each contour is as follows: In each frame of the tunnel grayscale image, the average level of all peaks on the histogram curve of each sub-region of each pixel point on each contour is analyzed, and the texture complexity coefficient of each sub-region of each pixel point on each contour is the discrete degree of the distance between all adjacent peaks on the histogram curve of each sub-region of each pixel point on each contour, and the result of fusion with the average level.

4. The method for visually detecting foreign objects in a smart tunnel according to claim 1, characterized in that: The expression of the texture feature coefficient of each pixel point on each contour in each frame of the tunnel grayscale image is: ; In the formula, Represents the texture feature coefficient of pixel k on contour j in the i-th frame tunnel grayscale image; Represents the grayscale variation coefficient of pixel k on contour j in the i-th frame tunnel grayscale image; Represents the difference in texture complexity coefficients between two sub-regions of pixel k on contour j in the i-th frame tunnel grayscale image.

5. The method for visually detecting foreign objects in a smart tunnel according to claim 1, characterized in that: The expression of the texture difference coefficient of each contour in the tunnel grayscale image of each frame is: ; In the formula, represents the texture difference coefficient of contour j in the i-th frame tunnel grayscale image; represents the fractal dimension of contour j in the i-th frame tunnel grayscale image; Represents the mean value of the texture feature coefficients of all pixels on contour j in the i-th frame tunnel grayscale image.

6. The method for visually detecting foreign objects in a smart tunnel according to claim 1, characterized in that: The method for determining the light and shadow variation coefficient of each contour in each frame of the tunnel grayscale image is: The contours of the same target in different frame tunnel grayscale images are set to the same sequence number, and the light and shadow change coefficient of contour j in the i-th frame tunnel grayscale image is The expression is: ; In the formula, Indicates the discrete degree of the texture difference coefficients of all contours j in the tunnel grayscale image of the previous preset number of consecutive frames of the i-th frame; represents the Pearson correlation coefficient between the histogram of all superpixel blocks of contour j in the i-th frame tunnel grayscale image and the histogram of all superpixel blocks of contour j in the n-th frame tunnel grayscale image; Indicates a preset number of continuous frames of tunnel grayscale images before the i-th frame of tunnel grayscale image; Indicates a preset constant greater than 0.

7. The method for visually detecting foreign objects in a smart tunnel according to claim 1, characterized in that: The construction process of each frame feature analysis graph is as follows: The light and shadow variation coefficient of each contour in each frame of the tunnel grayscale image is used as the pixel value of all pixels on the corresponding contour, and the pixel values ​​of the remaining pixels are set to 0 to obtain the feature analysis graph of each frame.

8. The method for visually detecting foreign objects in a smart tunnel according to claim 1, characterized in that: The detecting of foreign matter inside the tunnel comprises: Each frame of the tunnel grayscale image is used as the input of the neural network, and each frame of the feature extraction image is output. The feature analysis image and the feature extraction image of each frame are used as the input of the neural network, and the foreign body detection result is output.

9. A foreign body visual detection system inside a smart tunnel, implementing a foreign body visual detection method inside a smart tunnel as claimed in claim 1, characterized in that: The system comprises: The tunnel data acquisition module is used to obtain the tunnel grayscale image at all acquisition moments within a preset time period; The tunnel foreign body analysis module is used to extract all contours in each frame of the tunnel grayscale image, and determine the grayscale change coefficient of each pixel on each contour based on the change trend of the grayscale values ​​of all pixels in the neighborhood of each pixel on each contour; The neighborhood of each pixel point on each contour is divided into two sub-regions, and the grayscale histograms of the two sub-regions are obtained, and the grayscale histogram of each sub-region is fitted to obtain a histogram curve, and the texture complexity coefficient of each sub-region of each pixel point on each contour is determined based on the distribution of all peaks on each histogram curve and the discrete degree of the distance between all adjacent peaks, and the texture feature coefficient of each pixel point on each contour in each frame of the tunnel grayscale image is determined in combination with the grayscale variation coefficient; Based on the complexity and average distribution of the texture feature coefficients of all pixels on each contour, the texture difference coefficient of each contour in each frame of the tunnel grayscale image is determined; Divide each frame of the tunnel grayscale image into multiple superpixel blocks, and obtain the grayscale histogram of each superpixel block. Each contour corresponds to a target. Based on the correlation of the grayscale histograms of all superpixel blocks in the contour of the same target between different frame tunnel grayscale images, and the discrete degree of the texture difference coefficient of the contour of the same target between different frame tunnel grayscale images, determine the light and shadow change coefficient of each contour in each frame tunnel grayscale image; The tunnel foreign body detection module is used to construct a feature analysis diagram for each frame based on the light and shadow variation coefficient and the tunnel grayscale image of each frame, and detect foreign bodies inside the tunnel in combination with a neural network model.

10. A foreign body visual detection medium inside a smart tunnel, comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that: When the processor executes the computer program, the steps of a method for visually detecting foreign objects inside a smart tunnel as described in any one of claims 1 to 8 are implemented.

Images