A bus safety early warning method and system supporting road condition shadow elimination

Abstract

The application relates to the field of artificial intelligence, and discloses a bus safety early warning method and system supporting road condition shadow removal, which comprises the following specific steps: acquiring an original image; inputting the original image into an HSI space and constructing a shadow identification index SI; obtaining a shadow preliminary detection image; obtaining a shadow image and a shadow preliminary removal image; performing intrinsic decomposition on the shadow preliminary detection image to obtain a color orientation template; constructing an edge contour verification image pair of the original image to obtain a structural skeleton as an edge orientation template; constructing a generative adversarial network; obtaining a detection result output by a discriminator based on the color orientation template and the edge orientation template; obtaining a road condition shadow removal image according to the detection result; and performing bus safety early warning according to the road condition shadow removal image. The application solves the problems that the existing bus safety early warning technology training depends on data, and the phenomena of fuzzy repair area, inconsistent chroma and boundary artifacts, and has the characteristics of high early warning accuracy.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, and more particularly, to a bus safety warning method and system supporting road condition shadow removal. BACKGROUND

[0002] With the development of automatic warning technology, automatic safety warning systems are widely used in public transportation, engineering construction and other fields. As the main public transportation tool in urban construction, the safe operation of buses is related to the safety of the lives and property of the general public. In recent years, frequent bus safety accidents have exposed the lack of warning and prevention and control capabilities during vehicle operation. To address these issues, researchers have proposed using advanced image intelligent processing technology to achieve real-time safety warning for vehicles. In this system, the quality of the image pieces collected by the camera is the main factor in the judgment. Obstructions such as roadblocks, buildings, and driving angles can cause shadows in the collected image pieces, which will lead to potential risks being missed, and thus affect the accurate decision-making of the warning system. For example, in road travel, it is not easy to find dark-colored vehicles or pedestrians wearing dark clothes under the shadow of trees or houses; for example, in low-light scenarios such as at night or in tunnels, pedestrians or vehicles appearing from the dimly lit area cannot be identified in time, which can easily cause traffic accidents; or even after a rainy, snowy, and windy day, there are shadows caused by rain and snow, or temporary attachments to the camera such as leaves and small stains, which can interfere with the identification. According to research, there is currently a lack of research on image shadow processing in bad weather.

[0003] Shadows caused by obstructions are an important factor affecting image quality. They can interfere with the accurate identification of image objects, leading to misjudgment of risks by the bus warning system.

[0004] For the identification and compensation of image shadows, most existing methods are based on the optical characteristics, time domain and spatial domain characteristics of shadows, and use mathematical and physical models to solve them for discrimination and removal. For example, in the paper "DC-ShadowNet: Single-Image Hard and Soft Shadow Removal Using Unsupervised Domain-Classifier Guided Network", the traditional entropy physical model and the perception loss of the observed feature image are combined to realize the removal of shadows in unsupervised learning. The existing technology also proposes a vehicle shadow removal method based on spatiotemporal multi-feature, which uses shadow chroma, texture and spectral features, and introduces time domain filtering to obtain a foreground mask to remove shadows. However, in the field of bus warning, the image data background contains various complex things, including buildings, pedestrians, vehicles, road conditions and other objects, and the detection and removal of shadows require understanding of abstract semantics and complex processing, such as balancing shallow information like color ratio, and identifying shadow geometry texture, etc.

[0005] However, the current method still has the problems of training dependence on data, blurred repair area, inconsistent color, and boundary artifact phenomenon. In order to solve these problems, how to invent a bus safety warning method supporting road condition shadow removal is a technical problem that needs to be solved in the technical field. SUMMARY

[0006] In order to solve the problems of training dependence on data, blurred repair area, inconsistent color and boundary artifact phenomenon in the existing bus safety warning technology, a bus safety warning method and system supporting road condition shadow removal are provided, which has the characteristics of high warning accuracy.

[0007] To achieve the above-mentioned purposes of the present application, the technical solutions adopted are as follows:

[0008] A bus safety warning method supporting road condition shadow removal, comprising the following specific steps:

[0009] Obtaining an original image;

[0010] Inputting the original image into HSI space and constructing a shadow identification index SI; using the maximum inter-class variance method to segment SI to extract the preliminary coarsely identified shadow area, thereby obtaining a shadow preliminary detection image;

[0011] Calculating the mapping relationship between the shadow preliminary detection image and the shadow-free image, and obtaining a shadow image and a shadow preliminary removal image according to the mapping relationship;

[0012] Performing intrinsic decomposition on the shadow preliminary detection image to obtain a reflectance intrinsic image reflecting essential color information; using the reflectance intrinsic image as a color guide template;

[0013] An edge contour verification image of the original image is constructed, and a structural skeleton is obtained as an edge guide template in combination with a binary contour structure prior of a shadow area generated;

[0014] A generative adversarial network including a detection model and a shadow removal model is constructed;

[0015] Based on the color guide template and the edge guide template, the original image, the shadow preliminary detection image and the shadow preliminary removal image are input into the detection model to obtain a detection result output by the discriminator; and based on the detection result, the original image, the shadow preliminary detection image and the shadow preliminary removal image are input into the shadow removal model to obtain a road condition shadow removal image;

[0016] A public transport safety warning is performed according to the road condition shadow removal image.

[0017] Preferably, the original image is input into an HSI space and a shadow recognition index SI is constructed; the SI is segmented to extract a preliminary coarse recognition shadow area by using the maximum inter-class variance method, so as to obtain a shadow preliminary detection image, and the specific steps are as follows:

[0018] The original image is input into an HSI space, and a shadow recognition index SI matrix is constructed according to a shadow imaging principle and a shadow characteristic in the HSI space:

[0019]

[0020] Wherein, S, I are corresponding components of the HSI space respectively, H represents hue, S represents color saturation, and I represents brightness;

[0021] The maximum inter-class variance method is used to take the gray value with the maximum inter-class variance of different regions of the SI matrix as a threshold value, the shadow foreground and the non-shadow background region are divided according to the threshold value, the preliminary coarse recognition shadow area is segmented and extracted, and a shadow preliminary detection image is obtained.

[0022] Further, a mapping relationship between the shadow preliminary detection image and a shadow-free image is calculated, and a shadow preliminary removal image is obtained according to the mapping relationship, and the specific steps are as follows:

[0023] It is assumed that there is a mapping relationship between the shadow preliminary detection image and a shadow-free preliminary detection image:

[0024] ψ:I sf (x,y)—→→I s (x,y)

[0025] Wherein, x and y are x-axis coordinates and y-axis coordinates in the image respectively, I sf is the shadow preliminary detection image, I s is a shadow image, and ψ is an atmospheric comprehensive action index including direct radiation, reflection and scattering;

[0026] The inverse mapping of atmospheric transmission is characterized by the ratio of the radiation received by the ground object without shadow to the radiation received by the ground object with shadow, and the inverse mapping model is obtained by simplifying the model:

[0027]

[0028] Where t(μ s ) is the atmospheric downward transmittance of light reaching the ground object, and τ is the atmospheric attenuation coefficient.

[0029] The transmission mapping in the inverse mapping model is retained, and the mapping model for mapping the shadow image to the shadow preliminary removal image is obtained by solving:

[0030] I sf (x,y)=I s (x,y)·ψ -1 (x,y)

[0031] The shadow preliminary detection image is dilated for several times, the outline image of the preliminary detection shadow image is obtained by subtracting the original image from the image obtained after the dilation operation, and t(μ s ) and τ are obtained by fitting the pixels in the shadow region of the shadow image and the pixels in the non-shadow region within the outline range through the least square method.

[0032] t(μ s ) and τ are brought into the mapping model to obtain the shadow preliminary removal image.

[0033] Further, the reflectance eigenimage reflecting the essential color information is obtained by eigen-decomposing the shadow image, and the specific steps are as follows:

[0034] According to the premise of the ideal Lambertian surface assumption, a total variation model is constructed:

[0035]

[0036] Where S is the to-be-solved reflectance eigenimage, p is the index of the image pixel, λ is the weight coefficient, V x(y) (p) and N x(y) (p) are the window total gradient variation and the window inherent total gradient variation of the pixel p in the x and y directions, and I p is the image pixel of the shadow image.

[0037]

[0038] Where g p,qcharacterize the spatial affinity weight between p and q; the closer q is to p, the larger the weight is; define σ as a spatial scale factor to control the shape of the Gaussian function,

[0039]

[0040] define the nonlinear term u x(y),q , x(y),q :

[0041]

[0042] where G σ is a Gaussian kernel function with a standard deviation of σ, and * is a convolution operation;

[0043] solving the total variation model to calculate the reflectance eigenimage.

[0044] Further, the total variation model is simplified to convert the non-convex optimization problem of the total variation model into a linear equation system problem:

[0045]

[0046] where v S(I) is a matrix converted into a one-dimensional vector form, C x(y) is a Toeplitz matrix with a forward difference gradient operator, U x(y) , W x(y) is a diagonal matrix rewritten from u x(y) , w x(y) ,

[0047]

[0048] calculating the linear equation system problem to obtain the reflectance eigenimage reflecting the intrinsic color property.

[0049] Further, an edge contour verification image pair of the original image is constructed, and a binary contour structure prior of the generated shadow area is combined, and the specific steps are:

[0050] constructing an edge contour verification image pair: converting the original image into a grayscale image, denoted as I gray ;

[0051] applying a canny operator to the preliminary image to extract edges; performing Gaussian filtering on the extracted image to remove interfering fine impurities, thereby obtaining a binary edge contour image;

[0052] denote the binary edge contour image as C gt , randomly generate an irregular mask, denoted as M; cover the irregular mask M on the grayscale image I gray ,

[0053]

[0054] wherein is Hadamard product;

[0055] the gray image after covering the mask the binary edge contour image is C gt and the irregular mask M is input into the edge prediction network G for training to obtain the predicted contour completion image C pred :

[0056]

[0057] Further, the edge prediction network comprises a generator and a discriminator; the generator comprises an encoder, a decoder, and a plurality of residual blocks; the discriminator comprises a plurality of convolutional layers, the plurality of convolutional layers are connected using a LeakyReLu function, and finally a sigmoid function is used for output;

[0058] the gray image after covering the mask the binary edge contour image is C gt When the irregular mask M is input into the edge prediction network for training, the input data is input into the generator:

[0059] The encoder performs 3-line mirror padding around the matrix composed of the input data; the padded data is input into 3 sequentially connected sequence models, the input data is represented in a lower dimension form to extract key features of the data; each sequence model comprises a two-dimensional convolution, an instance normalization layer, and a Relu activation function;

[0060] The low-dimensional data is input into a plurality of connected residual blocks; the convolution of the residual block uses dilated convolution;

[0061] The output of the plurality of connected residual blocks is input into the decoder for decoding to obtain the image generated by the generator; the decoder performs up-sampling through 2 sequence models; the sequence model comprises a two-dimensional transpose convolution, an instance normalization layer, a Relu activation function, a mirror padding block, a two-dimensional convolution, and a sigmoid function layer connected in sequence;

[0062] The generated image is input into the discriminator;

[0063] The edge prediction network is trained using an adversarial loss and a feature loss function;

[0064] The trained edge prediction network outputs a prediction result C pred :

[0065]

[0066] wherein, is the output result of the edge prediction network, mask is a shadow mask, Z edg is a binary contour image of the original road image.

[0067] Further, a generative adversarial network including a detection model and a shadow removal model is constructed based on the color-oriented template and the edge-oriented template, specifically: two generative adversarial models with the same architecture are constructed, the first generative adversarial model is taken as the detection model, and the second generative adversarial model is taken as the shadow removal model; the detection model and the adversarial model both include a generator and a discriminator; the discriminator of the generative adversarial model adopts a conditional discriminator PatchGAN.

[0068] The original image and the preliminary shadow detection image are input into the detection model to obtain a detection result output by the discriminator; according to the detection result, the original image, the preliminary shadow detection image and the preliminary shadow removal image are input into the shadow removal model to obtain a road shadow removal image, and the specific steps are as follows:

[0069] The original image is input into the generator of the detection model; the image generated by the generator is combined with the original image and the preliminary shadow detection image, and is input into the discriminator of the detection model; the discriminator performs authenticity judgment on the image generated by the generator based on the color-oriented template and the edge-oriented template, assuming that the shadow removal result image has color and edge prior information, so as to screen the image generated by the generator;

[0070] The screened image generated by the generator is input into the generator of the shadow removal model, and the image generated by the generator is combined with the original image, the preliminary shadow detection image and the preliminary shadow removal image, and is input into the discriminator of the shadow removal model after being connected in pairs in the channel to identify the authenticity of the generated image and focus on the corresponding relationship on the image information position;

[0071] The generative adversarial network is trained; and a road shadow removal image is output by the shadow removal model of the trained generative adversarial network.

[0072] Further, when the generative adversarial network is trained, the loss function used is specifically:

[0073] L total = L chroma + L Mix + L possion

[0074] wherein, L chroma is a color consistency loss, L mix is a structure perception loss, and L possion is a Poisson-oriented loss; specifically:

[0075]

[0076] where c∈{r,g,b} represents a corresponding color channel of the image, respectively, are pixel values of the corresponding color channel of the generated image and the brightened image;

[0077]

[0078] wherein,

[0079]

[0080] wherein,

[0081]

[0082] wherein, S is a closed subset of the image domain, Ω is a closed subset of S representing a shadow region to be reconstructed, f * is a known scalar function defined on the boundary and outside of Ω, representing known information of non-shadow region.

[0083] A bus safety warning system supporting road condition shadow removal, comprising an image extraction module, a prior detection module, a microscopic removal module and a safety warning module connected in sequence; the prior detection module comprises a shadow preliminary detection component, a shadow preliminary removal component, a color guide template component and an edge guide template component connected in sequence;

[0084] The image extraction module is used for acquiring an original image;

[0085] The preliminary detection component is used for inputting the original image into HSI space and constructing a shadow identification index SI; the maximum inter-class variance method is used to segment and extract the SI to obtain a preliminary coarse-identified shadow region, thereby obtaining a shadow preliminary detection image;

[0086] The shadow preliminary removal component is used for calculating a mapping relationship between the shadow preliminary detection image and a shadow-free image, and obtaining a shadow image and a shadow preliminary removal image according to the mapping relationship;

[0087] The color guide template component is used for obtaining a reflectance eigenimage reflecting essential color information by eigen-decomposing the shadow preliminary detection image; and the reflectance eigenimage is used as a color guide template;

[0088] The edge guide template component is used for constructing an edge contour verification image pair of the original image, and obtaining a structural skeleton as an edge guide template in combination with a binary contour structure prior of the generated shadow region;

[0089] The micro-removal module is used to build a generative adversarial network including a detection model and a shadow removal model; based on a color guide template and an edge guide template, the original image, the shadow preliminary detection image are input into the detection model to obtain a detection result output by a discriminator; according to the detection result, the original image, the shadow preliminary detection image and the shadow preliminary removal image are input into the shadow removal model to obtain a road condition shadow removal image.

[0090] The safety warning module is used for bus safety warning according to the road condition shadow removal image.

[0091] The beneficial effects of the present application are as follows:

[0092] The present application realizes a bus safety warning method supporting road condition shadow removal through coarse-grained prior detection and fine-grained micro-removal. In coarse-grained prior detection, the mapping relationship between the shadow preliminary detection image and the shadow-free image is calculated, the shadow image and the shadow preliminary removal image are obtained according to the mapping relationship, the edge contour verification image pair of the original image is constructed, and the structural skeleton is obtained as the edge guide template. In fine-grained micro-removal, the present application builds a generative adversarial network including a detection model and a shadow removal model, and based on a color guide template and an edge guide template, the original image and the shadow preliminary detection image are input into the detection model to obtain a detection result output by a discriminator; according to the detection result, the original image, the shadow preliminary detection image and the shadow preliminary removal image are input into the shadow removal model to obtain a road condition shadow removal image. Thus, the present application solves the problems of the existing bus safety warning technology training relying on data, repairing fuzzy areas, inconsistent color and boundary artifact phenomenon, and has the characteristics of high warning accuracy. BRIEF DESCRIPTION OF DRAWINGS

[0093] Figure 1 is a specific flow diagram of a bus safety warning method supporting road condition shadow removal.

[0094] Figure 2 is an edge prediction network structure schematic diagram of a bus safety warning method supporting road condition shadow removal,

[0095] Figure 3 is a post-network structure schematic diagram of a bus safety warning method supporting road condition shadow removal.

[0096] Figure 4 is a system flow schematic diagram of a bus safety warning system supporting road condition shadow removal. DETAILED DESCRIPTION

[0097] The present application will be described in detail below in combination with the drawings and specific embodiments.

[0098] Example 1

[0099] As Figure 1 shown, a bus safety warning method supporting road condition shadow removal includes the following specific steps:

[0100] Obtain an original image;

[0101] Input the original image into the HSI space and construct a shadow recognition index SI; use the maximum inter-class variance method to segment SI to extract a preliminarily coarsely recognized shadow region, thereby obtaining a shadow preliminary detection image;

[0102] Calculate the mapping relationship between the shadow preliminary detection image and the shadow-free image, and according to the mapping relationship, obtain a shadow image and a shadow preliminary removal image;

[0103] Perform eigen-decomposition on the shadow preliminary detection image to obtain a reflectance eigen-image reflecting essential color information; use the reflectance eigen-image as a color guide template;

[0104] Construct an edge contour verification image pair of the original image, and combine the generated binary contour structure prior of the shadow region to obtain a structure skeleton as an edge guide template;

[0105] Construct a generative adversarial network including a detection model and a shadow removal model;

[0106] Based on the color guide template and the edge guide template, input the original image and the shadow preliminary detection image into the detection model to obtain a detection result output by a discriminator; according to the detection result, input the original image, the shadow preliminary detection image, and the shadow preliminary removal image into the shadow removal model to obtain a road condition shadow removal image;

[0107] According to the road condition shadow removal image, perform bus safety warning.

[0108] Embodiment 2

[0109] In one specific embodiment, the original image is input into the HSI space and a shadow recognition index SI is constructed; the maximum inter-class variance method is used to segment SI to extract a preliminarily coarsely recognized shadow region, thereby obtaining a shadow preliminary detection image, and the specific steps are as follows:

[0110] Input the original image into the HSI space, and construct a shadow recognition index SI matrix according to the shadow imaging principle and the characteristics of the shadow in the HSI space:

[0111]

[0112] Wherein S, I are the corresponding components of the HSI space respectively, H represents hue, S represents color saturation, and I represents brightness;

[0113] In this embodiment, the saturation value difference index (SI) of the shadow region pixels will be much larger than that of the non-shadow region pixels, and the HSI color space is based on the human visual system and uses hue (H), saturation (S), and intensity (I) to describe color. The HSI color space can be described by a conical space model.

[0114] The maximum inter-class variance method is used to obtain the threshold value of the gray value with the maximum inter-class variance in different regions of the SI matrix, and the shadow foreground and non-shadow background regions are divided according to the threshold value, so as to segment and extract the preliminarily recognized shadow region and obtain a shadow preliminary detection image.

[0115] In this embodiment, the greater the inter-class variance between the background and the foreground, the greater the difference between the two parts of the image.

[0116] In one specific embodiment, the mapping relationship between the shadow preliminary detection image and the shadow-free image is calculated, and the shadow preliminary removal image is obtained according to the mapping relationship. The specific steps are as follows:

[0117] The mapping relationship between the original image and the theoretical shadow-free image is calculated by using the atmospheric transmission model. It is assumed that the shadow image is formed by the comprehensive influence of direct reflection, reflection and other atmospheric effects on the shadow-free image, that is, there is a mapping relationship between the shadow preliminary detection image and the shadow-free preliminary detection image:

[0118] ψ: I sf (x,y)→I s (x,y)

[0119] Where x and y are the x-axis and y-axis coordinates in the image, I sf is the shadow preliminary detection image, I s is the shadow image, and ψ is the comprehensive atmospheric action index including direct reflection, reflection and scattering.

[0120] The inverse mapping of the atmospheric transmission effect is represented by the ratio of the radiation received by the ground object target when it is shadowed to the radiation received when it is not shadowed, the model is simplified by division, and the inverse mapping model is obtained as follows:

[0121]

[0122] Where t(μ s ) is the atmospheric downward transmittance of the light reaching the object target, and τ is the atmospheric attenuation coefficient.

[0123] In this embodiment, the influence of the radiant flux on the shadow removal of the image is far less significant than the influence of the transmission mapping, so only the main transmission mapping is retained in the formula. The mapping model for mapping the shadow image to the shadow preliminary removal image is obtained by solving, and is specifically as follows:

[0124] I sf (x,y)=I s (x,y)·ψ -1 (x,y)

[0125] The preliminary shadow detection image is subjected to several times of dilation operation; the image obtained after the dilation operation is subtracted from the original image to obtain a contour image of the preliminary shadow detection image; the pixels in the shadow region of the shadow image and the pixels in the non-shadow region within the contour range are fitted by the least square method, so that the shadow region is compared and learned with the surrounding non-shadow region to obtain t(μ s ) and τ;

[0126] t(μ s ) and τ are brought into the mapping model calculation to obtain a preliminary shadow removal image.

[0127] In this embodiment, the shadow is a special light and shadow phenomenon, and the brightness changes obviously before and after the shadow shielding, while the texture change is not obvious. According to this characteristic, the method uses the total variation method to decompose the shadow image into a reflectance eigenimage reflecting the essential color information and an illumination image reflecting the light and shadow situation.

[0128] In one specific embodiment, the shadow image is eigen-decomposed to obtain a reflectance eigenimage reflecting the essential color information, and the specific steps are as follows:

[0129] According to the premise of the ideal Lambertian surface assumption, a total variation model is constructed:

[0130]

[0131] Wherein, S is the reflectance eigenimage to be solved, p is the index of the image pixel point, λ is the weight coefficient, V x(y) (p) and N x(y) (p) are the window total gradient change and the window inherent total gradient change of the pixel point p in the x and y directions, I p is the image pixel point of the shadow image;

[0132]

[0133] Wherein, g p,q characterizes the spatial affinity weight between p and q points; the closer the q point to the p point, the greater the weight; σ is defined as a spatial scale factor, which is used to control the shape of the Gaussian function,

[0134]

[0135] The nonlinear term u x(y),q is defined as w x(y),q :

[0136]

[0137] In the formula, G σ is a Gaussian kernel function with a standard deviation of σ, and * is a convolution operation.

[0138] Solving the total variation model calculates the reflectance eigenimage.

[0139] In one embodiment, the total variation model is solved to calculate the reflectance eigenimage, specifically: the total variation model is simplified, and a non-convex optimization problem of the total variation model is converted into a linear equation set problem:

[0140]

[0141] Wherein, v S(I) is a matrix converted into a one-dimensional vector form, C x(y) is a Toeplitz matrix with a forward difference gradient operator, U x(y) , W x(y) is a diagonal matrix rewritten from u x(y) , w x(y) ,

[0142]

[0143] The linear equation set problem is calculated to obtain the reflectance eigenimage reflecting the intrinsic color property.

[0144] In this embodiment, the edge contour verification image pair of the original image is constructed, and when the binary contour structure prior of the generated shadow area is combined, the repair of high-frequency information in the shadow removal process is mainly focused.

[0145] In one embodiment, the edge contour verification image pair of the original image is constructed, and the binary contour structure prior of the generated shadow area is combined, and the specific steps are:

[0146] The edge contour verification image pair is constructed: the original image is converted into a gray image, denoted as I gray ;

[0147] The canny operator is used for edge extraction on the preliminary removal image; the extracted image is subjected to Gaussian filtering to remove interfering impurities, and a binary edge contour image is obtained.

[0148] The binary edge contour image is denoted as C gt , a random irregular mask is generated, denoted as M; the irregular mask M is overlaid on the gray image I gray to obtain

[0149]

[0150] Wherein, is Hadamard product;

[0151] The gray image after covering the mask The binary edge contour image is C gt And the irregular mask M is input into the edge prediction network G to obtain a predicted contour completion image C pred :

[0152]

[0153] As Figure 2 shown, in one specific embodiment, the edge prediction network includes a generator and a discriminator; the generator includes an encoder, a decoder, and a plurality of residual blocks; the discriminator includes a plurality of convolutional layers, the plurality of convolutional layers are connected using a LeakyReLu function, and finally a sigmoid function is used for output;

[0154] The gray image after covering the mask The binary edge contour image is C gt When the irregular mask M is input into the edge prediction network for training, the input data is input into the generator:

[0155] The encoder performs 3-line mirror padding around the matrix composed of the input data; the padded data is input into 3 sequentially connected sequence models, the input data is represented in a lower dimension form to extract key features of the data; each sequence model includes a two-dimensional convolution, an instance normalization layer, and a Relu activation function;

[0156] The low-dimensional data is input into a plurality of connected residual blocks; the convolution of the residual block uses dilated convolution;

[0157] The output of the plurality of connected residual blocks is input into the decoder for decoding to obtain an image generated by the generator; the decoder performs up-sampling through 2 sequence models; the sequence model includes a two-dimensional transpose convolution, an instance normalization layer, a Relu activation function, a mirror padding block, a two-dimensional convolution, and a sigmoid function layer connected in sequence;

[0158] The generated image is input into the discriminator;

[0159] In this embodiment, specifically, 8 residual blocks are included; the discriminator uses 5 convolutional layers with a size of 4x4;

[0160] The edge prediction network is trained using an adversarial loss and a feature loss function;

[0161] The trained edge prediction network outputs a prediction result C pred :

[0162]

[0163] wherein, is the output result of the edge prediction network, mask is a shadow mask, Z edg is a binary contour image of the original road image.

[0164] As Figure 3 shown, in one specific embodiment, a generative adversarial network including a detection model and a shadow removal model is constructed based on the color-oriented template and the edge-oriented template, specifically: 2 generative adversarial models with the same architecture are constructed, the first generative adversarial model is taken as the detection model, and the second generative adversarial model is taken as the shadow removal model; the detection model and the adversarial model both include a generator and a discriminator; the discriminator of the adversarial model adopts a conditional discriminator PatchGAN;

[0165] The original image and the preliminary shadow detection image are input into the detection model to obtain a detection result output by the discriminator; according to the detection result, the original image, the preliminary shadow detection image and the preliminary shadow removal image are input into the shadow removal model to obtain a road shadow removal image, and the specific steps are as follows:

[0166] The original image is input into the generator of the detection model; the image generated by the generator is combined with the original image and the preliminary shadow detection image, and is input into the discriminator of the detection model; the discriminator performs authenticity judgment on the image generated by the generator based on the color-oriented template and the edge-oriented template, assuming that the shadow removal result image has color and edge prior information, so as to screen the image generated by the generator;

[0167] The screened image generated by the generator is input into the generator of the shadow removal model, and the image generated by the generator is combined with the original image, the preliminary shadow detection image and the preliminary shadow removal image, and is respectively paired and connected in the channel to be jointly input into the discriminator of the shadow removal model to discriminate the authenticity of the generated image and focus on the corresponding relationship on the image information position;

[0168] The generative adversarial network is trained; and the road shadow removal image is output by the shadow removal model of the trained generative adversarial network.

[0169] In one specific embodiment, when the generative adversarial network is trained, the loss function used is specifically:

[0170] L total = L chroma + L Mix + L possion

[0171] wherein, L chroma is a color consistency loss, L mix is a structure perception loss, and L possionis the Poisson guided loss; in particular:

[0172]

[0173] where c e {r, g, b} represents a corresponding color channel of the image, are pixel values of the corresponding color channel of the generated image and the brightened image, respectively;

[0174]

[0175] where,

[0176]

[0177] where,

[0178]

[0179] where S is a closed subset of the image domain, and is a closed subset of S representing a shadow region to be reconstructed, f * is a known scalar function defined on the boundary and outside of the domain, representing known information of a non-shadow region.

[0180] In this embodiment, the BSTLD data set is still used to experimentally verify the bus safety warning method supporting road shadow removal; wherein the BSTLD data set contains 13427 various road traffic images, which are divided into a training set of 5093 and a test set of 8334.

[0181] From the experimental results, the originally unclear person or vehicle under the shadow in the collected image is highlighted, achieving the shadow removal and enhancement effect of the road image. After processing, the interference of the shadow in the original image data is reduced, which is conducive to improving the decision accuracy of the bus safety warning system. Therefore, the present application solves the problems of existing bus safety warning technology training dependence on data, repair area blur, color inconsistency and boundary artifact phenomenon, and has the characteristics of high warning accuracy.

[0182] Embodiment 3

[0183] As shown in Figure 4 , a bus safety warning system supporting road shadow removal includes an image extraction module, a prior detection module, a microscopic removal module, and a safety warning module connected in sequence; the prior detection module includes a shadow preliminary detection component, a shadow preliminary removal component, a color guided template component, and an edge guided template component connected in sequence.

[0184] In the prior detection module, the position of the shadow is first detected preliminarily and macroscopic shadow removal is performed, then the color prior template based on reflectance eigenvalue is calculated in the color guide template component, then the edge template is generated to assist the repair of the texture structure. Finally, in the microscopic removal module, the generated adversarial network is used for fine removal and enhancement of the shadow, and the output result is obtained.

[0185] Specifically,

[0186] The image extraction module is used to obtain an original image.

[0187] The preliminary detection component is used to input the original image into the HSI space and construct a shadow identification index SI; the maximum inter-class variance method is used to segment the SI to extract the preliminarily identified shadow area, so as to obtain a shadow preliminary detection image.

[0188] The shadow preliminary removal component is used to calculate the mapping relationship between the shadow preliminary detection image and the shadow-free image, and according to the mapping relationship, a shadow image and a shadow preliminary removal image are obtained.

[0189] The color guide template component is used to perform eigenvalue decomposition on the shadow preliminary detection image to obtain a reflectance eigenimage reflecting the essential color information; and the reflectance eigenimage is used as a color guide template.

[0190] The edge guide template component is used to construct an edge contour verification image pair of the original image, and combine the binary contour structure prior of the generated shadow area to obtain a structure skeleton as an edge guide template.

[0191] The microscopic removal module is used to construct a generative adversarial network including a detection model and a shadow removal model; based on the color guide template and the edge guide template, the original image and the shadow preliminary detection image are input into the detection model to obtain a detection result output by a discriminator; according to the detection result, the original image, the shadow preliminary detection image and the shadow preliminary removal image are input into the shadow removal model to obtain a road condition shadow removal image.

[0192] The safety warning module is used to perform bus safety warning according to the road condition shadow removal image.

[0193] In this embodiment, the present patent proposes a method of combining a traditional physical model and a generative adversarial network for shadow removal, and the overall model is divided into two modules: a prior detection module and a microscopic removal module. After the preliminary position and removal state of the shadow are calculated by using the atmospheric transmission model and the difference component method, the original image is further reconstructed by using the generated adversarial network as a reference in the microscopic removal module.

[0194] Compared with the prior art, the unsupervised shadow removal system is provided, the problem that the removal effect is affected due to large difference of data sets in different road conditions is overcome, and the generalization of the model is improved.

[0195] For the multi-objective task of clear internal structure of the region after shadow removal, consistent color tone with the surrounding non-shadow region and natural transition of shadow removal boundary, the method decouples three dimensions of color, edge and boundary. The total task is divided into multiple sub-problems, and the corresponding prior information is calculated.

[0196] By introducing the prior information, using image eigenvalue decomposition and edge generator to extract the color and texture essential attribute information of the shadow region as a template to help removal and enhancement, the removal result has more prominent effect in maintaining color, edge and texture consistency compared with other methods. In addition, the boundary constraint design based on the Poisson equation also makes the transition of the shadow removal boundary more natural. It can better assist the efficient and accurate operation of the public transportation safety warning system.

[0197] Obviously, the above embodiments of the present application are only examples for clearly illustrating the present application, and are not intended to limit the embodiments of the present application. Any modification, equivalent replacement and improvement made within the spirit and principle of the present application shall be included in the protection scope of the claims of the present application.

Claims

1. A public transport safety early warning method supporting road condition shadow removal, characterized in that: The specific steps include the following: Obtain the original image; The original image is input into the HSI space and a shadow recognition index SI is constructed. The maximum inter-class variance method is used to segment the SI to extract the shadow region for preliminary coarse recognition, thereby obtaining a preliminary shadow detection image. Calculate the mapping relationship between the preliminary shadow detection image and the shadow-free image, and obtain the shadow image and the preliminary shadow removal image based on the mapping relationship; The reflectance eigenimage, which reflects essential color information, is obtained by eigenvalue decomposition of the preliminary shadow detection image; the reflectance eigenimage is then used as a color-guided template. The process of decomposing the initial shadow detection image into an intrinsic reflectance image that reflects essential color information involves the following steps: Based on the assumption of an ideal Lambert surface, a total variation model is constructed: Where S is the intrinsic reflectance image to be solved, p is the index of the image pixel, λ is the weighting coefficient, and V x(y) (p), N x(y) (p) represents the total window gradient change and the window's inherent total gradient change of pixel p in the x and y directions, respectively. p The image pixels of the shadow image; Among them, g p,q The spatial affinity weights between points p and q are used to characterize the relationship; the closer point q is to point p, the larger the weight. σ is defined as the spatial scale factor, used to control the shape of the Gaussian function. Define the nonlinear term u x(y),q w x(y),q : In the formula, G σ σ represents a Gaussian kernel function with a standard deviation of σ, and * represents the convolution operation; Solve the total variation model to calculate the intrinsic reflectance image; Construct edge contour verification image pairs from the original image, and combine them with the prior binary contour structure of the generated shadow region to obtain the structural skeleton as an edge guiding template; Construct a generative adversarial network that includes a detection model and a shadow removal model; Based on color-guided templates and edge-guided templates, the original image and the preliminary shadow detection image are input into the detection model to obtain the detection results output by the discriminator; based on the detection results, the original image, the preliminary shadow detection image, and the preliminary shadow removal image are input into the shadow removal model to obtain the road condition shadow removal image; Public transport safety warnings are issued based on images with road condition shadows removed.

2. The public transport safety early warning method supporting road condition shadow removal according to claim 1, characterized in that: The original image is input into the HSI space and a shadow recognition index SI is constructed. The maximum inter-class variance method is used to segment the SI to extract the shadow regions for preliminary coarse recognition, thereby obtaining a preliminary shadow detection image. The specific steps are as follows: The original image is input into the HSI space, and a shadow recognition index SI matrix is ​​constructed based on the shadow imaging principle and the characteristics of shadows in the HSI space: Where S and I are the corresponding components of the HSI space, H represents hue, S represents color saturation, and I represents brightness; Using the maximum inter-class variance method, the gray value with the largest inter-class variance in different regions of the SI matrix is ​​used as the threshold. Based on the threshold, the shadow foreground and non-shadow background regions are divided, and the shadow regions that are initially identified are segmented and extracted to obtain the preliminary shadow detection image.

3. The public transport safety early warning method supporting road condition shadow removal according to claim 2, characterized in that: The mapping relationship between the initially detected shadow image and the image without shadows is calculated. Based on the mapping relationship, the image with preliminary shadow removal is obtained. The specific steps are as follows: Assume there is a mapping relationship between the preliminary shadow detection image and the preliminary shadow-free detection image: ψ:I sf (x,y)→I s (x,y) Where x and y are the x-axis and y-axis coordinates of the image, respectively. sf For the initial shadow detection image, I s The image is a shaded image, and ψ is an index of the combined effects of the atmosphere, including direct sunlight, reflection, and scattering. The inverse mapping of atmospheric transport is characterized by the ratio of the amount of radiation received by ground objects when they are unshaded to the amount received when they are shaded. After simplifying the model by reduction, the inverse mapping model is obtained as follows: Wherein, t(μ) s ) represents the atmospheric downdraft transmittance of light reaching the target object, and τ represents the atmospheric attenuation coefficient; By preserving the transport mapping in the inverse mapping model, we obtain a mapping model that maps the shadow image to the image with initial shadow removal: I sf (x,y)=I s (x,y)·ψ -1 (x,y) Several dilation operations are performed on the initial shadow detection image; the original image is subtracted from the dilated image to obtain the contour image of the initial shadow detection image; the pixels within the shadow region and the pixels within the non-shadow region of the contour image are fitted using the least squares method, thereby comparing and learning between the shadow region and the surrounding non-shadow region to obtain t(μ). s ) and τ; t(μ) s The initial shadow removal image is obtained by substituting τ into the mapping model.

4. The public transport safety early warning method supporting road condition shadow removal according to claim 1, characterized in that: Solving the total variation model to calculate the intrinsic reflectance image involves simplifying the total variation model and transforming its non-convex optimization problem into a system of linear equations. Among them, v S(I) To convert a matrix into a one-dimensional vector, C x(y) It is the Topletz matrix with the forward difference gradient operator, U x(y) W x(y) It is u x(y) w x(y) The rewritten diagonal matrix The problem of calculating a system of linear equations yields an intrinsic reflectance image that reflects the essential color attribute.

5. The public transport safety early warning method supporting road condition shadow removal according to claim 4, characterized in that: Constructing edge contour verification image pairs from the original image, combined with the prior binary contour structure of the generated shadow region, the specific steps are as follows: Constructing edge contour verification image pairs: Convert the original image to a grayscale image, denoted as I. gray ; The Canny operator is used to extract edges from the initially removed image; the extracted image is then subjected to Gaussian filtering to remove interfering details and impurities, resulting in a binary edge contour image. Let the binary edge contour image be C. gt Randomly generate an irregular mask, denoted as M; cover the grayscale image I with the irregular mask M. gray Available above Where ⊙ represents the Hadama product; grayscale image after masking The binary edge contour image is C gt An irregular mask M is input to an edge prediction network G for training, resulting in a predicted contour completion image C. pred :

6. The public transport safety early warning method supporting road condition shadow removal according to claim 5, characterized in that: The edge prediction network includes a generator and a discriminator; the generator includes an encoder, a decoder, and several residual blocks; the discriminator includes several convolutional layers, which are connected by the LeakyReLu function, and finally output using the sigmoid function. grayscale image after masking The binary edge contour image is C gt When training the irregular mask M input edge prediction network, the input data is fed into the generator: The encoder performs three rows of mirror padding around the matrix formed by the input data; the padded data is then input into three sequentially connected sequence models to represent the input data in a lower-dimensional form and extract the key features of the data; each sequence model contains a two-dimensional convolution, an instance normalization layer, and a ReLU activation function; Low-dimensional data is input into several connected residual blocks; the convolution of the residual blocks uses dilated convolution. The output of several connected residual blocks is input to the decoder for decoding to obtain the image generated by the generator; the decoder is upsampled through two sequence models; the sequence model includes a two-dimensional transposed convolution, an instance normalization layer, a ReLU activation function, a mirror padding block, a two-dimensional convolution, and a sigmoid function layer connected in sequence. The generated image is input into the discriminator; The edge prediction network is trained using adversarial loss and feature loss functions; The trained edge prediction network outputs the prediction result C. pred : in, This is the output of the edge prediction network, where mask is the shadow mask, and Z is the value of Z. edg This is a binary contour image of the original road condition image.

7. The public transport safety early warning method supporting road condition shadow removal according to claim 6, characterized in that: Generative Adversarial Networks (GANs) including detection and shadow removal models are constructed based on color-guided templates and edge-guided templates. Specifically, two GANs with the same architecture are constructed, with the first GAN serving as the detection model and the second GAN serving as the shadow removal model. Both the detection and adversarial models include generators and discriminators. The discriminator of the GANs uses the PatchGAN conditional discriminator. The original image and the preliminary shadow detection image are input into the detection model to obtain the detection results output by the discriminator. Based on the detection results, the original image, the preliminary shadow detection image, and the preliminary shadow removal image are input into the shadow removal model to obtain the road condition shadow removal image. The specific steps are as follows: The original image is input into the generator of the detection model; the image generated by the generator is combined with the original image and the preliminary shadow detection image, and then input into the discriminator of the detection model. The discriminator, based on the color-guided template and the edge-guided template, assumes that the removed image has prior information on color and edge, and judges the authenticity of the image generated by the generator, thereby filtering the image generated by the generator. The image generated by the filtered generator is input into the generator of the shadow removal model. The image generated by the generator is combined with the original image, the preliminary shadow detection image, and the preliminary shadow removal image. After channel connection, the images are input into the shadow removal model for discrimination, to identify the authenticity of the generated image and to pay attention to the correspondence of the image information positions. Train a generative adversarial network; output a road condition shadow removal image using the shadow removal model of the trained generative adversarial network.

8. The public transport safety early warning method supporting road condition shadow removal according to claim 7, characterized in that: The loss function used when training a generative adversarial network is as follows: L total =L chroma +L Mix +L possion Among them, L chroma For color consistency loss, L Mix For structure-aware loss, L possion For Poisson guiding loss; specifically: Where c∈{r,g,b} represents a corresponding color channel of the image. These are the pixel values ​​of the corresponding color channels in the generated image and the brightened image, respectively; in, in, Where the closed subset S is the image domain, and the closed subset Ω of S represents the shadow region to be reconstructed, f * For a known scalar function defined on and outside the Ω domain, it represents known information about the unshaded region.

9. A public transport safety early warning system supporting road condition shadow removal, characterized in that: The method for implementing the method as described in any one of claims 1 to 8 includes an image extraction module, a priori detection module, a microscopic removal module, and a safety warning module connected in sequence; the priori detection module includes a preliminary shadow detection component, a preliminary shadow removal component, a color-guided template component, and an edge-guided template component connected in sequence. The image extraction module is used to acquire the original image; The preliminary detection component is used to input the original image into the HSI space and construct the shadow recognition index SI; the maximum inter-class variance method is used to segment the SI to extract the shadow region for preliminary coarse recognition, thereby obtaining the preliminary shadow detection image; The aforementioned preliminary shadow removal component is used to calculate the mapping relationship between the preliminary shadow detection image and the image without shadows, and obtain the shadow image and the preliminary shadow removal image based on the mapping relationship; The color-guided template component is used to intrinsically decompose the shadow preliminary detection image to obtain a reflectance intrinsic image that reflects essential color information; Use the intrinsic reflectance image as a color-guided template; The edge-guided template component is used to construct an edge contour verification image pair of the original image, and combined with the prior binary contour structure of the generated shadow region, to obtain a structural skeleton as an edge-guided template. The microscopic removal module is used to construct a generative adversarial network that includes a detection model and a shadow removal model; Based on color-guided templates and edge-guided templates, the original image and the preliminary shadow detection image are input into the detection model to obtain the detection results output by the discriminator; Based on the detection results, the original image, the preliminary shadow detection image, and the preliminary shadow removal image are input into the shadow removal model to obtain the road condition shadow removal image; The aforementioned safety warning module is used to provide public transportation safety warnings based on road condition shadow removal images.

Images