Hybrid network parking space state detection method based on CNN and ViT

Abstract

The invention provides a hybrid network parking space state detection method based on CNN and ViT. The method comprises the following steps: constructing a split residual network; reconstructing an embedded layer of the ViT model based on the split residual network to obtain a hybrid network model; obtaining parking space image data sets of the parking lot under different scene conditions, and adding a new classification from the parking space image data sets to screen out a training data set and a verification data set; training the hybrid network model based on the training data set; verifying the trained hybrid network model based on the verification data set to obtain a verified hybrid network model; and inputting a parking lot image of which the parking space state is to be detected into the verified hybrid network model for detection to obtain parking space state information. According to the method, the inductive bias characteristic of the CNN is introduced to overcome the limitation of the ViT model on a small data set, and the advantage that the ViT model captures the global characteristics of the image reduces the misjudgment probability of the algorithm on the shielded parking space. Therefore, the real-time performance, efficiency and accuracy of parking space state detection are improved.

Description

Technical Field

[0001] The present application relates to the technical field of parking space status detection, and in particular to a hybrid network parking space status detection method based on CNN and ViT. Background Art

[0002] With the development of economy and the continuous improvement of people's living standards, cars have become an essential means of transportation for every family. A large number of family cars have poured into cities, which has accelerated economic circulation and brought huge pressure to traffic. The problems of difficulty in finding parking spaces and parking spaces have also become the troubles of every car owner. According to the requirements of the "National Smooth Traffic Project", the number of motor vehicles and parking spaces in my country must reach 1:0.45. However, there are currently 131 million parking spaces in my country, and the ratio of motor vehicles to parking spaces is 1:0.39. There is still a gap of 125 million parking spaces, and the gap is expanding over time. With such a huge disparity between the number of vehicles and parking spaces, the average parking space utilization rate in my country's parking lots is 30%, and it shows the characteristics of "the larger the parking lot, the lower the utilization rate". Therefore, developing a scientific parking space detection method will become an effective means to solve the problem of parking difficulties.

[0003] At present, parking systems mainly rely on various sensors to identify the status of parking spaces. These include ground-sensing coil parking sensors, which are based on the principle of electromagnetic induction and judge the status of parking spaces based on the change in the inductance of the coil loop caused by the vehicle passing through the energized coil; electromagnetic wave or ultrasonic sensors, which judge the status of parking spaces based on the reflected signals of microwaves or ultrasonic waves; and geomagnetic sensors, which detect the status of parking spaces based on the principle that objects that can conduct current can disturb the surrounding magnetic field and the change in the disturbance of the earth's magnetic field caused by the vehicle nearby. These sensor technologies are relatively mature, but the cost of parking lot renovation, wiring, and sensor purchase and maintenance is relatively high. In contrast, using cameras to obtain image information and combining computer vision technology for parking space detection can significantly reduce the detection cost. This method is particularly suitable for outdoor parking lots, and the image information obtained by the camera can also be used for other services. Therefore, the parking space status detection method based on visual perception has a broader development prospect.

[0004] Early visual parking space detection technology used grayscale and denoising methods to process parking lot images, and determined the parking space status by judging the number of noise points in the parking area. With the development of deep learning, convolutional neural networks (CNNs) have shown great advantages in processing various image tasks. The application of image convolution has made a qualitative leap in visual parking space detection technology. There are currently two mainstream visual parking space detection methods:

[0005] (1) Identify the vehicle in the image through the target detection algorithm, calculate the intersection over union (IOU) between the vehicle position anchor frame and the pre-calibrated parking space frame, and determine whether there is a vehicle in the parking space by setting the IOU threshold. The accuracy of this method mainly depends on the target detection algorithm. At present, a new vehicle detection algorithm SMA-YOLO is proposed based on the YOLOv7 model and MobileNetV3 is used as a new feature extraction network to achieve model lightweight. In order to reduce the impact of vehicle size and shape differences on the detection effect, a vehicle detection algorithm based on a graph network is also proposed. This method has high detection accuracy, but the detection speed is slow and cannot meet the requirements of real-time detection. In addition, an adaptive multi-scale feature fusion network (AMFFN) is proposed to replace the feature fusion module in YOLOv4, thereby achieving better target detection performance. In order to make the vehicle detection network more flexible and meet the needs of different environments.

[0006] (2) Use image classification methods to determine the parking space status of the image. The parking lot image is cropped into block images with a pre-annotated mask, and each block image corresponds to only one parking space. The parking space labels in the block image are divided into two categories (occupied or vacant), and the parking space status is determined by image classification methods. In the prior art, a method based on bag of features (BOF) and support vector machine (SVM) is used to detect parking spaces. This method requires a small number of images to train the classifier and can accept non-rectangular and variable-sized images as input. At present, a lightweight convolutional neural network model called mAlexNet is proposed for judging the parking space status. The model has a small number of parameters and layers, and has achieved better results than traditional machine learning methods. However, there is still a lot of room for improvement in the recognition accuracy of occluded images. Based on mAlexNet, dilated convolution is added to construct a parking space detection network CarNet. The addition of dilated convolution increases the receptive field of the network, which plays a certain role in judging occluded parking spaces. As research continues to deepen, new challenges and demands have prompted researchers to explore more efficient network architectures. The Vision Transformer (ViT) proposed in the prior art has made its mark in the field of computer vision (CV) and opened up a new track for image classification tasks. ViT uses the self-attention mechanism to capture long-distance dependencies and obtain global context information, overcoming the shortcomings of traditional convolutional neural networks in feature extraction and significantly enhancing the accuracy of image classification. In combination with the ViT architecture, a global perceptual feature extraction module (GPFE) for binary classification is proposed, and the module is integrated into the convolutional neural network ResNet-50 to realize parking space status detection.

[0007] Therefore, the technical problem that urgently needs to be solved is: how to improve the real-time performance of parking space status detection, improve the efficiency of parking space status detection, and improve the accuracy of parking space status detection. Summary of the invention

[0008] The purpose of this application is to provide a XXX, and propose a hybrid network model PDViT. The embedding layer of the visual Transformer is reconstructed using a convolutional neural network, and the limitations of the ViT model on small data sets are overcome by introducing the inductive bias characteristics of CNN. The advantage of the ViT model in capturing the global features of the image reduces the probability of the algorithm misjudging the blocked parking space. Thereby improving the real-time, efficiency and accuracy of parking space status detection.

[0009] To achieve the above-mentioned objectives, the present application provides a hybrid network parking space status detection method based on CNN and ViT, the method comprising: constructing a split residual network; based on the split residual network, reconstructing the embedding layer of the ViT model to obtain a hybrid network model; obtaining a parking space image dataset of a parking lot under different scene conditions, adding a new classification to the parking space image dataset to filter out a training dataset and a verification dataset; training the hybrid network model based on the training dataset; verifying the trained hybrid network model based on the verification dataset to obtain a verified hybrid network model; inputting the parking lot image of the parking space status to be detected into the verified hybrid network model for detection to obtain parking space status information.

[0010] In the hybrid network parking space status detection method based on CNN and ViT as described above, constructing a split residual network includes: constructing a split residual module; and constructing a split residual network based on the split residual module.

[0011] In the hybrid network parking space status detection method based on CNN and ViT as described above, constructing a split residual module includes: calculating the total memory access of the convolutional layer, and obtaining the relationship between the number of input channels and the number of output channels of the convolutional layer corresponding to the minimum total memory access of the convolutional layer; using the relationship between the number of input channels and the number of output channels of the convolutional layer corresponding to the minimum total memory access of the convolutional layer as the criterion of the split residual module; using channel splitting to divide the feature map of the input split residual module into two parts, namely the left part and the right part; sending the left part to the stacker after convolution processing with the same number of input and output channels; and sending the right part to the stacker after adjusting the feature map size through the maximum pooling operation.

[0012] As described above, the hybrid network parking space status detection method based on CNN and ViT, wherein, based on the split residual module, constructing a split residual network includes: establishing the first convolutional layer of the split residual network; using a channel-stacked split residual module in the second layer of the split residual network; using a channel-superimposed split residual module in the third layer of the split residual network; adding a channel rearrangement mechanism between the channel-stacked split residual module and the channel-superimposed split residual module to disrupt the channel order of the second layer output of the split residual network.

[0013] As described above, the hybrid network parking space status detection method based on CNN and ViT, wherein, based on the split residual network, the embedding layer of the ViT model is reconstructed to obtain the hybrid network model, including: using the split residual network as the embedding layer of the hybrid network model; using the ViT encoder as the encoder layer of the hybrid network model; constructing the classification layer of the hybrid network model to obtain the hybrid network model.

[0014] As described above, the hybrid network parking space status detection method based on CNN and ViT, wherein the training method of the hybrid network model is as follows: the parking space image is input through the embedding layer, and the image features are extracted through the split residual network to obtain a feature map; the second and third dimensions of the feature map are flattened and transposed as a whole to obtain a vector; the obtained vector is combined with the position encoding vector of the same dimension and sent to the encoder; in the encoder layer, the encoder module uses the GELU activation function to improve the convergence speed of the training process.

[0015] As described above, the hybrid network parking space status detection method based on CNN and ViT, wherein the split residual network includes 3 convolutional layers; the first layer uses a 5x5 convolution kernel, and the second and third layers use split residual modules to extract features.

[0016] As described above, the hybrid network parking space status detection method based on CNN and ViT, wherein the hybrid network model includes an embedding layer, an encoder layer, and a classification layer.

[0017] In the hybrid network parking space status detection method based on CNN and ViT as described above, the feature map processed by CNN is used as the original input of the ViT encoder.

[0018] In the hybrid network parking space status detection method based on CNN and ViT as described above, the criterion for splitting the residual module is that the number of input channels and the number of output channels of the convolutional layer are the same.

[0019] The beneficial effects achieved by this application are as follows:

[0020] (1) This application designs a new convolutional network embedding layer in the ViT architecture to overcome the limitations of the ViT network on small datasets.

[0021] (2) The hybrid network model (PDViT) of this application combines the local feature extraction of CNN with the ability of Transformer to connect to the global context, further improving the performance of the network in complex scenarios, especially for occlusion problems.

[0022] (3) The hybrid network model (PDViT) of this application realizes efficient and accurate parking space status detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present application. For those skilled in the art, other drawings can also be obtained based on these drawings.

[0024] Figure 1This is a flow chart of a hybrid network parking space status detection method based on CNN and ViT according to an embodiment of the present application.

[0025] Figure 2 This is a flow chart of a method for constructing a split residual network according to an embodiment of the present application.

[0026] Figure 3 This is a flow chart of a method for reconstructing the embedding layer of the ViT model based on a split residual network to obtain a hybrid network model according to an embodiment of the present application.

[0027] Figure 4 A schematic diagram of a channel stacked split residual module according to an embodiment of the present application.

[0028] Figure 5 A schematic diagram of a split residual module for channel superposition according to an embodiment of the present application.

[0029] Figure 6 This is a structural diagram of a split residual network according to an embodiment of the present application. DETAILED DESCRIPTION

[0030] The following is a clear and complete description of the technical solutions in the embodiments of the present application in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present application.

[0031] like Figure 1 As shown, the present application provides a hybrid network parking space status detection method based on CNN and ViT, the method comprising the following steps:

[0032] Step S1, construct a split residual network.

[0033] It should be explained that in the Visual Transformer (ViT) network model, the embedding layer is only used as a preliminary processing module for the input image and does not extract image features. Therefore, this application uses a convolutional network (CNN) to reconstruct the embedding layer of the ViT model, so that the reconstructed network model can extract image features while processing the image and build a reference layer that connects the local feature relationship of the image. Improve the algorithm efficiency, detection efficiency and detection accuracy of the reconstructed network model.

[0034] like Figure 2 As shown, step S1 includes:

[0035] Step S110, constructing a split residual module.

[0036] It needs to be explained that using FLOPs (Floating Point Operations Per Second) as the only indicator for calculating network complexity is unreasonable and insufficient. When a computer performs floating-point operations, it must first load the data into the cache before it can proceed to the next step. The data loading process takes time. The larger the memory read MAC (Memory Access Cost) of the network, the more time it takes to load. The running speed of the network is not only determined by the network computing complexity FLOPs, but also depends on the size of the memory read. Therefore, this application calculates the total memory access of the convolutional layer, and obtains the relationship between the number of input channels and the number of output channels of the convolutional layer corresponding to the minimum total memory access of the convolutional layer.

[0037] Step S110 includes the following sub-steps:

[0038] Step S111, calculating the total memory access amount of the convolutional layer, and obtaining the relationship between the number of input channels and the number of output channels of the convolutional layer corresponding to the minimum value of the total memory access amount of the convolutional layer.

[0039] Among them, the calculation formula of convolution layer FLOPs (floating point operations) is as follows:

[0040] FLOPs = HW(2C in K 2 -1)C out ; (1)

[0041] Among them, H and W are the length and width of the output feature map, respectively, and C in , C out are the number of input channels and output channels respectively, and K is the convolution kernel size.

[0042] Among them, 2C in K 2 -1=(C in K 2 )+(C in K 2 -1); (2)

[0043] The first term on the right side of formula (2) is the number of multiplication operations, and the second term is the number of addition operations. Usually FLOPs (floating point operations) only consider the number of multiplication operations, so formula (1) can be simplified to:

[0044] FLOPs=HW(C in K 2 )C out ; (3)

[0045] As a specific embodiment of the present invention, the calculation formula for the total memory access amount of the convolutional layer is as follows:

[0046] MAC=H in W in C in +H out W out C out +K 2 C in C out ; (4)

[0047] Among them, MAC represents the total memory access of the convolutional layer; H in W in C in , H out W out C out , K 2 C in C out They represent the memory access amount of the input feature map, the memory access amount of the output feature map, and the memory access amount of the convolution kernel respectively.

[0048] When the output feature map size is the same as the input feature map size, formula (4) can be simplified to:

[0049] MAC=HW(C in +C out )+K 2 C in C out ; (5)

[0050] Substituting formula (3) into formula (5), we obtain formula (6) as follows:

[0051]

[0052] According to the classic inequality chain (harmonic mean ≤ geometric mean ≤ arithmetic mean ≤ square mean):

[0053]

[0054] Simplifying the first two terms in equation (7) yields:

[0055]

[0056] Among them, a and b represent parameters.

[0057] Replace a and b in formula (8) with C in , C out Substituting into formula (6) we get:

[0058]

[0059] According to formula (9), if and only if C in =Cout When the above formula takes an equal sign, that is, when the number of input channels of the convolution is equal to the number of output channels, the MAC memory access amount reaches the minimum value. That is, when the convolution layer uses the same number of input and output channels, the memory access amount is the minimum, and the running speed is the fastest under the same amount of calculation.

[0060] Step S112, using the relationship between the number of input channels and the number of output channels of the convolutional layer corresponding to the minimum total memory access amount of the convolutional layer as a criterion for splitting the residual module.

[0061] This application proposes a split residual module based on the same input and output channel number criterion (that is, the number of input channels and the number of output channels are the same), which enhances the feature extraction capability of the network while ensuring the inference speed.

[0062] Step S113, using channel splitting method, divides the feature map of the input split residual module into two parts, the two parts are a left part and a right part.

[0063] In step S114, the left part is sent to the stacker after being processed by convolution with the same number of input and output channels; the right part is sent to the stacker after the feature map size is adjusted through the maximum pooling operation.

[0064] Among them, the stacked channels have the original shallow features and the deep features after convolution processing, which need to be output through batch normalization (Batch Normalization, BN) to obtain the results after processing by the split residual module.

[0065] As a specific embodiment of the present invention, the split residual module is divided into two types, and the difference between the two types lies in the final channel fusion method. The first type is channel stacking, which directly stacks the channels of the two outputs together without calculation; the second type is channel superposition, which accumulates the channels of the two outputs element by element, which requires calculation, and the final number of channels will be halved.

[0066] Step S120: construct a split residual network based on the split residual module.

[0067] It should be explained that a large number of parameters and complex structures are not suitable for feature extraction in the ViT embedding layer. A deeper convolutional network will ignore the shallow details of the image and generate a large number of redundant features, which will increase the difficulty of model training and pose the risk of overfitting. Therefore, this application constructs a split residual network to extract local features of the image in the ViT embedding layer, so that the network can combine local and global features to judge the status of the parking space image.

[0068] As a specific embodiment of the present invention, the split residual network includes 3 convolutional layers. The first layer uses a 5x5 convolution kernel to obtain a larger receptive field and improve the network's ability to perceive global information. The second and third layers use split residual modules to extract features and fully integrate the shallow features of the image with the deep features.

[0069] Step S120 includes the following sub-steps:

[0070] Step S121, establishing the first convolutional layer of the split residual network.

[0071] The first convolution layer uses a 5x 5 convolution kernel and the input is a 3x 224x 224 RGB image. In the first convolution layer, a 2D convolution with 32 channels and a stride of 4 is used, and a ReLU activation function is connected. The maximum pooling operation with a size of 3 and a stride of 2 is used to reduce the redundant information in the initial image, and the result is sent to the subsequent module for further feature extraction.

[0072] Step S122, using a split residual module with channel stacking in the second layer of the split residual network.

[0073] Specifically, a channel-stacked split residual module is used in the second layer of the split residual network, and the output 32-channel feature map contains 16-channel shallow features and 16-channel deep features.

[0074] like Figure 4 , which is a schematic diagram of a channel-stacked split residual module. The channel-stacked split residual module includes a channel splitting layer, which is respectively connected to a convolutional layer and a maximum pooling layer; the convolutional layer is connected to a Relu activation layer, and the Relu activation layer and the maximum pooling layer are connected to a stacked layer. Figure 4 BN refers to batch normalization.

[0075] Step S123, using a split residual module with channel superposition in the third layer of the split residual network.

[0076] Specifically, a split residual module with channel superposition is used in the third layer of the split residual network to fuse different features and output 16-channel fused features.

[0077] Step S124, adding a channel rearrangement mechanism between the channel-stacked split residual module and the channel-superimposed split residual module to disrupt the channel order of the second layer output of the split residual network.

[0078] like Figure 5 , which is a schematic diagram of a split residual module with channel superposition. The split residual module with channel superposition includes a channel splitting layer, which is respectively connected to a convolutional layer and a maximum pooling layer; the convolutional layer is connected to a Relu activation layer, and the Relu activation layer and the maximum pooling layer are connected to the superposition layer. Figure 5 BN refers to batch normalization.

[0079] Specifically, the split residual network adds a channel reordering mechanism between the two split residual modules, disrupting the channel order of the second layer output, so that the third layer module can extract richer image features. In the split residual network, the number of channels of the convolution input and output is ensured to be equal to obtain the minimum memory access and improve the network operation speed.

[0080] like Figure 6 The figure shows the schematic diagram of the architecture of the split residual network. The architecture of the split residual network includes sequentially connected convolutional layers, channel-stacked split residual modules, and channel-superimposed split residual modules, wherein a channel rearrangement mechanism is added between the channel-stacked split residual modules and the channel-superimposed split residual modules. The parking space image is input from the convolutional layer, after Relu+max pooling operation, input to the channel-superimposed split residual module, after channel rearrangement operation, input to the channel-superimposed split residual module.

[0081] It should be explained that the feature map without channel rearrangement is split into two different channels. The left side is further extracted by convolution, and the right side is retained by maximum pooling. This causes some features after the first convolution to undergo two maximum poolings, so that the final output loses the features after the second convolution. As a result, the maximum pooling in the third layer is invalid. To avoid the above phenomenon, the network adds a channel rearrangement operation between the two split residual modules.

[0082] It should be explained that the features of the maximum pooling include the results of the first and second convolutions, and the final output also includes the features after each convolution. Subsequent experiments have shown that the addition of channel rearrangement operations can enhance the network's ability to extract features from images and help improve the network's detection accuracy.

[0083] Step S2: Based on the split residual network, the embedding layer of the ViT model is reconstructed to obtain a hybrid network model.

[0084] It should be explained that the Visual Transformer model (ViT model) is a model that applies the Transformer architecture to computer vision tasks. The ViT model consists of an embedding layer, a Transformer encoder layer, and a classification layer. Unlike convolutional neural networks, the ViT model captures long-range dependencies in images through a self-attention mechanism and establishes a global understanding of each part of the input image. Compared with convolutional neural networks that extract local features, the ViT model can focus on different parts of the image at the same time, effectively capture global features and contextual relationships, and thus better process diverse visual data. However, the ViT model lacks the inductive bias (translation invariance and locality) inherent in CNN and lacks prior knowledge in feature learning. When faced with classification tasks with a small amount of data, the performance of the ViT model is often inferior to that of CNN. This section will redesign the embedding layer of the ViT model to build a hybrid network model that combines the characteristics of CNN and ViT. This model makes up for the shortcomings of CNN in dealing with occlusion problems and overcomes the limitations of the ViT model on small data sets.

[0085] It should be explained that the ViT model adds an embedding layer before the Transformer encoder to convert the two-dimensional image into a one-dimensional sequence. In the embedding layer, the input image is first divided into image blocks of fixed size, and then each image block is flattened and mapped into a one-dimensional vector, and the corresponding position code is added to each vector, finally realizing the conversion of the image into a sequence feature that the Transformer can process. Since the Transformer model itself does not have the ability to process the position information of elements in the sequence, ViT introduces position coding into the model to help the model understand the positional relationship between different image blocks. Among them, the sequence corresponding to sequence number 0 is a sequence specifically used for classification.

[0086] Among them, the hybrid network model is named PDViT.

[0087] The hybrid network model includes an embedding layer, an encoder layer, and a classification layer, wherein the embedding layer is reconstructed by a split residual network.

[0088] like Figure 3 As shown, step S2 includes:

[0089] Step S210, using the split residual network as an embedding layer of the hybrid network model.

[0090] First, the parking space image is input into the embedding layer, and the image features are extracted through the split residual network to convert the 3x 224x 224 RGB image into a 256x 14x 14 feature map.

[0091] Step S220, using the ViT encoder as the encoder layer of the hybrid network model.

[0092] Specifically, the ViT encoder (i.e., Transformer encoder) is used as the encoder layer of the hybrid network model. The feature map processed by the split residual network is used as the original input of the ViT encoder (i.e., Transformer encoder). In other words, the feature map processed by CNN is used as the original input of the ViT encoder. Using the feature map processed by CNN as the original input of the ViT encoder not only reduces the amount of data and reduces the computational complexity, but also retains the local features and spatial information of the image, making ViT more efficient in capturing long-distance dependencies and global context.

[0093] As a specific embodiment of the present invention, the sequence used for classification in the original ViT is deleted, and the classification result is obtained by using global average pooling on the sequence output by the encoder.

[0094] Step S230, constructing a classification layer of the hybrid network model to obtain the hybrid network model.

[0095] As a specific embodiment of the present invention, the training method of the hybrid network model is as follows:

[0096] Step T1: Input the parking space image through the embedding layer, extract the image features through the split residual network, and obtain the feature map.

[0097] Specifically, the parking space image is input into the embedding layer, and the image features are extracted through the split residual network to convert the 3x 224x224 RGB image into a 256x 14x 14 feature map.

[0098] In step T2, the second and third dimensions of the feature map are flattened and transposed as a whole to obtain a 196x256 vector.

[0099] Specifically, the second and third dimensions of the feature map are flattened using the flatten operation (the flatten operation refers to the process of converting a multidimensional array into a one-dimensional array), and the entire map is transposed to obtain a 196x 256 vector.

[0100] In step T3, the vector obtained in step T2 is combined with the position encoding vector of the same dimension and then fed into the Transformer encoder.

[0101] Specifically, the vector obtained in step T2 is combined with the position encoding vector of the same dimension and then fed into the Transformer encoder. The introduction of position encoding enables the network to obtain the relative positions between image features, which is particularly important for processing images with occlusion and complex backgrounds.

[0102] Step T4, in the encoder layer, the encoder module uses the GELU activation function to improve the convergence speed of the training process.

[0103] The GELU activation function is:

[0104] Where GELU(x) represents the GELU activation function; x is the input value, X is a Gaussian random variable with zero mean and unit variance, P(X<=x) is the probability that X is less than or equal to the given value x. erf(x) is the Gaussian error function.

[0105] The encoder module loops ten times to focus on different parts of the input feature map.

[0106] It should be explained that the hybrid network model combines the split residual network and the Transformer architecture. PDViT achieves higher parking space detection accuracy without increasing the computational complexity, especially when dealing with occluded parking spaces. In the early stage of training, PDViT shows the fast convergence characteristics of CNN, and in the later stage, it shows the high accuracy of Transformer.

[0107] Step S3, obtaining a parking space image dataset of a parking lot under different scene conditions, and adding a new classification to the parking space image dataset to filter out a training dataset and a verification dataset.

[0108] It should be explained that this application uses two public parking space image datasets to verify the hybrid network model of this application, aiming to evaluate the performance of the method proposed in this invention in the parking space detection task, including detection accuracy, speed, generalization, etc.

[0109] The first parking space image dataset (PKLot dataset) includes 12,417 parking area images and 695,853 parking space images cropped from them. The first parking space image dataset uses three independent cameras to capture images of two parking lots at different angles on sunny, rainy, and cloudy days, and is saved in JPEG color format with a resolution of 1280×720 pixels.

[0110] Among them, all cropped parking space images are divided into three subsets and named UFPR 04, UFPR 05 and PUCPR. Among them, UFPR04 and UFPR05 are image sets of the same parking lot from different perspectives, respectively taken by cameras deployed on the 4th and 5th floors of the UFPR building, and PUCPR is taken by a camera deployed on the roof of the 10th floor of the PUCPR administrative building. The three subsets are further divided into cloudy, sunny and rainy days.

[0111] The second parking space image dataset (CNRPark dataset) is another parking lot visual occupancy detection dataset. There are 144,965 square images with a resolution of 150x150. Nine cameras with different perspectives are used to record the same parking space images under different weather conditions. This dataset contains occluded images caused by obstacles such as lamp posts, trees, pedestrians and other cars, as well as exposed and dim images caused by light intensity. The dataset covers almost all situations encountered in real-life parking space detection. The second parking space image dataset is divided into three subsets according to various weather conditions: sunny, rainy and cloudy, but each subset is only divided according to time, without subdividing the parking space status. The second parking space image dataset replaces the time subset with the parking space status, and the final subset division is unified with the first parking space image dataset. In order to test the effect of the network on occluded parking spaces, all occluded parking space images in the dataset are filtered and made into a new subset called "Sheltered". Similarly, all dark light images are filtered out to create a new subset named "Dark" for subsequent testing.

[0112] Among them, the statistics of parking space occupancy and vacancy in different subsets of the second parking space image dataset (CNRPark dataset) are shown in Table 1:

[0113] Table 1, parking space statistics of CNRPark dataset:

[0114]

[0115] Among them, Rain, Overcast, and Sunny are the original subsets of the second parking space image dataset; Dark, Sheltered, and Dataset are new subsets derived from the three original subsets of Rain, Overcast, and Sunny. Dark is used to explore the impact of the network on light intensity (mainly used in Chapter 4); Sheltered is used to evaluate the network's ability to extract global features; Dataset is used to detect the classification performance of PDViT when facing small data sets, and only contains 2% of the parking space images in the CNRPark dataset.

[0116] The three subsets of the first parking space image dataset contain parking space images from two different parking lots, which can be used to test the generalization performance of the network through subset cross-validation. The second parking space image dataset contains more complex parking space situations, which can be used to evaluate the performance of the network in processing complex parking space images, such as occlusion, exposure, and dimness.

[0117] Step S4, training the hybrid network model based on the training data set.

[0118] As a specific embodiment of the present invention, the PyTorch deep learning framework (open source deep learning framework) is used to train the model. The Adam optimizer is used. The batch size is set to 256, the learning rate is set to 0.001, and 50 rounds of training are completed. When the training Loss curve is close to smooth, the model is considered to be basically converged.

[0119] Step S5, verifying the trained hybrid network model based on the verification data set to obtain a verified hybrid network model.

[0120] As a specific embodiment of the present invention, the performance of the split residual network is verified, and the specific method is as follows:

[0121] In order to verify the performance of the split residual network, two fully connected layers were added to the end to output the image classification results, and compared with the mainstream image classification networks (VGG16, ResNet50), lightweight networks (MobileNetV3, ShuffleNetV2) and parking space status detection network mAlexnet. Since the first four networks were designed for 1000 classifications, the number of neurons in the network output was changed before the experiment to make it suitable for parking space detection. The network parameters, calculation amount, verification accuracy and inference speed were used as indicators to judge the network. The first parking space image dataset with a large amount of data (PKLot dataset) was used, and one of its subsets UFPR 04, UFPR 05 and PUCPR was selected as the training set, and the other two were test sets. A total of three groups of experiments were conducted, each group of experiments was trained three times, and the model weight with the best training results was selected for verification on the test set. The accuracy test was also repeated three times, and the average of the three results was taken as the final result. Since there are many repeated images in each subset, the verification results of the subset divided into training set and test set are not representative. The use of subset cross-validation can not only eliminate the contingency of internal verification of the subset, but also test the generalization performance of the network.

[0122] Among them, the CNN network parameter comparison table is shown in Table 2, and the PKLot (the first parking space image dataset) subset cross-validation accuracy comparison is shown in Table 3.

[0123] Table 2, CNN network parameter comparison table:

[0124]

[0125] Table 3, cross-validation accuracy comparison of PKLot (the first parking space image dataset):

[0126]

[0127] The data in Tables 2 and 3 above show that in the task of parking space status detection, networks with large computational load and memory usage, such as VGG16 and ResNet50, did not achieve higher detection accuracy. Instead, network models with fewer parameters and smaller size performed better. The parameters, computational load, and memory usage of the split residual network are 0.063M, 0.008G, and 0.88MB, respectively, which are less than 1% of VGG16. The inference time is 0.002s on both GPU and CPU. Improving hardware performance does not speed up the inference speed. This shows that the split residual network has approached the limit of inference speed. The inference speed on the CPU is 100 times faster than VGG16, 10 times faster than MobileNetV3 and ShuffleNetV2, and is the fastest network in Table 2. In the three validation subset accuracy tests of PKLot (the first parking space image dataset), the average accuracy of the split residual network reached more than 90%, namely 95.45% (PUCPR), 93.92% (UFPR04), and 94.92% (UFPR05). The overall average accuracy was 7.54%, 2.09%, and 7.38% higher than that of MobileNetV3, ShuffleNetV2, and mAlexNet, respectively. The structural design of the split residual network reduces the model size and improves the reasoning speed without losing detection accuracy. In the task of parking space status detection, it achieves the feature extraction capability of MobileNetV3 and ShuffleNetV2, while the number of parameters and calculations is less than 5% of the two. In summary, the split residual network is more suitable as the infrastructure of the embedding layer of the hybrid network model (PDViT).

[0128] As a specific embodiment of the present invention, the channel rearrangement is verified, and the specific method is as follows:

[0129] The structure of split residual network plus fully connected layer is used to verify the actual impact of channel rearrangement on network performance. In order to ensure the reliability of the experimental results, the experiment verifies all subsets in PKLot (the first parking space image dataset) and CNRPark (the second parking space image dataset). The test results are shown in Table 4.

[0130] Table 4, channel rearrangement verification table:

[0131]

[0132] The data in Table 4 above show that in both datasets, the split residual network with channel rearrangement achieved a 1% to 2% performance improvement on each subset. Among them, the improvement on the "Sheltered" subset is particularly obvious, from 75.66% to 83.95%, an increase of 8%. The experimental results show that channel rearrangement can indeed enhance the expressiveness of network features, thereby effectively improving detection accuracy, and it has shown a significant effect on the judgment of blocked parking spaces.

[0133] Due to the limitations of shooting angles and environments, parking space images captured by cameras are often blocked, either by trees, pedestrians, or adjacent vehicles. Therefore, the blocked parking spaces are verified.

[0134] As a specific embodiment of the present invention, the blocked parking space is verified, and the specific verification method is as follows:

[0135] A subset of the CNRPark dataset was selected as the training set, and the occluded subset was used as the test set to evaluate the performance of different networks in processing occluded parking space images.

[0136] Table 5, occluded parking space image test:

[0137]

[0138] As shown in Table 5, the test accuracy of VGG16, ResNet50, MobileNetV3 and ShuffleNetV2 is significantly reduced when dealing with occluded parking spaces. Although the sliding window convolution of the CNN network can quickly extract image features, it only processes local windows each time, resulting in insufficient control of global features. CNN has inherent limitations when dealing with problems such as occlusion that require the combination of global features. In contrast, PDViT integrates the ViT architecture based on the split residual network, which can capture long-distance dependencies and global context information, so that it can improve the test accuracy of occluded parking spaces while increasing a small amount of inference time, and finally reached 87.78%, with a single inference speed of 0.008 seconds. Compared with MobileNetV3 and ShuffleNetV2, the inference speed is 0.001 seconds slower, and the test accuracy is improved by 7% and 6%, respectively.

[0139] As a specific embodiment of the present invention, this application reconstructs the embedding layer of ViT based on the split residual network, so that it combines the advantages of CNN and ViT networks, and exhibits inductive bias characteristics similar to CNN on small data sets, while reflecting the high precision advantage of ViT when there is sufficient training data. The performance of PDViT will be verified by the Sunny subset with a larger amount of data in CNRPark (a total of 62,925 pictures) and the Dataset subset with a smaller amount of data (a total of 2,790 pictures). The networks involved in the comparison in this experiment are ViT, the hybrid network LeViT, and the hybrid network model PDViT of the present invention. The experimental results refer to Table 6.

[0140] Table 6, network performance comparison of the hybrid network model PDViT and ViT, and the hybrid network LeViT of the present invention:

[0141]

[0142] According to the data in Table 6, ViT uses Dataset as the training set, and the test accuracy on the four validation subsets (Rain, Overcast, Sunny, Sheltered) is 82.09%, 82.93%, 84.47% and 80.60% respectively. The test accuracy of ViT is 5% to 15% lower than that of other networks, which shows that the performance of ViT is limited when the number of training samples is small. Under the same training set, the subset validation accuracy of PDViT (hybrid network model) is 92.07%, 93.22%, 91.99%, 87.78% respectively, which is 9% higher than ViT on average, and compared with LeViT, which is also a hybrid network, the accuracy is improved by 2% on average. When using the Sunny subset with a larger amount of data for training (to avoid contingency, only the Rain, Overcast, and Sheltered subsets are used as validation sets), the validation accuracy of the Rain subset and Overcast subset of the four networks in Table 6 is relatively close, and the main difference appears on the Sheltered subset. PDViT outperforms ViT and LeViT on this subset. PDViT combines the characteristics of CNN inductive bias, enabling it to gradually build more complex feature representations from limited features, effectively overcoming the limitations of the ViT network on small data sets. On large data sets, PDViT relies on its ability to model global context information and achieves higher detection accuracy on each validation subset, especially on the occlusion subset, with a detection accuracy of 96.88%, reflecting its superiority in dealing with occlusion problems.

[0143] Step S6: input the parking lot image of the parking space status to be detected into the verified hybrid network model for detection to obtain parking space status information.

[0144] This application designs a new convolutional network embedding layer in the ViT architecture, aiming to combine the advantages of CNN and Transformer to overcome the limitations of the ViT network on small data sets. PDViT combines the local feature extraction of CNN with the ability of Transformer to connect to the global context, further improving the performance of the network in complex scenarios, especially for occlusion problems. Experimental results show that PDViT has an accuracy of up to 96% in the verification of occluded parking space images, far exceeding the performance of traditional CNN networks. In practical applications, the hybrid network model PDViT provides strong support for efficient and accurate parking space status detection.

[0145] The present application also provides a computer storage medium, wherein the computer storage medium stores computer instructions, and when the computer instructions are called, they are used to execute the address mapping method of the large-capacity solid-state hard disk. The computer storage medium contains one or more program instructions, and the one or more program instructions are used by the processor to execute a hybrid network parking space status detection method based on CNN and ViT.

[0146] The embodiment disclosed in the present invention provides a computer-readable storage medium, in which computer program instructions are stored. When the computer program instructions are executed on a computer, the computer executes the above-mentioned hybrid network parking space status detection method based on CNN and ViT.

[0147] An embodiment of the present invention provides a processor for processing the above-mentioned hybrid network parking space status detection method based on CNN and ViT.

[0148] In the embodiment of the present invention, the processor may be an integrated circuit chip having the ability to process signals. The processor may be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0149] The methods, steps and logic block diagrams disclosed in the embodiments of the present invention can be implemented or executed. The general processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in the embodiments of the present invention can be directly embodied as a hardware decoding processor for execution, or can be executed by a combination of hardware and software modules in the decoding processor. The software module can be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The processor reads the information in the storage medium and completes the steps of the above method in combination with its hardware.

[0150] The storage medium may be a memory, which may be, for example, a volatile memory or a nonvolatile memory, or may include both volatile and nonvolatile memory.

[0151] Among them, the non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory can be a random access memory (RAM), which is used as an external cache. By way of example but not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM) and direct RAM bus random access memory (DRRAM).

[0152] The beneficial effects achieved by this application are as follows:

[0153] (1) This application designs a new convolutional network embedding layer in the ViT architecture to overcome the limitations of the ViT network on small datasets.

[0154] (2) The hybrid network model (PDViT) of this application combines the local feature extraction of CNN with the ability of Transformer to connect to the global context, further improving the performance of the network in complex scenarios, especially for occlusion problems.

[0155] (3) The hybrid network model (PDViT) of this application realizes efficient and accurate parking space status detection.

[0156] In the description of this application, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features. In the description of this application, "plurality" means two or more, unless otherwise clearly and specifically defined.

[0157] In the description of the present application, the word "for example" is used to mean "used as an example, illustration or explanation". Any embodiment described as "for example" in the present application is not necessarily to be construed as being more preferred or advantageous than other embodiments. The following description is given to enable any technician in the field to implement and use the present invention. In the following description, details are listed for the purpose of explanation. It should be understood that a person of ordinary skill in the art can recognize that the present invention can be implemented without using these specific details. In other examples, well-known structures and processes will not be elaborated in detail to avoid obscuring the description of the present invention with unnecessary details. Therefore, the present invention is not intended to be limited to the embodiments shown, but is consistent with the widest scope consistent with the principles and features disclosed in the present application.

[0158] The above description is only an embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent substitution, improvement, etc. made within the spirit and principle of the present invention should be included in the scope of the claims of the present invention.

Claims

1. A hybrid network parking space status detection method based on CNN and ViT, characterized in that: The method includes: Construct a split residual network; Based on the split residual network, the embedding layer of the ViT model is reconstructed to obtain a hybrid network model; Obtain parking space image datasets under different scene conditions, add new classifications to the parking space image dataset to filter out training datasets and verification datasets; Training the hybrid network model based on the training data set; Verify the trained hybrid network model based on the verification data set to obtain a verified hybrid network model; The parking lot image with the parking space status to be detected is input into the verified hybrid network model for detection to obtain the parking space status information.

2. The hybrid network parking space status detection method based on CNN and ViT according to claim 1 is characterized in that: Building a split residual network includes: Construct a split residual module; Based on the split residual module, a split residual network is constructed.

3. The hybrid network parking space status detection method based on CNN and ViT according to claim 2 is characterized in that: Building a split residual module includes: Calculate the total memory access of the convolution layer, and obtain the relationship between the number of input channels and the number of output channels of the convolution layer when the total memory access of the convolution layer is the minimum; The relationship between the number of input channels and the number of output channels of the convolutional layer corresponding to the minimum total memory access of the convolutional layer is used as the criterion for splitting the residual module; By using channel splitting, the feature map of the input split residual module is divided into two parts, the left part and the right part; The left part is sent to the stacker after convolution processing with the same number of input and output channels; the right part is sent to the stacker after adjusting the feature map size through the maximum pooling operation.

4. The hybrid network parking space status detection method based on CNN and ViT according to claim 2 is characterized in that: Based on the split residual module, constructing a split residual network includes: Establish the first convolutional layer of the split residual network; A split residual module with channel stacking is used in the second layer of the split residual network; A split residual module with channel stacking is used in the third layer of the split residual network; A channel rearrangement mechanism is added between the channel-stacked split residual module and the channel-superimposed split residual module to disrupt the channel order of the second layer output of the split residual network.

5. The hybrid network parking space status detection method based on CNN and ViT according to claim 1 is characterized in that: Based on the split residual network, the embedding layer of the ViT model is reconstructed to obtain a hybrid network model including: Use the split residual network as the embedding layer of the hybrid network model; Use the ViT encoder as the encoder layer of the hybrid network model; Construct the classification layer of the hybrid network model to obtain the hybrid network model.

6. The hybrid network parking space status detection method based on CNN and ViT according to claim 1 is characterized in that: The training method of the hybrid network model is as follows: The parking space image is input through the embedding layer, and the image features are extracted through the split residual network to obtain the feature map; Flatten the second and third dimensions of the feature map and transpose it as a whole to get a vector; The obtained vector is combined with the positional encoding vector of the same dimension and fed into the encoder; In the encoder layer, the encoder module uses the GELU activation function to improve the convergence speed of the training process.

7. The hybrid network parking space status detection method based on CNN and ViT according to claim 4 is characterized in that: The split residual network consists of 3 convolutional layers; The first layer uses a 5x5 convolution kernel. The second and third layers use split residual modules to extract features.

8. The hybrid network parking space status detection method based on CNN and ViT according to claim 1 is characterized in that: The hybrid network model includes an embedding layer, an encoder layer, and a classification layer.

9. The hybrid network parking space status detection method based on CNN and ViT according to claim 5 is characterized in that: The feature map processed by CNN is used as the original input of the ViT encoder.

10. The hybrid network parking space status detection method based on CNN and ViT according to claim 3 is characterized in that: The criterion for splitting the residual module is that the number of input channels and output channels of the convolutional layer are the same.

Images