A Detection Method for Marked Parking Spaces Based on Semantic Feature Attention Fusion

Abstract

The present invention relates to the technical field of video image processing, and provides a method for detecting marked parking spaces based on semantic feature attention fusion. The method includes the steps of: a. using a backbone network to extract initial feature information of a parking space image; b. using a feature attention fusion module to perform feature fusion on the initial feature information to obtain fused features; c. using a Region Proposal Network (RPN) on the fused features to generate candidate region features; d. using an object detector to detect the candidate regions to directly obtain the four corner coordinate information of the marked parking space targets in the image. The method for detecting marked parking spaces based on semantic feature attention fusion provided by the embodiments of the present invention can effectively improve the average precision of marked parking space detection and has high robustness.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and particularly relates to a method for detecting marked parking spaces in parking lot images. Background Art

[0002] With the development of computer technology, automatic parking technology has also been developed. As one of the most critical technologies in automatic parking technology, the detection of marked parking spaces has become a very important research direction.

[0003] In recent years, with the development of image processing technology and deep learning, the image-based marked parking space detection technology has attracted much attention. The existing image-based marked parking space detection methods can be divided into two types according to the different types of image features used in the methods: The first is the marked parking space detection method based on the straight line features of parking spaces in images. Such methods extract the straight line features in parking lot images, and use a rule matching method to match the extracted straight line features and combine them into a complete marked parking space target. The second is the marked parking space detection method based on the corner point features of parking spaces in images. Such methods extract the corner point features in parking lot images, and use a rule matching method to match the extracted corner point features and combine them into a complete marked parking space target.

[0004] The disadvantages of the above-mentioned image-based marked parking space detection methods are as follows: The rule matching method for features is usually implemented through manual design, and the position and distance relationships of the features are written into the rule matching method in a hard-coded manner. For example, the marked parking spaces are matched and combined according to the corresponding position relationships and relative distances of four corner point features of parking spaces. The manually designed features of this type have poor adaptability. For example, when the resolution size of the input image changes, parameters such as distances in the manually designed rule matching method will become invalid; when complex parking space combination targets appear, parameters such as position correspondence relationships in the manually designed rule matching method will also become invalid. Summary of the Invention

[0005] In view of the above defects existing in the existing image-based detection method for marked parking spaces, the present invention provides a detection method for marked parking spaces based on semantic feature attention fusion to solve the problem of poor adaptability of the manually designed rule matching method. Aiming at the problem of low adaptability of the manually designed rule matching method, the process of the existing detection method for marked parking spaces is improved, and a detection method for marked parking spaces that does not rely on the rule matching method is proposed. In the last output part of the deep neural network of the method, parameters related to the four corner coordinates of the marked parking space target are directly output. After conversion, the four corner coordinate values of the marked parking space target are directly obtained. In addition, a feature fusion method called semantic feature attention fusion method is provided in this detection method to fuse the features of different hierarchical structures extracted by the backbone network to improve the final detection accuracy of the network.

[0006] To achieve the above object, a detection method for marked parking spaces based on semantic feature attention fusion is provided, including the following steps:

[0007] Step S1: Provide an image of a marked parking space to be detected, and extract the backbone network features of the image of the marked parking space to obtain corresponding mapped features.

[0008] Step S2: Build a semantic feature attention fusion network module, the network module includes a global average pooling layer and a convolutional layer; input the mapped features obtained in S1 into the semantic feature attention fusion network module to obtain fused features.

[0009] Step S3: Build an RPN network layer and an ROI layer, the RPN network includes a classification module and a regression module; input the fused features obtained in S2 into the RPN network to obtain preliminary candidate region boxes, and use the ROI layer to map the candidate region boxes and the fused features in S2 into candidate region features.

[0010] Step S4: Build a target detector, the target detector includes a fully connected layer and a prediction head; input the candidate region features obtained in S3 into the target detector to obtain the detection result of the marked parking space, and the form of the result is the four corner coordinate values of the marked parking space.

[0011] Optionally, in an embodiment of the present invention, in step S1, the ResNet101 network and a feature pyramid structure are used to extract multi-layer residual mapped features of the image, which are represented by R1-Conv1~5.

[0012] Optionally, in an embodiment of the present invention, there are 4 semantic feature attention fusion modules in the entire marked parking space detection network in step S2. Each semantic feature attention fusion module has two inputs. One input is R1-Conv-5, and the other input is R1-Conv-1 to 4 respectively in the four semantic feature attention fusion modules.

[0013] Optionally, in an embodiment of the present invention, the semantic feature attention fusion module in step S2 is composed of a convolutional layer, global average pooling, and an activation function, and can be expressed as:

[0014]

[0015] Where

[0016]

[0017] In the above formula, Y is R1-Conv-5, X is R1-Conv-1 to 4 respectively in the four semantic feature attention fusion modules, Z is the output feature in the semantic feature attention fusion module, σ1 is the Sigmoid activation function, σ2 is the ReLU activation function, represents the direct multiplication of the corresponding elements in the matrix, represents the matrix addition, represents a convolution operation, where the size of the convolution kernel is 3×3, and the number of channels changes from c to c / r, also represents a convolution operation, and the size of the convolution kernel is 1×1, and the number of channels changes from c / r to c. AvgPool represents global average pooling.

[0018] Optionally, in an embodiment of the present invention, the prediction head of the target detector in step S4 directly outputs parameters related to the four corner points of the marked parking space in the image. After conversion, the coordinate values of the four corner points can be directly obtained.

[0019] Optionally, in an embodiment of the present invention, the output parameters of the prediction head are x, y, w, h, α1, α2, α3, α4, where x and y are the center points of the rectangle, w and h are the width and height of the rectangle respectively. Through the above four parameters x, y, w, h, the four vertices of the predicted rectangle can be obtained. Let the four vertices of the rectangle be a, b, c, d respectively, and the coordinates of the four vertices a, b, c, d can be expressed as Let the center point of the rectangle be O, then the angle values of the lines connecting the four vertices a, b, c, d of the marked parking space line to the center point O are α1, α2, α3, α4 respectively.

[0020] The present invention proposes a new method for detecting marked parking spaces. The method flow proposed by the present invention does not rely on the feature rule matching method designed manually, has better robustness, and can adapt to the combination changes of the image size and the marked parking space targets in the parking lot image. The semantic feature attention fusion module proposed by the present invention can effectively fuse features at different levels to improve the detection accuracy. The present invention can achieve better detection accuracy than the current state-of-the-art algorithms for detecting marked parking spaces. In summary, our method has the following contributions:

[0021] 1) The present invention proposes a new detection process for marked parking spaces that is not used in the current methods for detecting marked parking spaces. It can directly output parameters related to the four corner points of the finally detected marked parking space target. After conversion, the coordinate values of the four corner points of the finally detected marked parking space target can be directly obtained.

[0022] 2) The present invention proposes a new feature fusion method. By fusing features at different levels output by the backbone network, the final accuracy of the network is improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 The network design block diagram of the model used in the method for detecting marked parking spaces based on semantic feature attention fusion according to the present invention is shown;

[0024] Figure 2 The flowchart of the semantic feature fusion module according to the present invention is shown;

[0025] Figure 3 The comparison of the visualization results of the present invention and different object detection methods on the parking lot dataset is shown. DETAILED IMPLEMENTATION METHOD

[0026] The embodiments of the present invention are described in detail below. The examples of the embodiments are shown in the drawings, where the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the drawings are exemplary and are intended to explain the present invention and should not be construed as limiting the present invention. Referring to Figure 1 , the method of the embodiment of the present invention operates according to the following steps: S1. Feature extraction of the marked parking space image; S2. Semantic feature attention fusion; S3. Generation of candidate region boxes; S4: The target detector outputs the coordinate values of the four corner points of the marked parking space.

[0027] For step S1, the present invention uses the ResNet101 network and the feature pyramid structure to extract the multi-layer residual mapping features of the image, which are denoted by R1-Conv1~5.

[0028] There are 4 semantic feature attention fusion modules for step S2 in the entire marked parking space detection network. Each semantic feature attention fusion module has two inputs. One input is R1-Conv-5, and the other input is R1-Conv-1 to 4 respectively in the four semantic feature attention fusion modules.

[0029] The semantic feature attention fusion module for step S2 consists of a convolutional layer, global average pooling, and an activation function, as Figure 2 shown, and can be expressed as:

[0030]

[0031] where

[0032]

[0033] In the above formula, Y is R1-Conv-5, X is R1-Conv-1 to 4 respectively in the four semantic feature attention fusion modules, Z is the output feature in the semantic feature attention fusion module, σ1 is the Sigmoid activation function, σ2 is the ReLU activation function, represents the element-wise multiplication in the matrix, represents the matrix addition, represents the convolution operation, where the size of the convolution kernel is 3×3, and the number of channels changes from c to c / r, also represents the convolution operation, and the size of the convolution kernel is 1×1, and the number of channels changes from c / r to c. AvgPool represents the global average pooling.

[0034] For step S4, the prediction head of the object detector directly outputs the parameters related to the four corner points of the marked parking space in the image. After conversion, the coordinate values of the four corner points can be directly obtained.

[0035] The output parameters of the prediction head are x, y, w, h, α1, α2, α3, α4, where x and y are the center points of the rectangle, w and h are the width and height of the rectangle respectively. Through the above four parameters x, y, w, h, the four vertices of the predicted rectangle can be obtained. Let the four vertices of the rectangle be a, b, c, d respectively, and the coordinates of the four vertices a, b, c, d can be expressed as Let the center point of the rectangle be O. Then the angle values of the lines connecting the four vertices a, b, c, d of the marked parking space line to the center point O are α1, α2, α3, α4 respectively.

[0036] Although the embodiments of the present invention have been shown and described above, it can be understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those of ordinary skill in the art can make changes, modifications, substitutions, and variations to the above embodiments within the scope of the present invention.

Claims

1. A method for detecting marked parking spaces based on semantic feature attention fusion, characterized in that Including the following steps: Step S1: Provide an image of a marked parking space to be detected, and perform backbone network feature extraction on the image of the marked parking space to obtain corresponding mapped features; wherein the backbone network feature extraction uses a ResNet101 network and a feature pyramid structure to extract multi-layer residual mapped features of the image, and is represented by R1-Conv1~5; Step S2: Build a semantic feature attention fusion network module, which includes a global average pooling layer and a convolutional layer; input the mapped features obtained in S1 into the semantic feature attention fusion network module to obtain fused features; There are 4 semantic feature attention fusion network modules in the entire marked parking space detection network; each semantic feature attention fusion network module has two inputs, one of which is R1-Conv-5, and the other input is R1-Conv-1~4 in the four semantic feature attention fusion network modules respectively; the semantic feature attention fusion network module is composed of a convolutional layer, global average pooling and an activation function, and can be expressed as: Wherein Among them, Y is R1-Conv-5, X is R1-Conv-1 to 4 respectively in the four semantic feature attention fusion network modules, Z is the output feature in the semantic feature attention fusion network module, σ1 is the Sigmoid activation function, and σ2 is the ReLU activation function. Denotes the direct multiplication of corresponding elements in the matrix. Denotes the matrix addition. Denotes a convolution operation, where the size of the convolution kernel is 3×3 and the number of channels changes from c to c / r. Similarly, it also denotes a convolution operation, and the size of the convolution kernel is 1×1, and the number of channels changes from c / r to c. AvgPool denotes global average pooling. Step S3: Build an RPN network layer and an ROI layer, and the RPN network includes a classification module and a regression module; input the fused features obtained in S2 into the RPN network to obtain preliminary candidate region boxes, and use the ROI layer to map the candidate region boxes and the fused features in S2 into candidate region features; Step S4: Build an object detector, which includes a fully connected layer and a prediction head; input the candidate region features obtained in S3 into the object detector to obtain the detection result of the marked parking space, and the result is in the form of the coordinate values of the four corner points of the marked parking space target in the image.

2. The method according to claim 1, characterized in that, In the prediction head of the object detector in step S4, parameters related to the four corner points of the marked parking space in the image are directly output, and after conversion, the coordinate values of the four corner points can be directly obtained.

3. The method according to claim 2, characterized in that, The output parameters of the prediction head are x, y, w, h, α1, α2, α3, α4, where x and y are the center points of the rectangular box, and w and h are the width and height of the rectangular box respectively. Through the above four parameters x, y, w, h, the four vertices of the predicted rectangular box can be obtained. Let the four vertices of the rectangular box be a, b, c, d respectively, and the coordinates of the four vertices a, b, c, d can be expressed as Let the center point of the rectangular box be O. Then the angular values of the lines connecting the four vertices a, b, c, d of the marked parking space line to the center point O are α1, α2, α3, α4 respectively.

Images