Parking area detection method and device based on visual model SAM
Through the large-scale visual model SAM in parking area detection, using zero sample migration capability, intelligent analysis of diversified environments is realized, and the problem of insufficient generalization capabilities in the existing technology is solved. It is suitable for parking assistance functions of autonomous driving systems. .
Patent Information
- Application Number
- CN202410405638.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-04-03
- Publication Date
- 2024-11-22
AI Technical Summary
The prior art lacks generalization ability in parking area detection, especially in different scenarios, and significantly affects performance on the types of parking areas not included in practical applications.
The large visual model SAM is used to detect the zero-sample parking area. By obtaining the top-view surrounding image of the vehicle, the SAM model is used to segment the image into multiple masks, and the effective mask is filtered through the multi-mask filter, and the marking points are extracted. Infer the location and shape of the parking area to achieve effective detection without the need for specific annotation data and additional training data.
In the context of diverse environments and complexity, intelligent analysis and processing of parking areas are realized, which significantly improves the generalization ability of detection and is suitable for parking assistance functions in autonomous driving systems.
Smart Images

Figure CN119007159A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the field of autonomous driving environment perception, and specifically relates to a method and device for realizing zero-sample parking area detection by using a large visual model (SAM). Background Art
[0002] In recent years, methods for parking area detection using traditional computer vision techniques have undergone extensive research. These methods can be divided into line-based methods and corner-based methods. In line-based methods, various edge detection algorithms are first used to detect parking lines in the surround image, and then line detection algorithms are used to predict line parameters to fit the lines, and then manually formulated geometric constraints are used to infer parking areas. Similarly, corner-based methods first identify corners in the surround image through the Harris corner detector, and then use template matching techniques to locate parking areas. Although these traditional methods often produce effective results, they are sensitive to environmental changes and are not well suited for complex and changing real-world environments.
[0003] As deep learning technology has made significant progress in computer vision tasks such as image recognition, image segmentation, and object detection, parking area detection methods based on deep convolutional neural networks (CNNs) have shown high accuracy and have become the preferred method in autonomous driving perception systems. These deep learning-based methods are mainly divided into two types: one is a two-stage process, which first detects markers through CNN and then uses manually designed geometric constraints for post-processing to predict parking areas; the other is an end-to-end method that uses a model to integrate feature extraction and parking area reasoning, and can directly output predicted parking areas. However, deep learning-based methods require the use of comprehensive and extensive datasets for training, and usually show poor generalization ability between different scenarios. The types of parking areas not included in the training set will significantly affect performance in practical applications. Summary of the invention
[0004] In view of the shortcomings of the prior art in the generalization ability of parking area detection, the present invention proposes a zero-sample parking area detection method based on a visual model SAM, comprising: obtaining a bird's-eye view surround image around a vehicle; segmenting the bird's-eye view surround image into segmentation results with multiple masks through the visual model SAM; filtering the segmentation results to screen out valid masks containing parking area information; using predefined rules to obtain marked points in the valid mask, and obtaining the parking area information according to the marked points.
[0005] Furthermore, a multi-mask filter is used to filter the segmentation result to obtain the effective mask; the multi-mask filter includes: using an area threshold to select a candidate mask, obtaining edge information of the candidate mask through edge detection and line detection algorithms, and matching the edge information with a standard template to select the effective mask from the candidate masks.
[0006] Furthermore, the edge of the effective mask is obtained and transformed into a straight line, and the marking point is extracted based on the intersection relationship and symmetry between the straight lines; the position and shape of the parking area are obtained based on the geometric rule matching relationship of the marking point.
[0007] Furthermore, the mask decoder of the visual model SAM divides the overhead surround image into evenly spaced grids, and maps the embedding and output labels of the overhead surround image into a mask based on all point cues within the grid, thereby generating a segmentation result for the overhead surround image.
[0008] The present invention also proposes a parking area detection device based on a visual model SAM, comprising: an image acquisition module, used to acquire a bird's-eye view image of the surrounding area of a vehicle; an image segmentation module, used to segment the bird's-eye view image into segmentation results with multiple masks through the visual model SAM; a mask filtering module, used to filter the segmentation results to screen out valid masks containing parking area information; and an area detection module, used to obtain marking points in the valid mask using predefined rules, and obtain the parking area information based on the marking points.
[0009] Furthermore, in the mask filtering module, a multi-mask filter is used to filter the segmentation result to obtain the effective mask; an area threshold is used to select a candidate mask, and edge information of the candidate mask is obtained through edge detection and line detection algorithms, and the edge information is matched with a standard template to select the effective mask from the candidate masks.
[0010] Furthermore, the area detection module includes: a marker point extraction module, which is used to obtain the edge of the effective mask and transform the edge into a straight line, and extract the marker point based on the intersection relationship and symmetry between the straight lines; an information acquisition module, which is used to obtain the position and shape of the parking area according to the geometric rule matching relationship of the marker point.
[0011] Furthermore, the image segmentation module divides the overhead view image into evenly spaced grids through the mask decoder of the visual model SAM, maps the embedding and output labels of the overhead view image into a mask based on all point prompts in the grid, and generates a segmentation result for the overhead view image.
[0012] The present invention also proposes a computer-readable storage medium storing computer-executable instructions. When the computer-executable instructions are executed, the parking area detection method based on the visual model SAM as described above is implemented.
[0013] The present invention also provides an electronic device, comprising the parking area detection device based on the visual model SAM as described above.
[0014] This method uses the powerful zero-sample transfer capability of the SAM model to intelligently analyze and process surround images in diverse environments and complex background conditions. It can effectively detect and locate parking areas without the need for specific parking area annotation data and additional training data, and is suitable for parking assistance functions in autonomous driving systems. BRIEF DESCRIPTION OF THE DRAWINGS
[0015] Figure 1 This is a flow chart of the zero-sample parking area detection method based on the visual model SAM of the present invention.
[0016] Figure 2 This is the overall structure diagram of the visual model SAM of the present invention.
[0017] Figure 3 Schematic diagram of a zero-sample parking area detection device based on a visual model SAM according to the present invention.
[0018] Figure 4 It is a schematic diagram of the area detection module of the present invention.
[0019] Figure 5 It is a schematic diagram of an electronic device of the present invention.
[0020] Figure 6 It is a schematic diagram of the hardware structure of an electronic device of the present invention. DETAILED DESCRIPTION
[0021] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings. It should be understood that the specific implementation method described herein is only used to explain the present invention and is not used to limit the present invention.
[0022] After research, the inventors found that applying the large visual model "SegmentAnything Model" (SAM) to the field of parking area detection is a solution to these problems. SAM is an image segmentation model trained on a large-scale dataset. The dataset contains 11 million images and more than 1 billion masks, and does not include overhead and surround images. The high adaptability of the model enables it to generate high-quality masks and demonstrate strong performance in a series of downstream tasks of zero-sample migration. Using the high-quality masks output by SAM, the inventors designed a multi-mask filter and extracted the location, type, and direction information of the markers in the effective mask to infer the location and shape of the parking area, thereby achieving zero-sample parking area detection without the need for training data.
[0023] In order to achieve the above technical objectives, the present invention proposes a zero-sample parking area detection method and device based on a large visual model.
[0024] First, the technical terms of the instruction manual are explained:
[0025] SAM: SegmentAnything Model, a large visual model applied to image segmentation, can output multiple masks of the original image. It consists of an image encoder and a mask decoder, trained on large-scale datasets, and has good zero-sample migration capabilities.
[0026] Image Encoder: Image Encoder uses Visual Transformer (ViT) to extract features from images.
[0027] Mask Decoder: Mask Decoder modifies the decoder block and dynamic mask prediction head of Transformer and embeds the image output by the image encoder into multiple masks.
[0028] MAE pre-training method: MAE pre-training method is a natural language processing technology based on neural networks, which aims to improve the performance of text understanding and generation tasks; it encodes language knowledge as parameters in the model by pre-training the model on large-scale text data, so that the model has better semantic understanding and generation capabilities.
[0029] Figure 1 FIG. 1 is a flow chart of a zero-sample parking area detection method based on a visual model SAM of the present invention. Figure 1 As shown, the first embodiment of the present invention provides a zero-sample parking area detection method, which includes the following steps:
[0030] Step S1, acquiring images around the vehicle through an image acquisition device around the vehicle, where the image acquisition device may be a device on the vehicle or a device outside the vehicle and located around the vehicle; aligning multiple images according to timestamps to obtain images at the same moment, dedistorting, calibrating and projecting the aligned images using a computer vision method according to parameters of the image acquisition device, and splicing and fusing the projected top-view images to obtain a top-view surround image;
[0031] Step S2: input the obtained bird's-eye view surround image into the large visual model SAM, and after being processed by the image encoder and the mask decoder, obtain an image segmentation result with multiple masks;
[0032] The SAM model uses a visual Transformer pre-trained on MAE to process high-resolution input. The mask decoder maps the image embedding and output tags into a mask. For parking area detection, the mask decoder of the SAM model selects a uniformly spaced grid within the image, uses all its points as cues, segments the entire image, and generates multiple segmentation results of the top-down surround image;
[0033] Figure 2 FIG. 1 is a diagram showing the overall structure of the visual model SAM of the present invention. Figure 2 As shown in Figure 1, the SAM model is based on a two-stage architecture. The feature extractor of the model is the Visual Big Model SAM, which consists of an image encoder and a mask decoder; for the image encoder part, the MAE pre-trained Visual Transformer (ViT) is used to process high-resolution inputs. The image encoder converts the input image resolution to 1024×1024 by scaling the image and padding its shorter dimension. Due to the patch step of ViT, the resulting image embedding has a size of 64×64. SAM applies a 1×1 convolution kernel to generate 256 channels, and then applies a 3×3 convolution kernel to generate 256 channels, reducing the channel dimension from 768 to 256. According to the form of the Visual Big Model SAM, the decoder block and dynamic mask prediction head of the Transformer are modified to form a mask decoder. The modified decoder block uses the hint self-attention and cross-attention mechanisms to update all embeddings in both directions. After executing these two blocks, the image embedding is upsampled, and the output label is assigned to a dynamic linear classifier through MLP, and then the mask foreground probability of each image position is calculated. For parking area detection, instead of manually inputting hints, the mask decoder selects a uniformly spaced grid within the image, uses all its points as hints, and segments the entire image, resulting in multiple segmentation result masks for the top-down surround view image.
[0034] Step S3, using a "multi-mask filter" to filter the image segmentation result, and select a candidate mask containing parking area information;
[0035] For these candidate masks, use the "multi-mask filter" to filter out masks that do not contain parking area information based on the area of the mask and the area threshold. For example, use the minimum area threshold to exclude candidate masks with too small an area, and use the maximum area threshold to exclude candidate masks with too large an area. Apply edge detection and line detection algorithms to the remaining candidate masks to identify lines and corner points on the edge of the mask. Determine the validity of the mask based on the type and direction of the corner points and the degree of matching with the standard template to obtain a valid mask.
[0036] Step S4: Use a visual processing method to detect the effective mask and obtain the marking point information;
[0037] For valid masks, Sobel filter is used for edge detection, and Hough transform is used for line detection on these edges; valid coded marker points are extracted based on the intersection relationship and symmetry between straight lines; marker points are detected and classified to obtain information such as the position, type and direction of the marker points.
[0038] Step S5: Match adjacent marking points according to actual geometric rules, infer the position and shape of the parking area, and obtain a parking area detection result;
[0039] Among them, the marked points in the valid mask are paired, and distance constraints and direction constraints are used to exclude marked point pairs that do not conform to the characteristics of the actual parking area; the distance between any two marked points is calculated, and the distance between the marked point pairs and the type of the marked points are used to determine whether the two marked points can constitute the two corner points of the parking area. If so, the detection is successful, thereby accurately inferring the location of the parking area.
[0040] It should be noted that, in various embodiments of the present invention, the sequence numbers of the above steps do not mean the order of execution. The order of execution of the steps should be determined by their functions and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.
[0041] The following is a system embodiment corresponding to the above method embodiment. This embodiment can be implemented in conjunction with the above embodiment. The relevant technical details mentioned in the above embodiment are still valid in this embodiment. In order to reduce repetition, they are not repeated here. Accordingly, the relevant technical details mentioned in this embodiment can also be applied in the above embodiment.
[0042] Figure 3 Schematic diagram of a zero-sample parking area detection device based on a visual model SAM of the present invention. Figure 3As shown, in the second embodiment of the present invention, a zero-sample parking area detection device based on a visual model SAM is proposed, comprising:
[0043] The image acquisition module 10 is used to acquire a bird's-eye view surround image of the vehicle's periphery; wherein, the image of the vehicle's periphery is acquired by an image acquisition device around the vehicle, and the image acquisition device may be a device on the vehicle or a device disposed outside the vehicle and located around the vehicle; multiple images are aligned according to timestamps to acquire images at the same moment, and the aligned images are dedistorted, calibrated and projected using a computer vision method according to the parameters of the image acquisition device, and the multiple bird's-eye views after projection are spliced and fused to obtain a bird's-eye view surround image.
[0044] The image segmentation module 20 is used to segment the overhead surround image into segmentation results with multiple masks through the visual model SAM; wherein, the overhead surround image is divided into evenly spaced grids through the mask decoder of the visual model SAM, and the embedded and output labels of the overhead surround image are mapped into masks according to all point prompts in the grid to generate a segmentation result for the overhead surround image.
[0045] The mask filtering module 30 is used to filter the segmentation result and screen out the valid mask containing the parking area information; wherein, the segmentation result is filtered using a multi-mask filter to obtain a valid mask; the multi-mask filter includes: using an area threshold to select a candidate mask, obtaining edge information of the candidate mask through edge detection and line detection algorithms, matching the edge information with a standard template to select the valid mask from the candidate masks; in this embodiment, a minimum area threshold is used to exclude a candidate mask with an area that is too small, and a maximum area threshold is used to exclude a candidate mask with an area that is too large.
[0046] The area detection module 40 is used to obtain the marking points in the effective mask using predefined rules, and obtain the parking area information according to the marking points. Figure 4 FIG. 1 is a schematic diagram of a region detection module of the present invention. Figure 4 As shown, the area detection module 40 specifically includes:
[0047] The marker point extraction module 41 is used to extract the marker points of the effective mask; specifically, it includes: using the Sobel filter to detect the edges of the effective mask, and using the Hough transform to detect the straight lines on these edges; extracting the marker points of the effective coding according to the intersection relationship and symmetry between the straight lines; detecting and classifying the marker points, so as to obtain the position, type and direction of the marker points and other information.
[0048] An information acquisition module 42 is used to acquire the position and shape of the parking area according to the geometric rule matching relationship of the marking points, and obtain a parking area detection result;
[0049] Among them, the marked points in the valid mask are paired, and distance constraints and direction constraints are used to exclude marked point pairs that do not conform to the characteristics of the actual parking area; the distance between any two marked points is calculated, and the distance between the marked point pairs and the type of the marked points are used to determine whether the two marked points can constitute the two corner points of the parking area. If so, the detection is successful, thereby accurately inferring the location of the parking area.
[0050] For the second embodiment of the present invention, since it basically corresponds to the method embodiment, the relevant parts can refer to the partial description of the method embodiment. The above description is only schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of the present invention. Ordinary technicians in this field can understand and implement it without paying creative work.
[0051] In the third embodiment of the present invention, a computer-readable storage medium is proposed. The zero-sample parking area detection device based on the visual model SAM of the present invention, if its function is implemented in the form of a software functional unit and sold or used as an independent product, can be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a computer-readable storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. Therefore, the computer-readable storage medium of the present invention is used to store a computer program for executing a zero-sample parking area detection method based on the visual model SAM. It should be understood that the computer-readable storage medium in the embodiment of the present invention can be a volatile memory and / or a non-volatile memory. Among them, the non-volatile memory can be a read-only memory (ROM), a programmable ROM (PROM), an erasable PROM (EPROM), an electrically erasable programmable read-only memory (EEPROM) or a flash memory. The volatile memory can be a random access memory (RAM), which is used as an external cache. By way of example but not limitation, many forms of random access memory (RAM) are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous connection dynamic random access memory (SLDRAM) and direct memory bus random access memory (DR RAM).
[0052] Figure 5 Schematic diagram of an electronic device of the present invention. Figure 5As shown, in the fourth embodiment of the present invention, an electronic device is proposed, including the zero-sample parking area detection device based on the visual model SAM as described above. A person of ordinary skill in the art can understand that all or part of the steps in the above method can be completed by instructing related hardware (such as a processor, FPGA, ASIC, etc.) through a program. All or part of the steps of the above embodiment can also be implemented using one or more integrated circuits. Accordingly, each module in the above embodiment can be implemented in the form of hardware, such as implementing its corresponding functions through an integrated circuit, or in the form of a software functional module, such as implementing its corresponding functions through a processor executing a program / instruction stored in a memory. The embodiments of the present invention are not limited to any specific form of combination of hardware and software.
[0053] It should be noted that the structure of the electronic device shown in the drawings of the present invention does not constitute a limitation thereto, and the actual knowledge structure recognition device may include more or fewer components than shown in the drawings, or a combination of certain components, or a different arrangement of components.
[0054] The electronic device of the present invention may be any device with data processing capability, and the device with data processing capability may be a device or apparatus such as a computer. The device embodiment may be implemented by software, or by hardware or a combination of software and hardware. Taking software implementation as an example, as a device in a logical sense, the processor of any device with data processing capability reads the corresponding computer program instructions in the non-volatile memory into the memory and runs the instructions. Figure 6 This is a schematic diagram of the hardware structure of an electronic device of the present invention. Figure 6 As shown in the figure, from the hardware level, it is a hardware structure diagram of any device with data processing capability where the zero-sample parking area detection device based on the visual large model of the present invention is located. Figure 6 In addition to the processor, memory, network interface, and non-volatile memory shown, any device with data processing capabilities in which the apparatus in the embodiments is located may also include other hardware, generally based on the actual functions of the device with data processing capabilities, which will not be described in detail.
[0055] In order to verify the effectiveness of the zero-shot target detection method, this embodiment conducted experiments on the PSDD dataset and compared it with some parking area detection methods based on deep learning. The experimental results shown in Table 1 prove the effectiveness of the present invention. In terms of Base category division (common types of parking spaces: open and rectangular parking spaces), compared with GCN, this embodiment brings 7.77% and 2.57% increase in the Recall and F1 scores of the Base class respectively; in terms of the Recall and F1 scores of the Novel category (special types of parking spaces: bricks, grass, and three-dimensional parking spaces), compared with the best DMPR-PS, it brings 29.96% and 32.19% increase respectively. The above results show that the method proposed in the present invention can have a strong zero-shot migration capability without training on a dedicated parking space dataset.
[0056]
[0057] Table 1
[0058] Compared with the prior art, the beneficial effect of the present invention is that by introducing the large visual model SAM and combining its zero-sample migration strategy, the generalization ability of parking area detection is significantly improved. The present invention not only reduces the dependence on a large amount of labeled data, but also can adapt to changing environmental conditions, has a wide range of application prospects, and is particularly suitable for parking assistance functions in autonomous driving systems.
[0059] Those skilled in the art will readily appreciate other embodiments of the present application after considering the description and practicing the contents disclosed herein. The present application is intended to cover any variations, uses or adaptations of the present application, which follow the general principles of the present application and include common knowledge or customary technical means in the art that are not disclosed in the present application. The description and examples are intended to be exemplary only.
[0060] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.
[0061] The above embodiments are only used to illustrate the design ideas and features of the present invention, and their purpose is to enable those skilled in the art to understand the content of the present invention and implement it accordingly. The protection scope of the present invention is not limited to the above embodiments. Therefore, any equivalent changes or modifications made based on the principles and design ideas disclosed by the present invention are within the protection scope of the present invention.
Claims
1. A parking area detection method based on a visual model SAM, characterized in that: include: Obtaining a bird's-eye view image of the vehicle's surroundings; The top-down surround image is segmented into segmentation results with multiple masks by using a visual model SAM; The segmentation result is filtered to select valid masks containing parking area information; The marking points of the effective mask are extracted, and the parking area information is obtained according to the marking points.
2. The parking area detection method according to claim 1, characterized in that: Using a multi-mask filter to filter the segmentation result to obtain the effective mask; The multi-mask filter includes: using an area threshold to select a mask to be selected, obtaining edge information of the mask to be selected through edge detection and line detection algorithms, and matching the edge information with a standard template to select the effective mask from the mask to be selected.
3. The parking area detection method according to claim 2, characterized in that: The edge of the effective mask is obtained and transformed into a straight line, and the marking point is extracted based on the intersection relationship and symmetry between the straight lines; the position and shape of the parking area are obtained based on the geometric rule matching relationship of the marking point.
4. The parking area detection method according to claim 1, characterized in that: The mask decoder of the visual model SAM divides the overhead surround image into evenly spaced grids, maps the embedding and output labels of the overhead surround image into a mask based on all point cues within the grid, and generates a segmentation result for the overhead surround image.
5. A parking area detection device based on a visual model SAM, characterized in that: include: An image acquisition module is used to acquire a bird's-eye view image of the vehicle's surroundings; An image segmentation module is used to segment the top-down and surround view image into segmentation results with multiple masks through a visual model SAM; A mask filtering module is used to filter the segmentation result and select valid masks containing parking area information; The area detection module is used to obtain the marking points in the effective mask using predefined rules, and obtain the parking area information according to the marking points.
6. The parking area detection device according to claim 5, characterized in that: In the mask filtering module, a multi-mask filter is used to filter the segmentation result to obtain the effective mask; The multi-mask filter includes: using an area threshold to select a mask to be selected, obtaining edge information of the mask to be selected through edge detection and line detection algorithms, and matching the edge information with a standard template to select the effective mask from the mask to be selected.
7. The parking area detection device according to claim 6, characterized in that: The area detection module includes: A marker point extraction module is used to obtain the edge of the effective mask and transform the edge into a straight line, and extract the marker point based on the intersection relationship and symmetry between the straight lines; The information acquisition module is used to obtain the position and shape of the parking area according to the geometric rule matching relationship of the marking point.
8. The parking area detection device according to claim 5, characterized in that: The image segmentation module divides the overhead surround image into evenly spaced grids through the mask decoder of the visual model SAM, maps the embedding and output labels of the overhead surround image into a mask based on all point prompts in the grid, and generates a segmentation result for the overhead surround image.
9. A computer-readable storage medium storing computer-executable instructions, characterized in that: When the computer executable instructions are executed, the parking area detection method based on the visual model SAM as described in any one of claims 1 to 4 is implemented.
10. An electronic device, comprising the parking area detection device based on a visual model SAM as claimed in any one of claims 5 to 8.
Citation Information
Patent Citations
Vehicle counting method and device, electronic equipment and readable storage medium
CN117710294A
Image semantic segmentation for parking space detection
US20220207277A1
Cited By
High-resolution sea ice intelligent detection method and device, electronic equipment and storage medium
CN121353922A
High-resolution sea ice intelligent detection method and device, electronic equipment and storage medium
CN121353922B
Information processing method, information processing device, and program
WO2026163494A1
