A parking space detection method and device, electronic equipment and vehicle

Abstract

The application provides a parking space detection method and device, electronic equipment and vehicle. The method comprises the following steps: acquiring a scene image centered on a target vehicle, wherein the scene image comprises environment information of an environment in which the target vehicle is located; performing multi-scale feature extraction on the scene image to obtain N image features of different scales, wherein N is greater than or equal to 2; cross-fusing M image features in the N image features to obtain at least M fused image features, wherein each fused image feature is obtained by taking one image feature in the M image features as a reference and fusing at least one remaining image feature in the M image features; detecting each parking space contained in the scene image based on the at least M fused image features, and outputting a detection result. In this way, accurate parking space information can be obtained through simple processing logic, the complexity of parking space detection is reduced, and the accuracy of parking space detection is improved.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a parking space detection method, device, electronic equipment, and vehicle. Background Technology

[0002] Automated parking assist (APA) is a core function of Level 2 autonomous driving. The core of APA is visual parking space detection and scene perception, primarily relying on visual sensors to acquire parking space and scene information to guide subsequent parking actions. With the rise of artificial intelligence (AI) and deep learning, visual parking space detection and scene perception have evolved from traditional computer vision methods to neural network-based methods. However, current neural network-based visual parking space detection and scene perception often require complex processing to obtain parking space results, and these results are relatively simple, affecting the final parking performance and leading to a poor user experience. Summary of the Invention

[0003] This application provides a parking space detection method, device, electronic device, vehicle, computer storage medium, and computer program product, which can obtain accurate parking space information through simple processing logic, reducing the complexity of parking space detection and improving the accuracy of parking space detection.

[0004] Firstly, this application provides a parking space detection method, which includes: acquiring a scene image centered on a target vehicle, the scene image including environmental information of the environment in which the target vehicle is located; performing multi-scale feature extraction on the scene image to obtain N image features at different scales, where N≥2; cross-fusing M image features from the N image features to obtain at least M fused image features, wherein each fused image feature is obtained by fusing one of the M image features with at least one of the remaining M image features; detecting each parking space contained in the scene image based on the at least M fused image features, and outputting the detection results. In this way, accurate parking space information can be obtained through simple processing logic, reducing the complexity of parking space detection and improving its accuracy.

[0005] In one possible implementation, the M image features include a first image feature and a second image feature, wherein the scale of the first image feature is higher than that of the second image feature; the at least M fused image features include a first fused image feature and a second fused image feature, wherein the first fused image feature is obtained by fusing the first image feature with the second image feature after downsampling, and the second fused image feature is obtained by fusing the second image feature with the first image feature after upsampling.

[0006] In one possible implementation, before cross-fusion of M image features out of N image features, the method further includes pooling each of the M image features. This increases the receptive field of the extracted image features and / or improves the semantic resolution of the image features.

[0007] In one possible implementation, the M image features include a first image feature and a second image feature, where the scale of the first image feature is larger than that of the second image feature. Pooling is performed on each of the M image features, specifically: spatial pyramid pooling is applied to the first image feature, and dilated spatial pyramid pooling is applied to the second image feature. Thus, by applying spatial pyramid pooling to the high-scale image features, the receptive field of feature extraction can be increased while reducing computational cost. Conversely, by applying dilated spatial pyramid pooling to the low-scale image features, semantic resolution can be improved while maintaining spatial resolution, thereby balancing subsequent segmentation and fusion tasks. This combination of different pooling methods improves the accuracy of subsequent parking space detection.

[0008] In one possible implementation, at least M fused image features include a first fused image feature and a second fused image feature, wherein the scale of the first fused image feature is higher than the scale of the second fused image feature; based on the at least M fused image features, each parking space contained in the scene image is detected, specifically including: based on the first fused image feature, detecting the corner points of the entrances of each parking space contained in the scene image to obtain corner point detection results; based on the first fused image feature, detecting the entrance lines of each parking space contained in the scene image to obtain a first detection result; and based on the second fused image feature, detecting the entrance lines of each parking space contained in the scene image to obtain a second detection result; based on the corner point detection results, the first detection result, and the second detection result, each parking space contained in the scene image is obtained.

[0009] In one possible implementation, corner points of the entrances to each parking space in the scene image are detected based on the first fused image features. Specifically, this includes: extracting horizontal and vertical features from the first fused image features; and detecting corner points of the entrances to each parking space in the scene image based on the horizontal and vertical features. By fusing the horizontal and vertical features of the fused image features, corner point information in the detected parking space corner points can be extracted more effectively, improving the regression accuracy of the parking space corner angles and thus enhancing the accuracy of parking space detection.

[0010] In one possible implementation, the parking spaces contained in the scene image are obtained based on the corner detection results, the first detection results, and the second detection results. Specifically, this includes: determining at least one first parking space based on the corner detection results and the first detection results; determining at least one second parking space based on the corner detection results and the second detection results; and selecting each parking space contained in the scene image from the at least one first parking space and the at least one second parking space.

[0011] In one possible implementation, the target detection result is either a first detection result or a second detection result, and the third parking space is either the first parking space or the second parking space. The third parking space is determined based on the corner detection result and the target detection result, specifically including: determining two corner points at both ends of the first entrance line included in the target detection result; determining a first type of the first entrance line and a first length of the first dividing line of the third parking space based on the distance between the two corner points at both ends of the first entrance line; determining the positions of the other two corner points of the third parking space based on the direction and first length of the first dividing line, wherein the direction of the first dividing line is obtained based on the parking space type of the third parking space and the respective corner angles of the two corner points at both ends of the first entrance line; and determining the third parking space based on its four corner points.

[0012] In one possible implementation, the method further includes: determining a target route for the target vehicle to drive from the target location into the target parking space in the scene image based on the target location of the target vehicle and the various parking spaces contained in the detected scene image; and controlling the target vehicle to drive into the target parking space via the target route.

[0013] In one possible implementation, the method further includes: obtaining a semantic segmentation image with the same scale as the scene image based on at least two of the N image features, and using the semantic segmentation image to plan the parking route of the target vehicle.

[0014] Secondly, this application provides a parking space detection device, comprising: a communication module and a processing module. The communication module is used to acquire a scene image centered on a target vehicle, the scene image including environmental information of the target vehicle's environment; the processing module is used to perform multi-scale feature extraction on the scene image to obtain N image features at different scales, where N≥2; and to cross-fuse M of the N image features to obtain at least M fused image features, wherein each fused image feature is obtained by fusing one of the M image features with at least one of the remaining M image features; and to detect each parking space contained in the scene image based on the at least M fused image features, and output the detection results.

[0015] In one possible implementation, the M image features include a first image feature and a second image feature, wherein the scale of the first image feature is higher than that of the second image feature; the at least M fused image features include a first fused image feature and a second fused image feature, wherein the first fused image feature is obtained by fusing the first image feature with the second image feature after downsampling, and the second fused image feature is obtained by fusing the second image feature with the first image feature after upsampling.

[0016] In one possible implementation, before the processing module cross-fuses M of the N image features, the processing module is further configured to: pool the M image features separately.

[0017] In one possible implementation, the M image features include a first image feature and a second image feature, wherein the scale of the first image feature is higher than the scale of the second image feature; when the processing module performs pooling on the M image features respectively, the processing module is specifically used to: perform spatial pyramid pooling on the first image feature and hollow spatial pyramid pooling on the second image feature.

[0018] In one possible implementation, at least M fused image features include a first fused image feature and a second fused image feature, wherein the scale of the first fused image feature is higher than the scale of the second fused image feature; when the processing module detects each parking space contained in the scene image based on the at least M fused image features, the processing module is specifically used to: detect the corner points of the entrances of each parking space contained in the scene image based on the first fused image features to obtain corner point detection results; detect the entrance lines of each parking space contained in the scene image based on the first fused image features to obtain a first detection result; and detect the entrance lines of each parking space contained in the scene image based on the second fused image features to obtain a second detection result; and obtain each parking space contained in the scene image based on the corner point detection results, the first detection result, and the second detection result.

[0019] In one possible implementation, when the processing module detects the corner points of the entrances to each parking space in the scene image based on the first fused image features, the processing module is specifically used to: extract the horizontal and vertical features from the first fused image features respectively; and detect the corner points of the entrances to each parking space in the scene image based on the horizontal and vertical features.

[0020] In one possible implementation, when the processing module obtains the parking spaces contained in the scene image based on the corner detection results, the first detection results, and the second detection results, the processing module is specifically used to: determine at least one first parking space based on the corner detection results and the first detection results; determine at least one second parking space based on the corner detection results and the second detection results; and filter out the parking spaces contained in the scene image from the at least one first parking space and the at least one second parking space.

[0021] In one possible implementation, the target detection result is either the first detection result or the second detection result, and the third parking space is either the first parking space or the second parking space. When the processing module determines the third parking space based on the corner detection result and the target detection result, the processing module is specifically used to: determine two corner points at both ends of the first entrance line included in the target detection result based on the corner detection result and the target detection result; determine the first type of the first entrance line based on the distance between the two corner points at both ends of the first entrance line, and determine the first length of the first dividing line of the third parking space; determine the positions of the other two corner points of the third parking space based on the direction and the first length of the first dividing line, wherein the direction of the first dividing line is obtained based on the parking space type of the third parking space and the corner angles of the two corner points at both ends of the first entrance line; and determine the third parking space based on the four corner points of the third parking space.

[0022] In one possible implementation, the processing module is further configured to: determine the target route for the target vehicle to drive from the target location into the target parking space in the scene image based on the target location of the target vehicle and the various parking spaces contained in the detected scene image; and control the target vehicle to drive into the target parking space via the target route.

[0023] In one possible implementation, the processing module is further configured to: obtain a semantic segmentation image with the same scale as the scene image based on at least two of the N image features, the semantic segmentation image being used to plan the parking route of the target vehicle.

[0024] Thirdly, this application provides an electronic device, comprising: at least one memory for storing a program; and at least one processor for executing the program stored in the memory; wherein, when the program stored in the memory is executed, the processor is used to execute the method provided in the first aspect.

[0025] Fourthly, this application provides a vehicle that includes the parking space detection device provided in the second aspect, or the electronic device provided in the third aspect.

[0026] Fifthly, this application provides a computer-readable storage medium storing a computer program that, when run on an electronic device, causes the electronic device to perform the method provided in the first aspect.

[0027] In a sixth aspect, this application provides a computer program product, characterized in that, when the computer program product is run on an electronic device, it causes the electronic device to perform the method provided in the first aspect.

[0028] It is understood that the beneficial effects of the second to sixth aspects mentioned above can be found in the relevant descriptions in the first aspect mentioned above, and will not be repeated here. Attached Figure Description

[0029] Figure 1 This is a schematic diagram illustrating an application scenario provided in an embodiment of this application;

[0030] Figure 2 This is a schematic diagram of the hardware structure of a vehicle provided in an embodiment of this application;

[0031] Figure 3 This is a schematic diagram of an environmental image obtained by a visual perception component on a vehicle, and an image obtained after performing inverse perspective transformation and stitching on the environmental image, according to an embodiment of this application.

[0032] Figure 4 This is a schematic diagram of the structure of a parking space sensing network provided in an embodiment of this application;

[0033] Figure 5 This is a schematic diagram of the structure of the fusion layer in the feature fusion unit of a parking space perception network provided in an embodiment of this application;

[0034] Figure 6 This is a schematic diagram of the corner detection layer in the feature detector of a parking space perception network provided in an embodiment of this application;

[0035] Figure 7 This is a schematic diagram of the structure of the entrance line detection layer in the feature detector of a parking space perception network provided in an embodiment of this application;

[0036] Figure 8 This is a schematic diagram of a parking space detection process provided in an embodiment of this application;

[0037] Figure 9 This is a schematic diagram illustrating a process for obtaining the detection results of corner points and entrance lines in a parking space from fused image features, as provided in an embodiment of this application.

[0038] Figure 10 This is a schematic diagram of a process for obtaining parking spaces from the detection results of corner points and entrance lines provided in an embodiment of this application;

[0039] Figure 11 This is a schematic diagram of a parking space provided in an embodiment of this application;

[0040] Figure 12 This is a flowchart illustrating a parking space detection method provided in an embodiment of this application;

[0041] Figure 13 This is a schematic diagram of the structure of a parking space detection device provided in an embodiment of this application. Detailed Implementation

[0042] In this article, the term "and / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. The symbol " / " in this article indicates that the related objects are in an "or" relationship; for example, A / B means A or B.

[0043] The terms "first" and "second," etc., used in the specification and claims herein are used to distinguish different objects, not to describe a specific order of objects. For example, "first response message" and "second response message," etc., are used to distinguish different response messages, not to describe a specific order of response messages.

[0044] In the embodiments of this application, the terms "exemplary" or "for example" are used to indicate that something is an example, illustration, or description. Any embodiment or design that is described as "exemplary" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design. Specifically, the use of the terms "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.

[0045] In the description of the embodiments of this application, unless otherwise stated, "multiple" means two or more, for example, multiple processing units means two or more processing units, multiple elements means two or more elements, etc.

[0046] For example, Figure 1 This illustrates an application scenario. For example... Figure 1 As shown, vehicle 100 needs to drive into one of the parking spaces on either side of it. Vehicle 100 is equipped with an automatic parking function, which allows it to drive into one of the parking spaces to complete the parking process.

[0047] For example, Figure 2 A schematic diagram of the hardware structure of a vehicle is shown. (For example...) Figure 2 As shown, the vehicle 100 may include: a visual sensing component 110, an ultrasonic sensing component 120, an inertial sensing component 130, a processor 140, and a memory 150.

[0048] The visual perception component 110 is primarily used to acquire environmental images surrounding the vehicle 100. The visual perception component 110 may include cameras, such as fisheye cameras. For example, multiple cameras can be used in combination to ensure 360-degree surround view without blind spots.

[0049] The ultrasonic sensing component 120 is mainly used to sense obstacles near the vehicle 100. The ultrasonic sensing component 120 may include ultrasonic sensors, such as lidar, millimeter-wave radar, and ultrasonic radar.

[0050] The inertial sensing component 130 is mainly used to sense the pose information of the vehicle 100. The inertial sensing component 130 may include an inertial sensor.

[0051] Processor 140 may include one or more processing units. Processor 140 may include one or more of the following: application processor (AP), modem, graphics processing unit (GPU), image signal processor (ISP), controller, video codec, digital signal processor (DSP), baseband processor, and / or neural network processing unit (NPU). Different processors may be independent devices or integrated into one or more processors.

[0052] In some embodiments, the processor 140 may perform inverse perspective mapping (IPM) on the environmental image acquired by the visual perception component 110, and stitch the transformed images together to obtain a scene image centered on the vehicle 100; wherein, the scene image may include environmental information about the environment in which the vehicle is located. For example, the scene image may be a top-down view of the environment in which the vehicle 100 is located. Figure 3 As shown, the visual perception component 110 acquires the environmental images around the vehicle 100 as images 31, 32, 33 and 34 respectively. The processor 140 can perform inverse perspective transformation on these four images respectively, and then stitch the transformed images together to obtain image 30 (i.e., the scene image centered on the vehicle 100).

[0053] In some embodiments, the processor 140 may process the acquired scene image centered on the vehicle 100 to detect parking spaces around the vehicle 100.

[0054] In some embodiments, the processor 140 can plan a reasonable parking route and issue corresponding vehicle control commands based on the parking space information it detects, the obstacle information around the vehicle 100 sensed by the ultrasonic sensing component 120, and the pose information of the vehicle 100 sensed by the inertial sensing component 130, in order to complete the parking.

[0055] The memory 150 may store programs, which can be executed by the processor 140. The memory 150 may also store data. The processor 140 may read the data stored in the memory 150. The memory 150 and the processor 140 may be configured separately. Alternatively, the memory 150 may be integrated into the processor 140.

[0056] It is understood that the structures illustrated in the embodiments of this application do not constitute a specific limitation on vehicle 100. In other embodiments of this application, vehicle 100 may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0057] For example, when detecting parking spaces around vehicle 100, a deep learning-based parking space detection method integrating global and local information can be used. This method first divides the acquired visual scene image into a grid, then decomposes the specific parking space information into global parking space information and local corner point information, performs regression on each separately, and then fuses them to achieve high-precision parking space detection. The global information includes: the probability that a grid is within a parking space, the relative offset of the two corner points of the parking space entrance relative to the grid center, the type of parking space (vertical, horizontal, and inclined), and whether the parking space is occupied. The local corner point information includes: the probability that a grid covers a parking space corner point, the relative offset of the corresponding corner point relative to the grid center, and the parking space angle of the corresponding corner point. While this method can improve the detection accuracy of parking space corner points, the combination logic of global and local information is complex, making the processing complicated. Furthermore, since fixed grid division is not robust, the detection performance of this method is significantly affected by image size and the image's perceptual range.

[0058] To improve the accuracy of parking space detection, this application also proposes a parking space detection method. This method is also a deep learning-based parking space detection method. It mainly involves extracting image features from the acquired scene image centered on the vehicle to obtain N (N≥2) image features of different scales. Then, M (M≥2) of the N image features are fused together to obtain at least M fused image features. Next, at least one of these M fused image features is used to obtain the detection results of the parking space corner points and the parking space entrance line. Finally, the detection results of the parking space corner points and the entrance line are combined to obtain the required parking space information. Thus, accurate parking space information can be obtained through simple processing logic, reducing the complexity of parking space detection and improving its accuracy. Furthermore, by fusing at least two of the N image features and performing bilinear interpolation on the fused image features, a semantic segmentation image corresponding to the scene image can be obtained, providing richer scene information and thus improving the final parking success rate. For example, each of the at least M fused image features can be obtained by fusing one of the target image features from the M image features with at least one other image feature from the N image features, excluding the target image feature.

[0059] For example, Figure 4 A schematic diagram of a parking space sensing network is shown. Figure 4 The parking space perception network shown is a deep learning-based neural network, which is a deep learning architecture. Deep learning architecture refers to learning at multiple levels of abstraction through machine learning algorithms. For example... Figure 4 As shown, the parking space perception network 400 may include: a feature extractor 410, a feature fusion unit 420, a parking space detector 430, and a semantic segmenter 440.

[0060] The feature extractor 410 is primarily used to extract image features from the scene image centered on vehicle 100, obtaining N (N≥2) image features at different scales. For example, the feature extractor 410 may include at least one convolutional layer, each of which may include multiple convolution operators, also known as kernels. In image processing, a convolution operator acts as a filter, extracting specific information from the input image matrix. Essentially, a convolution operator can be a weight matrix, which is typically predefined. During the convolution operation, the weight matrix is ​​usually processed horizontally along the input image, pixel by pixel (or two pixels by two pixels… depending on the stride value), to extract specific features from the image. The size of the weight matrix should be related to the image size. It is important to note that the depth dimension of the weight matrix is ​​the same as the depth dimension of the input image; during the convolution operation, the weight matrix extends to the entire depth of the input image. Therefore, convolving with a single weight matrix produces a convolutional output with a single depth dimension. However, in most cases, a single weight matrix is ​​not used; instead, multiple weight matrices of the same dimension are applied. The outputs of each weight matrix are stacked to form the depth dimension of the convolutional image. Different weight matrices can be used to extract different features from the image. This can be simply understood as one weight matrix extracting edge information, another extracting specific colors, yet another blurring unwanted noise, and so on. Since these multiple weight matrices have the same dimension, the feature maps extracted by these weight matrices also have the same dimension. These extracted feature maps with the same dimension are then merged to form the output of the convolution operation.

[0061] In some embodiments, the feature extractor 410 may further include at least one pooling layer (not shown in the figure). After extracting image features at different scales, the feature extractor 410 can use the pooling layer to pool the image features at each scale, thereby increasing the receptive field of the extracted image features and / or improving the semantic resolution of the image features. For example, the pooling layer can use spatial pyramid pooling (SPP) and / or atrous spatial pyramid pooling (ASPP). In addition, multiple 3*3 convolutional kernels can also be used in the pooling layer. In order to keep the output feature size unchanged, the stride of the 3*3 convolutional kernel can be set to 1, and the padding can be set to 1. Of course, other pooling methods can also be used during pooling, such as overlapping pooling, stochastic pooling, global adaptive pooling, etc., depending on the actual situation.

[0062] The feature fusion unit 420 is mainly used to fuse M (M≥2) of the N image features extracted by the feature extractor 410 to obtain at least M fused image features.

[0063] For example, taking N=2 as an example, using the image features at the first scale as a reference, fusing the image features at the first scale with the image features at the second scale yields a fused image feature; using the image features at the second scale as a reference, fusing the image features at the first scale with the image features at the second scale yields another fused image feature. Specifically, when the first scale is higher than the second scale, when fusing image features based on the first scale, the image features at the second scale can be downsampled first, and then the two image features can be fused; when fusing image features based on the second scale, the image features at the first scale can be upsampled first, and then the two image features can be fused.

[0064] For example, taking N=3 as an example. Using the image features at the first scale as a reference, fusing the image features at the first scale with the image features at the second scale yields a fused image feature; fusing the image features at the first scale with the image features at the third scale yields a fused image feature; fusing the image features at the first scale with both the image features at the second and third scales yields a fused image feature. Similarly, using the image at the second scale as a reference, three fused image features can be obtained; using the image at the third scale as a reference, three fused image features can also be obtained. The number of fused image features obtained using a single image feature as a reference can be determined according to actual needs and is not limited here.

[0065] In some embodiments, continue reading Figure 4 The feature fusion unit 420 may include M fusion layers, namely fusion layers 421 to 42m. Each fusion layer may include multiple convolutional kernels of different sizes. Each fusion layer corresponds to an image feature at a specific scale, and each fusion layer is used to fuse image features at its corresponding scale with image features at other scales. For example, ... Figure 5 As shown, the fusion layer 42m may include three convolutional kernels (i.e., convolutional kernels 42m1, 42m2, and 42m3). The first input corresponding to convolutional kernel 42m3 is image feature A at the scale corresponding to the fusion layer 42m, and the second input corresponding to convolutional kernel 42m1 is image feature B at another scale that is fused with image feature A. Convolutional kernel 42m1 is mainly used for feature optimization of the input image features, convolutional kernel 42m2 is mainly used to increase or decrease the scale of the image features obtained from convolutional kernel 42m1, and convolutional kernel 42m3 is mainly used to fuse the image features obtained from convolutional kernel 42m2 with the image feature A obtained from the first input. For example, the size of convolutional kernel 42m1 can be 1x1, the size of convolutional kernel 42m2 can be 3x3, and the size of convolutional kernel 42m3 can be 1x1.

[0066] The parking space detector 430 is mainly used to detect parking spaces from the image features fused by the feature fusion unit 420, in order to obtain parking space detection results, such as the detection results of parking space corners and parking space entrance lines. For example, the detection results of parking space corners may include the position of the corner, the bounding box (i.e., the corner bounding box), and the corner angle; the detection results of parking space entrance lines may include the position of the parking space line, the type of parking space (e.g., horizontal, vertical, inclined, etc.), and whether the parking space is occupied.

[0067] In some embodiments, continue reading Figure 4The parking space detector 430 may include a corner detection layer 431 and an entrance line detection layer 432. Both the corner detection layer 431 and the entrance line detection layer 432 may include multiple convolutional kernels of different sizes. The corner detection layer 431 is primarily used to detect the corners of the parking space to obtain corner detection results, while the entrance line detection layer 432 is primarily used to detect the entrance line of the parking space to obtain detection results related to the entrance line. For example, there may be one or more corner detection layers 431 and entrance line detection layers 432. In some embodiments, the corner detection layer 431 may use the YOLO (you only look once) algorithm to detect the corners of the parking space, and the entrance line detection layer 432 may also use the YOLO algorithm to detect the entrance line of the parking space.

[0068] In some embodiments, such as Figure 6 As shown, the corner detection layer 431 may include a horizontal feature detection layer 4311, a vertical feature detection layer 4312, and a fusion layer 4313. Each of the horizontal feature detection layer 4311, vertical feature detection layer 4312, and fusion layer 4313 may include multiple convolutional kernels of different sizes. Specifically, the horizontal feature detection layer 4311 is mainly used to extract the horizontal features of the fused image features, the vertical feature detection layer 4312 is mainly used to extract the vertical features of the fused image features, and the fusion layer 4313 is mainly used to fuse the horizontal features extracted by the horizontal feature detection layer 4311 and the vertical features extracted by the vertical feature detection layer 4312 to obtain the corner detection result of the parking space. For example, the horizontal feature detection layer 4311 may consist of at least three convolutional kernels of size 1x3, the vertical feature detection layer 4312 may consist of at least three convolutional kernels of size 3x1, and the fusion layer 4313 may consist of at least two convolutional kernels of size 1x1.

[0069] In some embodiments, such as Figure 7 As shown, the entrance line detection layer 432 may include three convolutional kernels (i.e., convolutional kernels 4321, 4322, and 4323). Convolutional kernel 4321 is mainly used for feature optimization of the input image features; convolutional kernel 4322 is mainly used for scaling up the image features obtained from convolutional kernel 4321; and convolutional kernel 4323 is mainly used for regression of the image features obtained from convolutional kernel 4322 to obtain the detection result of the parking space entrance line. For example, the size of convolutional kernel 4321 can be 1x1, the size of convolutional kernel 4322 can be 3x3, and the size of convolutional kernel 4323 can be 1x1.

[0070] The semantic segmenter 440 is mainly used to fuse at least two image features from N image features and perform bilinear interpolation on the fused image features to obtain a semantic segmentation image corresponding to the scene image. In some embodiments, gradient backpropagation of the segmentation result obtained by the semantic segmenter 440 can optimize the parameters in the feature fusioner 420, thereby improving the processing efficiency and accuracy of the feature fusioner 420.

[0071] In some embodiments, the parking space perception network 400 may further include an image stitcher (not shown in the figure). The image stitcher can be used to perform inverse perspective transformation on the environmental images acquired by the visual perception components on the vehicle 100, and stitch the transformed images together to obtain a scene image centered on the vehicle 100.

[0072] The above is an introduction to the parking space perception network provided in the embodiments of this application. This parking space perception network can detect parking space information contained in a scene image centered on a vehicle.

[0073] To make it easier to understand, the following example will be used to illustrate the point.

[0074] like Figure 8 As shown, after inputting a scene image centered on vehicle 100 (such as the original IPM image) into the parking space perception network 400, the feature extractor 410 in the parking space perception network 400 can first scale the image, for example, to a size of 416x416. Of course, the image can also be scaled before being input into the parking space perception network 400. Next, the feature extractor 410 can extract image features at three scales (i.e., C3, C4, and C5). Among them, the scale of image feature C3 is 64x104x104, the scale of image feature C4 is 256x26x26, and the scale of image feature C5 is 512x13x13.

[0075] Next, C5 is input into Spatial Pyramid Pooling (SPP) to increase the receptive field for feature extraction while reducing computational cost, resulting in image feature F5. Then, C4 is processed through Dilated Spatial Pyramid Pooling (ASPP) to improve semantic resolution while maintaining spatial resolution, thus accommodating subsequent segmentation and fusion tasks, resulting in image feature F4. In some embodiments, the pooling process can be selected according to actual needs, and is not limited here.

[0076] Next, the feature fusion unit 420 can upsample image feature F5 and fuse it with image feature F4 to obtain image feature P4; and downsample image feature F4 and fuse it with image feature F5 to obtain image feature P5. For example, this process can be understood as a cross-fusion of image feature F4 and image feature F5.

[0077] Next, the parking space detector 430 can process image feature P5 to obtain detection results for the corner points and the entrance line of the parking space, and process image feature P4 to obtain another detection result for the entrance line of the parking space. For example, this process can be understood as determining the detection results for the corner points and the entrance line of the parking space during the detection of parking spaces in a scene image centered on the vehicle, based on image features P4 and P5.

[0078] As one possible implementation, the YOLO algorithm can be used for processing. For example, the YOLO algorithm can be used to regress corner bounding boxes and entrance wireframes. A corner bounding box can be defined as a square with a length and width of 21 pixels, centered at the corner of the parking space. An entrance wireframe can be defined as the smallest bounding rectangle containing the two corner bounding boxes of the corresponding parking space entrance. Simultaneously, the parking space angle corresponding to the corner bounding box is regressed. This regression can be performed using the sine and cosine values ​​of the parking space angle, with cosine similarity used for supervision. While regressing the entrance wireframe, the type of parking space (horizontal, vertical, and angled), and whether the parking space is occupied, can also be regressed. Before corner bounding box regression, three 1x3 convolutional kernels can be used to optimize image feature P5 to extract its horizontal features, and three 3x1 convolutional kernels can be used to optimize it to extract its vertical features. These extracted features are then fused to better extract corner information from the parking space corner detection results, improving the regression accuracy of parking space corner angles. This enables the detection of corner points and entrance lines of parking spaces.

[0079] For example, such as Figure 9 As shown in (A), when detecting corner points of parking spaces, the fused image features with a scale of (256x13x13) can be input into the aforementioned parking space detector 430. The parking space detector 430 may include a CNN convolutional neural network. After processing by the parking space detector 430, the detection result of the corner points of the parking space obtained from the fused image features can be obtained. In the figure, (13x13) is the feature scale obtained when processing the fused image features, 3 represents the prior box (Anchor), in (4+1+2), 4 represents the position information of the corner point, 1 represents the confidence of whether the corner point exists, and 2 represents the angle of the corner point (such as the sine and cosine values ​​of the corner point angle); Figure 9As shown in (B), when detecting the entrance line of a parking space, the fused image features with a scale of (256x13x13) can be input into the aforementioned parking space detector 430. After processing by the parking space detector 430, the detection result of the entrance line of the parking space obtained from the fused image features can be obtained. In the figure, (13x13) is the feature scale obtained when processing the fused image features, 3 represents the prior box (Anchor), in (4+1+3+1), 4 represents the position information of the entrance line, 1 represents the confidence of whether the entrance line exists, 3 represents the type of parking space (such as horizontal, vertical, inclined, etc.), and 1 represents whether the parking space is occupied; Figure 9 As shown in (C), when detecting the entrance line of a parking space using another fused image feature, the fused image feature with a scale of (128x26x26) can be input into the aforementioned parking space detector 430. After processing by the parking space detector 430, the detection result of the entrance line of the parking space obtained by the fused image feature can be obtained. In the figure, (26x26) is the feature scale obtained when processing the fused image feature, 3 represents the prior box (Anchor), and in (4+1+3+1), 4 represents the position information of the entrance line, 1 represents the confidence of whether the entrance line exists, 3 represents the type of parking space (such as horizontal, vertical, inclined, etc.), and 1 represents whether the parking space is occupied.

[0080] Furthermore, the semantic segmenter 440 can fuse features F3 and F4 obtained based on image feature C3 (e.g., using the same fusion method as Deeplabv3+), and then process them through bilinear interpolation to obtain a semantic segmentation image corresponding to the scene image. For example, the scale of image feature F4 can be increased or decreased using a bilinear interpolation algorithm to make image features F4 and F3 the same scale, then the two can be fused, and finally processed again using a bilinear interpolation algorithm to obtain a semantic segmentation image with the same scale as the input scene image. For example, this semantic segmentation image can be used to plan the parking route of vehicle 100.

[0081] In some embodiments, after obtaining the detection results of the corner points and the entrance lines of each parking space in the scene image centered on the vehicle through the parking space perception network 400, the detection results of the corner points and the entrance lines of each parking space can be fused together to obtain the specific parking space.

[0082] One possible implementation is to first use the non-maximum suppression (NMS) algorithm to process the detection results of corner points and entrance lines to remove redundant corner point boxes and entrance line boxes; then, combine the remaining corner point boxes and entrance line boxes to obtain the specific parking space.

[0083] As another possible implementation, such as Figure 10 As shown, the process of obtaining a specific parking space may include the following steps:

[0084] S1001. Based on the detection results of the entrance line and the detection results of the corner points, obtain the M corner points associated with the entrance line in the detection results of the entrance line, so as to obtain the mapping relationship table between the entrance line and the corner points.

[0085] Specifically, the detection results of each entry wireframe can be traversed, and the intersection-over-union (IOU) ratio between each entry wireframe and each corner frame contained in the corner detection results can be determined. The corners associated with the entry lines within the entry wireframes are then obtained from the obtained IOUs, thus generating a mapping table between entry lines and corners. When the IOU between an entry wireframe and a corner frame is greater than a preset threshold, the corner in that corner frame can be determined to be a corner of the entry line within that entry wireframe.

[0086] S1002. Determine the two corner points on each entrance line respectively.

[0087] Specifically, after obtaining the mapping table between entry lines and corner points, the table can be traversed. When an entry line matches one corner point, and the confidence level of the corresponding entry wireframe is greater than a preset confidence threshold, the position of the other corner point of the entry line can be inferred from the position of that corner point. When an entry line matches two or more corner points, the corner points in the two corner frames with the largest Intersection over Union (IOU) of the corresponding entry wireframe can be selected as the two corner points of the entry line.

[0088] S1003. Determine whether the distance between the two corner points of the i-th entry line among the M entry lines is greater than a preset threshold. The initial value of i is 1.

[0089] Specifically, after obtaining the positions of the two corner points of each entry line, the distance between the two corner points of each entry line can be calculated. If the distance between the two corner points of an entry line is greater than a preset threshold, then S1004 is executed; otherwise, S1003 is executed.

[0090] Specifically, if the distance between the two corner points of an entry line is greater than a preset distance threshold, the entry line and its two corner points are deemed invalid, and the entry line and its two corner points can be discarded, and the next entry line can be processed. If the distance between the two corner points of an entry line is less than or equal to the preset distance threshold, the entry line and its two corner points are deemed valid, and the entry line and its two corner points can be retained.

[0091] S1004, i = i + 1, and determine whether i is greater than M.

[0092] Specifically, i can be set to i + 1, meaning the next entry line can be processed. It can also be checked whether i is greater than M. When i is greater than M, it indicates that all pairs of entry lines out of the M have been processed, and the process can end. When j is less than or equal to M, execution returns to step S1003.

[0093] S1005. Determine the entry line type of the i-th entry line based on the distance between the two corner points of the i-th entry line, and determine the length of the dividing line of the parking space corresponding to the i-th entry line. The entry line type includes long side or short side.

[0094] Specifically, the distance between the two corner points of the i-th entry line can be used to determine whether the entry line corresponding to these two corner points is a long side or a short side. For example, when the distance between the two corner points of the i-th entry line is greater than a preset distance value, the entry line type of the i-th entry line is a long side; when the distance between the two corner points of the i-th entry line is less than or equal to the preset distance value, the entry line type of the i-th entry line is a short side. If the entry line is a long side, the length of the separator line is inferred based on the short side according to the preset long-short side ratio; similarly, the length of the separator line is inferred based on the long side. For example, such as... Figure 11 As shown in (A), the entrance line type for the parking space shown in the figure is the short side; as Figure 11 As shown in (B), the entrance line type for the parking space shown in the figure is the long side; as Figure 11 As shown in (C), the entrance line type for the parking space shown in the figure is the short side. See also... Figure 11 The dividing line of a parking space can refer to the line connected to the entrance line. When the length of the dividing line is greater than the length of the entrance line, the entrance line type is the short side; when the length of the dividing line is less than the length of the entrance line, the entrance line type is the long side. Figure 11 The direction indicated by the middle arrow can be understood as at least one direction of the route a vehicle needs to take to enter a parking space.

[0095] S1006. Determine the direction of the length of the dividing line of the parking space corresponding to the i-th entrance line.

[0096] Specifically, when the parking space type corresponding to the i-th entrance line is horizontal or vertical, the direction of the dividing line for that parking space can be taken as the direction that is perpendicular to the line connecting the two corner points of the i-th entrance line and has the smallest error with the predicted angles of the two corner points. When the parking space type corresponding to the i-th entrance line is an inclined parking space, the average of the predicted angles of the two corner points of the i-th entrance line can be directly selected as the direction of the dividing line for that parking space.

[0097] S1007. Determine the positions of the other two corner points of the parking space corresponding to the i-th entrance line.

[0098] Specifically, after obtaining the direction of the dividing line of the parking space corresponding to the i-th entrance line, the positions of the other two corner points can be obtained from the length of the dividing line of the parking space. At this time, the positions of the four corner points of the parking space are obtained.

[0099] S1008. Determine whether the included angles of the adjacent sides of the four sides of the parking space corresponding to the i-th entrance line are all too large or too small, and / or whether the lengths of the four sides are all too small.

[0100] Specifically, after obtaining the positions of the four corner points of the parking space corresponding to the i-th entrance line, the four sides of the parking space can be determined. If the included angle of all adjacent sides is too large or too small, and / or the length of all four sides is too small, the parking space can be considered invalid and discarded, and the next parking space can be processed, i.e., return to execute S1004; otherwise, execute S1008. This removes excessively tilted parking spaces and excessively small parking spaces.

[0101] S1009. Store the valid parking spaces to obtain K parking spaces.

[0102] S1010, Remove duplicates from K parking spaces to obtain S parking spaces, S≤K.

[0103] Specifically, after obtaining K parking spaces, the Intersection over Union (IOU) ratio between each pair of parking spaces can be calculated. One of the two parking spaces with an IOU greater than a preset IOU threshold is then deleted, resulting in S parking spaces. This removes parking spaces that were detected repeatedly (with large overlap). The resulting S parking spaces are the final legal parking spaces in the vehicle-centered scene image.

[0104] In some embodiments, when the entrance line detection result is obtained from fused image features of different scales, at least one parking space can be obtained by combining the entrance line detection result corresponding to each scale of fused image features with the corner detection result of the parking space. For example, when the entrance line detection result corresponding to the first scale of fused image features is the first detection result, and the entrance line detection result corresponding to the second scale of fused image features is the second detection result, at least one parking space can be obtained from the corner detection result and the first detection result, and at least one parking space can be obtained from the corner detection result and the second detection result. Then, it can be further explained using the above... Figure 10 The method described in S1010 performs deduplication of parking spaces, thereby obtaining each parking space in the scene image.

[0105] Furthermore, when the corner detection results are obtained from fused image features of different scales, at least one parking space can be obtained by combining the corner detection results corresponding to the fused image features of each scale with the detection results of the parking space entrance line. For example, when the corner detection result corresponding to the fused image features of the first scale is the third detection result, and the corner detection result corresponding to the fused image features of the second scale is the fourth detection result, at least one parking space can be obtained from the third and first detection results, at least one parking space can be obtained from the third and second detection results, at least one parking space can be obtained from the third and second detection results, and at least one parking space can be obtained from the fourth and second detection results. Then, as described above... Figure 10 The method described in S1010 performs deduplication of parking spaces, thereby obtaining each parking space in the scene image.

[0106] In some embodiments, after obtaining the parking spaces in the scene image centered on the vehicle, the vehicle can be controlled based on the obtained parking spaces. Specifically, the module in the vehicle used for vehicle control (such as the aforementioned processor 140) can construct a globally stable semantic map based on the obtained information of each parking space and the vehicle's pose information perceived by the vehicle's inertial sensing components (such as the aforementioned inertial sensing component 130). Then, the control module can plan a reasonable parking route based on the obstacle information near the vehicle perceived by the vehicle's ultrasonic sensing components (such as the aforementioned ultrasonic sensing component 120) and the constructed semantic map, and generate control commands. Finally, the vehicle can automatically park in a parking space according to the control commands, thus completing automatic parking. For example, when planning the route, a parking space where the vehicle can park can be determined based on the vehicle's current position and the parking spaces in the scene image. Then, a route from the vehicle's current position to the parking space can be planned, and the vehicle can be controlled to drive into the parking space along this route.

[0107] Furthermore, when planning parking routes, semantic segmentation images of the same scale as the scene image can be incorporated to improve parking success rates. For example, when there is an extremely narrow parking space, data from the vehicle's ultrasonic sensing components might indicate that parking is not possible. However, semantic segmentation images can analyze the situation to suggest that parking is feasible, allowing the vehicle to proceed and improving the success rate. Additionally, because the ultrasonic sensing components have a relatively short detection range, they cannot detect distant obstacles, making it easy for obstacles to appear on the planned route. Semantic segmentation images, on the other hand, can reveal the status of various obstacles in the vehicle's environment, allowing the planned route to avoid obstacles that ultrasonic sensors cannot detect, thus improving parking success rates.

[0108] Next, based on the content described above, a parking space detection method provided by an embodiment of this application will be introduced. It is understood that this method is proposed based on the content described above, and some or all of the content of this method can be found in the description above.

[0109] Please see Figure 12 , Figure 12 This is a flowchart illustrating a parking space detection method provided in an embodiment of this application. It is understood that this method can be executed by any device, equipment, platform, or equipment cluster with computing and processing capabilities. Figure 12 As shown, the parking space detection method may include:

[0110] In S1201, a scene image centered on the target vehicle is acquired. The scene image includes environmental information about the environment in which the target vehicle is located. For example, the scene image may be a scene image obtained from the environmental image acquired by the visual perception component 110 on the vehicle 100 as described above.

[0111] In S1202, multi-scale feature extraction is performed on the scene image to obtain N image features at different scales, where N ≥ 2. For example, this can be achieved through the above... Figure 4 The feature extractor 410 in the parking space perception network 400 described herein extracts features from the scene image. For example, the data input into the parking space perception network can be a scene image or an environmental image acquired by the visual perception component 110 on the vehicle 100, depending on the actual situation, and is not limited here.

[0112] In S1203, M image features from the N image features are cross-fused to obtain at least M fused image features. Each fused image feature is obtained by fusing one of the M image features with at least one of the remaining M image features. For example, this can be achieved through the above... Figure 4 The feature fusion unit 420 in the parking space perception network 400 described herein cross-fuses M image features out of N image features.

[0113] S1204. Based on at least M fused image features, detect each parking space contained in the scene image and output the detection results. For example, this can be achieved through the above... Figure 4 The parking space detector 430 in the parking space perception network 400 described herein detects each parking space contained in the scene image to obtain one or more of the following information: corner points, entrance lines, type, location, etc. of each parking space; furthermore, it can be obtained from the above... Figure 10The method described herein processes the data detected by the parking space detector 430 to obtain the detection results for each parking space. These results can then be output, for example, displayed on a screen of an electronic device, or input into a program module for parking path planning. For example, the parking space detection results may include one or more of the following: the shape, size, and location of the parking space.

[0114] Therefore, accurate parking space information in the vehicle's environment can be obtained through simple processing logic, reducing the complexity of parking space detection and improving its accuracy. Furthermore, a semantic segmentation image with the same scale as the scene image can be obtained based on at least two of N image features. This semantic segmentation image is used to plan the parking route for the target vehicle, thereby improving the success rate of subsequent parking. For example, this can be achieved through... Figure 4 The semantic segmenter 440 in the parking space perception network 400 described herein acquires the semantic segmentation image.

[0115] Based on the methods described in the above embodiments, this application provides a parking space detection device. Please refer to... Figure 13 , Figure 13 This is a schematic diagram of the structure of a parking space detection device provided in an embodiment of this application. Figure 13 As shown, the parking space detection device 1300 includes a communication module 1301 and a processing module 1302. The communication module 1301 acquires a scene image centered on the target vehicle, the scene image including environmental information about the target vehicle's environment. The processing module 1302 performs multi-scale feature extraction on the scene image to obtain N image features at different scales, where N≥2; cross-fuses M of the N image features to obtain at least M fused image features, where each fused image feature is based on one of the M image features and fused with at least one of the remaining M image features; and detects each parking space contained in the scene image based on the at least M fused image features, and outputs the detection results.

[0116] In some embodiments, the M image features include a first image feature and a second image feature, wherein the scale of the first image feature is higher than the scale of the second image feature; the at least M fused image features include a first fused image feature and a second fused image feature, wherein the first fused image feature is obtained by fusing the first image feature with the second image feature after downsampling, and the second fused image feature is obtained by fusing the second image feature with the first image feature after upsampling.

[0117] In some embodiments, before the processing module 13021302 cross-fused M of the N image features, the processing module 13021302 is further configured to: pool the M image features respectively.

[0118] In some embodiments, the M image features include a first image feature and a second image feature, wherein the scale of the first image feature is higher than the scale of the second image feature; when the processing module 1302 performs pooling on the M image features respectively, the processing module 1302 is specifically used to: perform spatial pyramid pooling on the first image feature and hollow spatial pyramid pooling on the second image feature.

[0119] In some embodiments, at least M fused image features include a first fused image feature and a second fused image feature, wherein the scale of the first fused image feature is higher than the scale of the second fused image feature; when the processing module 1302 detects each parking space contained in the scene image based on at least M fused image features, the processing module 1302 is specifically used to: detect the corner points of the entrances of each parking space contained in the scene image based on the first fused image features to obtain corner point detection results; detect the entrance lines of each parking space contained in the scene image based on the first fused image features to obtain a first detection result; and detect the entrance lines of each parking space contained in the scene image based on the second fused image features to obtain a second detection result; and obtain each parking space contained in the scene image based on the corner point detection results, the first detection result, and the second detection result.

[0120] In some embodiments, when the processing module 1302 detects the corner points of the entrances to each parking space in the scene image based on the first fused image features, the processing module 1302 is specifically used to: extract the horizontal features and vertical features from the first fused image features respectively; and detect the corner points of the entrances to each parking space in the scene image based on the horizontal features and vertical features.

[0121] In some embodiments, when the processing module 1302 obtains each parking space contained in the scene image based on the corner detection result, the first detection result, and the second detection result, the processing module 1302 is specifically used to: determine at least one first parking space based on the corner detection result and the first detection result; determine at least one second parking space based on the corner detection result and the second detection result; and filter each parking space contained in the scene image from the at least one first parking space and the at least one second parking space.

[0122] In some embodiments, the target detection result is a first detection result or a second detection result, and the third parking space is a first parking space or a second parking space. When the processing module 1302 determines the third parking space based on the corner detection result and the target detection result, the processing module 1302 is specifically used to: determine two corner points at both ends of the first entrance line included in the target detection result based on the corner detection result and the target detection result; determine a first type of the first entrance line based on the distance between the two corner points at both ends of the first entrance line, and determine a first length of the first dividing line of the third parking space; determine the positions of the other two corner points of the third parking space based on the direction and the first length of the first dividing line, wherein the direction of the first dividing line is obtained based on the parking space type of the third parking space and the corner angles of the two corner points at both ends of the first entrance line; and determine the third parking space based on the four corner points of the third parking space.

[0123] In some embodiments, the processing module 1302 is further configured to: determine the target route for the target vehicle to drive from the target location into the target parking space in the scene image based on the target location of the target vehicle and the parking spaces contained in the detected scene image; and control the target vehicle to drive into the target parking space via the target route.

[0124] In some embodiments, the processing module 1302 is further configured to: obtain a semantic segmentation image with the same scale as the scene image based on at least two of the N image features, the semantic segmentation image being used to plan the parking route of the target vehicle.

[0125] It should be understood that the above-described device is used to execute the methods in the above embodiments. The implementation principle and technical effect of the corresponding program modules in the device are similar to those described in the above methods. The working process of the device can be referred to the corresponding process in the above methods, and will not be repeated here.

[0126] Based on the methods in the above embodiments, this application provides an electronic device. The electronic device may include at least one memory for storing a program; and at least one processor for executing the program stored in the memory; wherein, when the program stored in the memory is executed, the processor executes the methods in the above embodiments.

[0127] Based on the methods described in the above embodiments, this application provides a vehicle. The vehicle may include the parking space detection device described above, or it may include the electronic equipment described above. For example, the vehicle may be the one described above... Figure 2 The vehicle 100 described in the text.

[0128] It is understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application. In addition, in some possible implementations, each step in the above embodiments may be selectively executed according to the actual situation, and may be partially or fully executed, which is not limited here.

[0129] It is understood that the processor in the embodiments of this application can be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. A general-purpose processor can be a microprocessor or any conventional processor.

[0130] The method steps in the embodiments of this application can be implemented in hardware or by a processor executing software instructions. The software instructions can consist of corresponding software modules, which can be stored in random access memory (RAM), flash memory, read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), registers, hard disks, portable hard disks, CD-ROMs, or any other form of storage medium known in the art. An exemplary storage medium is coupled to the processor, enabling the processor to read information from and write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and the storage medium can reside in an ASIC.

[0131] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted through the computer-readable storage medium. The computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state disk (SSD)).

[0132] It is understood that the various numerical designations used in the embodiments of this application are merely for descriptive convenience and are not intended to limit the scope of the embodiments of this application.

Claims

1. A parking space detection method, characterized in that, The method includes: Acquire a scene image centered on the target vehicle, the scene image including environmental information of the environment in which the target vehicle is located; Multi-scale feature extraction is performed on the scene image to obtain N image features at different scales, where N≥2; M image features from the N image features are cross-fused to obtain at least M fused image features, wherein each fused image feature is obtained by fusing one of the M image features with at least one of the remaining image features from the M image features; The at least M fused image features include a first fused image feature and a second fused image feature, wherein the scale of the first fused image feature is higher than the scale of the second fused image feature; Based on the at least M fused image features, each parking space contained in the scene image is detected, specifically including: Based on the first fused image features, the corner points of the entrances to each parking space in the scene image are detected to obtain the corner point detection results. Based on the first fused image features, the entrance lines of each parking space contained in the scene image are detected to obtain a first detection result; and based on the second fused image features, the entrance lines of each parking space contained in the scene image are detected to obtain a second detection result. Based on the corner detection results, the first detection result, and the second detection result, each parking space contained in the scene image is obtained; Output the detection results.

2. The method according to claim 1, characterized in that, The M image features include a first image feature and a second image feature, wherein the scale of the first image feature is higher than the scale of the second image feature; The at least M fused image features include a first fused image feature and a second fused image feature. The first fused image feature is obtained by fusing the first image feature with the second image feature after downsampling. The second fused image feature is obtained by fusing the second image feature with the first image feature after upsampling.

3. The method according to claim 1 or 2, characterized in that, Before cross-fusion of M image features from the N image features, the method further includes: Pooling is performed on the M image features respectively.

4. The method according to claim 3, characterized in that, The M image features include a first image feature and a second image feature, wherein the scale of the first image feature is higher than the scale of the second image feature; The pooling of the M image features specifically includes: Spatial pyramid pooling is performed on the first image features, and hollow spatial pyramid pooling is performed on the second image features.

5. The method according to claim 1, characterized in that, The step of detecting corner points at the entrances of each parking space in the scene image based on the first fused image features specifically includes: Extract the horizontal and vertical features from the features of the first fused image, respectively; Based on the horizontal and vertical features, the corner points of the entrances to each parking space in the scene image are detected.

6. The method according to claim 1 or 5, characterized in that, The step of obtaining each parking space contained in the scene image based on the corner detection results, the first detection result, and the second detection result specifically includes: Based on the corner detection results and the first detection results, at least one first parking space is determined; Based on the corner detection results and the second detection results, at least one second parking space is determined; Each parking space contained in the scene image is selected from the at least one first parking space and the at least one second parking space.

7. The method according to claim 6, characterized in that, The target detection result is either the first detection result or the second detection result, and the third parking space is either the first parking space or the second parking space; Based on the corner detection results and the target detection results, the third parking space is determined, specifically including: Based on the corner detection results and the target detection results, determine the two corner points at both ends of the first entry line included in the target detection results; Based on the distance between the two corner points at both ends of the first entrance line, the first type of the first entrance line is determined, and the first length of the first dividing line of the third parking space is determined. Based on the direction and length of the first dividing line, the positions of the other two corner points of the third parking space are determined. The direction of the first dividing line is obtained based on the parking space type of the third parking space and the corner angles of the two corner points at both ends of the first entrance line. The third parking space is determined based on its four corner points.

8. The method according to any one of claims 1-7, characterized in that, The method further includes: Based on the target location of the target vehicle and the parking spaces contained in the detected scene image, determine the target route for the target vehicle to drive from the target location into the target parking space in the scene image; Control the target vehicle to drive into the target parking space via the target route.

9. The method according to any one of claims 1-8, characterized in that, The method further includes: Based on at least two of the N image features, a semantic segmentation image with the same scale as the scene image is obtained, and the semantic segmentation image is used to plan the parking route of the target vehicle.

10. A parking space detection device, characterized in that, The device includes: The communication module is used to acquire a scene image centered on the target vehicle, the scene image including environmental information of the environment in which the target vehicle is located; The processing module is used to perform multi-scale feature extraction on the scene image to obtain N image features at different scales, where N≥2; The processing module is further configured to cross-fuse M image features from the N image features to obtain at least M fused image features, wherein each fused image feature is obtained by fusing one of the M image features with at least one remaining image feature from the M image features; The at least M fused image features include a first fused image feature and a second fused image feature, wherein the scale of the first fused image feature is higher than the scale of the second fused image feature; The processing module, used to detect each parking space contained in the scene image based on the at least M fused image features, specifically includes: Based on the first fused image features, the corner points of the entrances to each parking space in the scene image are detected to obtain the corner point detection results. Based on the first fused image features, the entrance lines of each parking space contained in the scene image are detected to obtain a first detection result; and based on the second fused image features, the entrance lines of each parking space contained in the scene image are detected to obtain a second detection result. Based on the corner detection results, the first detection result, and the second detection result, each parking space contained in the scene image is obtained; Output the detection results.

11. The apparatus according to claim 10, characterized in that, The M image features include a first image feature and a second image feature, wherein the scale of the first image feature is higher than the scale of the second image feature; The at least M fused image features include a first fused image feature and a second fused image feature. The first fused image feature is obtained by fusing the first image feature with the second image feature after downsampling. The second fused image feature is obtained by fusing the second image feature with the first image feature after upsampling.

12. The apparatus according to claim 10 or 11, characterized in that, Before the processing module performs cross-fusion on M of the N image features, the processing module is further configured to: Pooling is performed on the M image features respectively.

13. The apparatus according to claim 12, characterized in that, The M image features include a first image feature and a second image feature, wherein the scale of the first image feature is higher than the scale of the second image feature; When the processing module performs pooling on the M image features respectively, the processing module is specifically used for: Spatial pyramid pooling is performed on the first image features, and hollow spatial pyramid pooling is performed on the second image features.

14. The apparatus according to claim 10, characterized in that, When the processing module detects the corner points of the entrances to each parking space in the scene image based on the first fused image features, the processing module is specifically used for: Extract the horizontal and vertical features from the features of the first fused image, respectively; Based on the horizontal and vertical features, the corner points of the entrances to each parking space in the scene image are detected.

15. The apparatus according to claim 10 or 14, characterized in that, When the processing module obtains each parking space contained in the scene image based on the corner detection results, the first detection results, and the second detection results, the processing module is specifically used for: Based on the corner detection results and the first detection results, at least one first parking space is determined; Based on the corner detection results and the second detection results, at least one second parking space is determined; Each parking space contained in the scene image is selected from the at least one first parking space and the at least one second parking space.

16. The apparatus according to claim 15, characterized in that, The target detection result is either the first detection result or the second detection result, and the third parking space is either the first parking space or the second parking space; When the processing module determines the third parking space based on the corner detection results and the target detection results, the processing module is specifically used for: Based on the corner detection results and the target detection results, determine the two corner points at both ends of the first entry line included in the target detection results; Based on the distance between the two corner points at both ends of the first entrance line, the first type of the first entrance line is determined, and the first length of the first dividing line of the third parking space is determined. Based on the direction and length of the first dividing line, the positions of the other two corner points of the third parking space are determined. The direction of the first dividing line is obtained based on the parking space type of the third parking space and the corner angles of the two corner points at both ends of the first entrance line. The third parking space is determined based on its four corner points.

17. The apparatus according to any one of claims 10-16, characterized in that, The processing module is further configured to: Based on the target location of the target vehicle and the parking spaces contained in the detected scene image, determine the target route for the target vehicle to drive from the target location into the target parking space in the scene image; Control the target vehicle to drive into the target parking space via the target route.

18. The apparatus according to any one of claims 10-17, characterized in that, The processing module is further configured to: Based on at least two of the N image features, a semantic segmentation image with the same scale as the scene image is obtained, and the semantic segmentation image is used to plan the parking route of the target vehicle.

19. An electronic device, characterized in that, include: At least one memory for storing programs; At least one processor for executing the program stored in the memory; When the program stored in the memory is executed, the processor is used to execute the method as described in any one of claims 1-9.

20. A vehicle, characterized in that, It includes the parking space detection device as described in any one of claims 10-18, or the electronic device as described in claim 19.

21. A computer-readable storage medium storing a computer program that, when run on an electronic device, causes the electronic device to perform the method as described in any one of claims 1-9.

22. A computer program product, characterized in that, When the computer program product is run on an electronic device, it causes the electronic device to perform the method as described in any one of claims 1-9.

Images