Fish-eye camera-based parking space information generation method, device, equipment and medium

Abstract

This disclosure presents a method, apparatus, device, and medium for generating parking space information based on a fisheye camera. One specific implementation of the method includes: acquiring a fisheye image of a target parking space; inputting the target parking space fisheye image into a backbone network layer of a pre-trained parking space detection model to obtain a multi-scale feature map set; inputting the multi-scale feature map set into a feature fusion layer to obtain a multi-scale fused feature map set; inputting the multi-scale fused feature map set into a detection head layer to obtain a parking space detection information set; generating a parking space topology information set based on the parking space detection information set; and generating a parking space information set based on the parking space topology information set. This implementation improves the completeness of the extracted parking space information, thereby increasing the accuracy of the generated parking space information, and consequently reducing the number of parking accidents and improving vehicle parking safety.

Description

Technical Field

[0001] The embodiments disclosed herein relate to the field of computer technology, and more specifically to a method, apparatus, device, and medium for generating parking space information based on a fisheye camera. Background Technology

[0002] With the development of computer technology and autonomous driving technology, automatic parking space detection and automatic parking technologies are becoming increasingly mature. Currently, when generating parking space information, the common approach is to use a general parking space detection model (such as YOLOv5) to detect parking space images and obtain parking space detection results for vehicle parking.

[0003] However, the inventors discovered that when generating parking space information using the above method, the following technical problems often occur:

[0004] First, during the generation of parking space information, multi-scale feature extraction and fusion were not performed on the parking space images. This resulted in low completeness of the extracted parking space information when the parking spaces in the images were large or small. Consequently, the accuracy of the generated parking space information was low, leading to a higher number of parking accidents and reduced vehicle parking safety.

[0005] Secondly, in the process of generating parking space information, the predicted parking spaces were not further subdivided by type and the conditions of the parking space area were not judged, and the relevant parking space information was not filtered based on the judgment results. This resulted in a low accuracy rate of the generated parking space information, which in turn led to a large number of parking accidents and low safety when vehicles were parked.

[0006] The information disclosed in this background section is only intended to enhance the understanding of the background of the inventive concept, and therefore may contain information that does not constitute prior art known to those skilled in the art. Summary of the Invention

[0007] The summary portion of this disclosure is intended to provide a brief overview of the concepts, which will be described in detail in the detailed description portion. This summary portion is not intended to identify key or essential features of the claimed technical solutions, nor is it intended to limit the scope of the claimed technical solutions.

[0008] Some embodiments of this disclosure provide a method, apparatus, electronic device, and computer-readable medium for generating parking space information based on a fisheye camera to solve one or more of the technical problems mentioned in the background section above.

[0009] In a first aspect, some embodiments of this disclosure provide a method for generating parking space information based on a fisheye camera. The method includes: acquiring a fisheye image of a target parking space, wherein the target parking space fisheye image is a parking space image captured by a fisheye camera mounted on a target vehicle; inputting the target parking space fisheye image into a backbone network layer of a pre-trained parking space detection model to obtain a multi-scale feature map set, wherein the parking space detection model further includes a feature fusion layer and a detection head layer; inputting the multi-scale feature map set into the feature fusion layer to obtain a multi-scale fused feature map set; inputting the multi-scale fused feature map set into the detection head layer to obtain a parking space detection information set; generating a parking space topology information set based on the parking space detection information set; and generating a parking space information set based on the parking space topology information set.

[0010] Secondly, some embodiments of this disclosure provide a parking space information generation device based on a fisheye camera. The device includes: an acquisition unit configured to acquire a target parking space fisheye image, wherein the target parking space fisheye image is a parking space image captured by a fisheye camera mounted on a target vehicle; a first input unit configured to input the target parking space fisheye image into a backbone network layer of a pre-trained parking space detection model to obtain a multi-scale feature map set, wherein the parking space detection model further includes a feature fusion layer and a detection head layer; a second input unit configured to input the multi-scale feature map set into the feature fusion layer to obtain a multi-scale fused feature map set; a third input unit configured to input the multi-scale fused feature map set into the detection head layer to obtain a parking space detection information set; a first generation unit configured to generate a parking space topology information set based on the parking space detection information set; and a second generation unit configured to generate a parking space information set based on the parking space topology information set.

[0011] Thirdly, some embodiments of this disclosure provide an electronic device, including: one or more processors; and a storage device having one or more programs stored thereon, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the method described in any implementation of the first aspect above.

[0012] Fourthly, some embodiments of this disclosure provide a computer-readable medium having a computer program stored thereon, wherein the program, when executed by a processor, implements the method described in any of the implementations of the first aspect above.

[0013] The above-described embodiments of this disclosure have the following beneficial effects: The parking space information generation method based on a fisheye camera, as described in some embodiments of this disclosure, improves the completeness of the extracted parking space information, thereby increasing the accuracy of the generated parking space information, reducing the number of parking accidents, and improving vehicle parking safety. Specifically, the reason for the low completeness of the extracted parking space information, leading to a low accuracy of the generated parking space information, is that multi-scale feature extraction and multi-scale feature fusion were not performed on the parking space image during the generation process. This results in low completeness of the extracted parking space information when the parking space in the image is too large or too small. Consequently, the accuracy of the generated parking space information is low, leading to a higher number of parking accidents and lower vehicle parking safety. Therefore, the parking space information generation method based on a fisheye camera, as described in some embodiments of this disclosure, firstly acquires a target parking space fisheye image. This target parking space fisheye image is a parking space image captured by a fisheye camera mounted on the target vehicle. Thus, the target parking space fisheye image can be obtained and used to determine parking space information. Secondly, the fisheye image of the target parking space is input into the backbone network layer of a pre-trained parking space detection model to obtain a multi-scale feature map set. This parking space detection model also includes a feature fusion layer and a detection head layer. This multi-scale feature map set can be used to improve the completeness of the extracted parking space information. Then, the multi-scale feature map set is input into the feature fusion layer to obtain a multi-scale fused feature map set. This results in a multi-scale fused feature map set with high completeness of parking space information. Next, the multi-scale fused feature map set is input into the detection head layer to obtain a parking space detection information set. This results in a parking space detection information set representing the discriminative power of parking spaces. Subsequently, a parking space topology information set is generated based on the parking space detection information set. This results in a parking space topology information set representing the topology of each parking space area. Finally, a parking space information set is generated based on the parking space topology information set. This results in a parking space information set with high accuracy. Because fusing multi-scale feature map sets with different dimensions of representation data improves the completeness of the extracted parking space information. This improves the accuracy of generated parking space information, thereby reducing the number of parking accidents and enhancing vehicle safety when parking. Attached Figure Description

[0014] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. Throughout the drawings, the same or similar reference numerals denote the same or similar elements. It should be understood that the drawings are schematic, and elements are not necessarily drawn to scale.

[0015] Figure 1This is a flowchart of some embodiments of the method for generating parking space information based on a fisheye camera according to the present disclosure;

[0016] Figure 2 This is a schematic diagram of the structure of some embodiments of the parking space information generation device based on a fisheye camera according to the present disclosure;

[0017] Figure 3 This is a schematic diagram of the structure of an electronic device suitable for implementing some embodiments of the present disclosure. Detailed Implementation

[0018] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0019] It should also be noted that, for ease of description, only the parts relevant to the invention are shown in the accompanying drawings. Unless otherwise specified, the embodiments and features described in this disclosure can be combined with each other.

[0020] It should be noted that the concepts of "first" and "second" mentioned in this disclosure are used only to distinguish different devices, modules or units, and are not used to limit the order of functions performed by these devices, modules or units or their interdependencies.

[0021] It should be noted that the terms "a" and "a plurality of" used in this disclosure are illustrative rather than restrictive, and those skilled in the art should understand that, unless otherwise expressly indicated in the context, they should be understood as "one or more".

[0022] The names of messages or information exchanged between multiple devices in the embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of such messages or information.

[0023] This disclosure will now be described in detail with reference to the accompanying drawings and embodiments.

[0024] Figure 1 A flow 100 of some embodiments of the fisheye camera-based parking space information generation method according to this disclosure is shown. The fisheye camera-based parking space information generation method includes the following steps:

[0025] Step 101: Obtain the fisheye image of the target parking space.

[0026] In some embodiments, the execution entity (e.g., a computing device) of the parking space information generation method based on a fisheye camera can acquire a target parking space fisheye image from an image acquisition device via a wired or wireless connection. The target parking space fisheye image can be a parking space image captured by a fisheye camera mounted on a target vehicle. The image acquisition device can be a device with shooting or screenshot capabilities. For example, the image acquisition device can be any fisheye camera installed on the target vehicle. The target vehicle can be a vehicle equipped with the image acquisition device. It should be noted that the wireless connection method can include, but is not limited to, 3G / 4G connections, WiFi connections, Bluetooth connections, WiMAX connections, Zigbee connections, UWB (ultra-wideband) connections, and other currently known or future known wireless connection methods.

[0027] Step 102: Input the fisheye image of the target parking space into the backbone network layer of the pre-trained parking space detection model to obtain a multi-scale feature map set.

[0028] In some embodiments, the execution entity can input the target parking space fisheye image into the backbone network layer of a pre-trained parking space detection model to obtain a multi-scale feature map set. The parking space detection model may further include a feature fusion layer and a detection head layer. The multi-scale feature map set may include a first-scale feature map, a second-scale feature map, and a third-scale feature map. The data dimension of the first-scale feature map is greater than that of the second-scale feature map. The data dimension of the second-scale feature map is greater than that of the third-scale feature map. The parking space detection model can be a network model that takes the target parking space fisheye image as input and outputs a parking space detection information set. The backbone network layer can be a network layer capable of extracting features from the input image. Specifically, the backbone network layer may include four modules: a convolutional block + 1 residual convolutional block, (convolutional block + 3 residual convolutional blocks) * 2, and a convolutional block + SPP (Spatial Pyramid Pooling) layer + 5 convolutional blocks. Each convolutional block can consist of a 1x1 convolutional layer, a 3x3 convolutional layer, a Batch Normalization (BN) layer, and a Silu activation function layer. Residual convolutional blocks can consist of a 1x1 convolutional layer and a residual layer. The feature fusion layer described above can fuse features from the feature maps obtained from the backbone network layers, enabling better detection of features at multiple scales. The detection head layer described above can be a network layer that performs detection for different tasks. The detection head layer can include a fully connected head (fc-head) and a convolutional head (conv-head). The detection tasks described above can include classification tasks and localization tasks. The fully connected head is used for classification tasks. The convolutional head is used for localization tasks.

[0029] In practice, the aforementioned execution entity can input the fisheye image of the target parking space into the backbone network layer of the pre-trained parking space detection model to obtain a multi-scale feature map set.

[0030] In some alternative implementations of certain embodiments, the above parking space detection model can be trained in the following manner:

[0031] The first step is to obtain a sample set. This sample set includes fisheye images of sample parking spaces and a set of corresponding parking space detection information. The parking space detection information set can be the sample labels corresponding to the fisheye images of the sample parking spaces. It should be noted that the entity that trains the parking space detection model can be the aforementioned entity or other computing devices.

[0032] The second step involves performing the following training steps based on the sample set:

[0033] The first training step involves inputting the fisheye images of at least one sample parking space from the sample set into an initial parking space detection model to obtain a set of parking space detection information for each of the at least one sample. The parking space detection model can be an initial neural network capable of obtaining the parking space detection information set from the fisheye images. This initial neural network can be a neural network to be trained. Specifically, the initial neural network can be a neural network including a backbone network layer, a feature fusion layer, and a detection head layer.

[0034] The second training step involves comparing the parking space detection information set corresponding to each of the at least one sample with the corresponding sample parking space detection information set. Here, the comparison can be a comparison of the accuracy of each parking space detection information in the parking space detection information set corresponding to each of the at least one sample with that in the corresponding sample parking space detection information set.

[0035] The third training step involves determining whether the initial parking space detection model has reached a preset optimization objective based on the comparison results. Here, the optimization objective can refer to the accuracy of the initial parking space detection model being trained being greater than or equal to a preset threshold. This preset threshold can be 0.95.

[0036] The fourth training step is to determine the initial parking space detection model as the completed parking space detection model in response to the determination that the initial parking space detection model has achieved the above optimization objective.

[0037] Optionally, the steps for training the above parking space detection model may further include:

[0038] The fifth training step, in response to the determination that the initial parking space detection model has not achieved the above optimization objective, adjusts the network parameters of the initial parking space detection model, and uses unused samples to form a sample set. The adjusted initial parking space detection model is then used as the initial parking space detection model, and the above training steps are executed again. As an example, the back propagation algorithm (BP algorithm) and gradient descent methods (such as mini-batch gradient descent) can be used to adjust the network parameters of the initial parking space detection model.

[0039] Step 103: Input the multi-scale feature map set into the feature fusion layer to obtain the multi-scale fused feature map set.

[0040] In some embodiments, the execution entity may input the multi-scale feature map set into the feature fusion layer to obtain a multi-scale fused feature map set.

[0041] In some optional implementations of certain embodiments, the execution entity may input the multi-scale feature map set into the feature fusion layer through the following steps to obtain a multi-scale fused feature map set:

[0042] The first step is to perform convolution processing on the first-scale feature map to obtain the second-scale convolution feature map.

[0043] The second step is to fuse the second-scale convolutional feature map with the first-scale feature map to obtain a second-scale convolutional fused feature map. Here, the fusion can be a stitching operation.

[0044] The third step is to perform convolution processing on the above-mentioned second-scale convolution fusion feature map to obtain the third-scale convolution feature map.

[0045] The fourth step is to fuse the above third-scale convolutional feature map with the above third-scale feature map to obtain the third-scale convolutional fusion feature map as the third fusion feature map.

[0046] The fifth step is to perform upsampling on the above third-scale feature map to obtain the second-scale upsampled feature map.

[0047] The sixth step is to fuse the above-mentioned second-scale upsampled feature map, the above-mentioned second-scale feature map, and the above-mentioned second-scale convolutional feature map to obtain the second fused feature map.

[0048] Step 7: Perform upsampling processing on the above-mentioned second-scale upsampled feature map to obtain the first-scale upsampled feature map.

[0049] Step 8: The above-mentioned first-scale upsampled feature map and the above-mentioned first-scale feature map are fused to obtain the first fused feature map.

[0050] The ninth step is to determine the first fusion feature map, the second fusion feature map, and the third fusion feature map as a multi-scale fusion feature map set.

[0051] Step 104: Input the multi-scale fused feature map set into the detection head layer to obtain the parking space detection information set.

[0052] In some embodiments, the execution entity can input the multi-scale fused feature map set into the detection head layer to obtain a parking space detection information set. The parking space detection information set may include: parking space center grid point coordinates, parking space center grid point coordinate offset information, parking space vertex grid coordinates, parking space vertex grid coordinate offset information, parking space vertex confidence information, and parking space vertex relative offset information. The parking space center grid point coordinates can be the coordinates of the top-left vertex of the central grid within the grid of the parking space region corresponding to the above-mentioned parking space detection information. The parking space center grid point coordinate offset information can characterize the offset error of the parking space center grid point coordinates. The parking space vertex grid coordinates can include: top-left corner grid coordinates, bottom-left corner grid coordinates, top-right corner grid coordinates, and bottom-right corner grid coordinates. The top-left corner grid coordinates can be the coordinates of the top-left vertex of the top-left corner grid within the grid of the parking space region. The coordinates of the lower left grid can be the coordinates of the upper left vertex of the lower left grid within the grid encompassing the parking space area. The coordinates of the upper right grid can be the coordinates of the upper left vertex of the upper right grid within the grid encompassing the parking space area. The coordinates of the lower right grid can be the coordinates of the upper left vertex of the lower right grid within the grid encompassing the parking space area. For example, if the parking space area comprises 9 grids, the coordinates of the upper left grid could be (0, 3). The coordinates of the lower left grid could be (0, 0). The coordinates of the upper right grid could be (2, 3). The coordinates of the lower right grid could be (2, 1). The coordinates of the center grid point of the parking space could be (1, 2).

[0053] The aforementioned parking space vertex grid coordinate offset information group may include: top-left corner grid coordinate offset information, bottom-left corner grid coordinate offset information, top-right corner grid coordinate offset information, and bottom-right corner grid coordinate offset information. The bottom-left corner grid coordinate offset information represents the offset error of the bottom-left corner grid coordinate information. The top-right corner grid coordinate offset information represents the offset error of the top-right corner grid coordinate information. The bottom-right corner grid coordinate offset information represents the offset error of the bottom-right corner grid coordinate information. The aforementioned parking space vertex relative offset information group may include: top-left corner relative offset information, bottom-left corner relative offset information, top-right corner relative offset information, and bottom-right corner relative offset information. The top-left corner relative offset information represents the offset of the top-left corner relative to the parking space center grid point coordinate information. The bottom-left corner relative offset information represents the offset of the bottom-left corner relative to the parking space center grid point coordinate information. The top-right corner relative offset information represents the offset of the top-right corner relative to the parking space center grid point coordinate information. The bottom-right corner relative offset information represents the offset of the bottom-right corner relative to the parking space center grid point coordinate information. The aforementioned parking space vertex confidence information group may include: top-left corner parking space vertex confidence information, bottom-left corner parking space vertex confidence information, top-right corner parking space vertex confidence information, and bottom-right corner parking space vertex confidence information. The top-left corner parking space vertex confidence information characterizes the accuracy of predicting the top-left corner grid coordinates. The bottom-left corner parking space vertex confidence information characterizes the accuracy of predicting the bottom-left corner grid coordinates. The top-right corner parking space vertex confidence information characterizes the accuracy of predicting the top-right corner grid coordinates. The bottom-right corner parking space vertex confidence information characterizes the accuracy of predicting the bottom-right corner grid coordinates.

[0054] In practice, the aforementioned execution entity can input the aforementioned multi-scale fused feature map set into the aforementioned detection head layer to obtain the parking space detection information set.

[0055] Step 105: Generate a parking space topology information set based on the parking space detection information set.

[0056] In some embodiments, the execution entity may generate a parking space topology information set based on the aforementioned parking space detection information set.

[0057] In some optional implementations of certain embodiments, based on the aforementioned parking space detection information set, the executing entity can generate a parking space topology information set through the following steps:

[0058] The first step is to perform the following steps for each parking space detection record in the above parking space detection information set:

[0059] The first step is to generate the center point coordinates of the parking space based on the center grid point coordinates and offset information of the parking space, which are included in the parking space detection information. In practice, the executing entity can offset the center grid point coordinates of the parking space according to the offset information, and use the offset coordinates as the center point coordinates of the parking space.

[0060] The second step involves generating a first parking space vertex information group based on the parking space detection information, including the relative offset information group of parking space vertices and the coordinate information of the parking space center point. Here, the generation method for each first parking space vertex in the first parking space vertex information group is the same as the generation method for the parking space center point coordinate information, and will not be repeated here.

[0061] The third step involves generating a second parking space vertex information group based on the parking space vertex grid coordinate information group and the parking space vertex grid coordinate offset information group included in the aforementioned parking space detection information. The second parking space vertex information in this second parking space vertex information group corresponds one-to-one with the first parking space vertex information in the aforementioned first parking space vertex information group. The generation method for the second parking space vertex information group is the same as that for the first parking space vertex information group, and will not be repeated here.

[0062] The fourth step involves determining that, in response to the determination that at least one parking space vertex confidence information in the parking space vertex confidence information group included in the aforementioned parking space detection information is less than a preset confidence threshold, the first parking space vertex information group is identified as the parking space vertex information group. The preset confidence threshold can be a pre-defined confidence threshold. For example, the preset confidence threshold can be 0.3.

[0063] The fifth step, in response to determining that the confidence information of each parking space vertex in the above parking space vertex confidence information group is greater than or equal to the above preset confidence threshold, is to perform the following steps:

[0064] The first sub-step involves generating a set of circular regions based on the aforementioned first parking space vertex information group. Specifically, each first parking space vertex in the first parking space vertex information group corresponds one-to-one with a circular region in the aforementioned set of circular regions. Similarly, each second parking space vertex in the second parking space vertex information group corresponds one-to-one with a circular region in the aforementioned set of circular regions. In practice, for each first parking space vertex in the first parking space vertex information group, the executing entity can draw a circle with the first parking space vertex as the center and a preset distance threshold as the radius to obtain the set of circular regions.

[0065] The second sub-step involves determining the coordinate positions corresponding to each second parking space vertex in the aforementioned second parking space vertex information group as a second parking space vertex coordinate position group. These coordinate positions can be the coordinate positions of the corresponding second parking space vertex in the fisheye image of the target parking space. These coordinate positions can be two-dimensional coordinates. For example, the coordinate position could be (2, 5).

[0066] The third sub-step is to determine that each of the second parking space vertex coordinate positions in the above-mentioned second parking space vertex coordinate position group is within its corresponding circular area, and to determine the above-mentioned second parking space vertex information group as the parking space vertex information group.

[0067] The fourth sub-step is to determine the first parking space vertex information group as the parking space vertex information group in response to determining that at least one of the second parking space vertex coordinate positions in the second parking space vertex coordinate position group is not within the corresponding circular area.

[0068] The fifth sub-step involves determining the topology corresponding to each parking space vertex information in the determined parking space vertex information group as the parking space topology. The topology corresponding to each parking space vertex information can be a planar graphic determined by the parking space vertex information. For example, the topology corresponding to each parking space vertex information can be a rectangle.

[0069] The second step is to determine the obtained parking space topology structure as a parking space topology structure information set.

[0070] Step 106: Generate a parking space information set based on the parking space topology information set.

[0071] In some embodiments, the executing entity can generate a parking space information set based on the aforementioned parking space topology information set. The parking space detection information in the aforementioned parking space detection information set may further include: parking space type information, parking space occupancy information, and a parking space vertex visibility information group. The parking space vertex visibility information in the aforementioned parking space vertex visibility information group corresponds one-to-one with the parking space vertex grid coordinate information in the aforementioned parking space vertex grid coordinate information group. The aforementioned parking space type information may include any one of the following: vertical type, horizontal type, and diagonal type. The aforementioned parking space occupancy information can characterize whether the parking space corresponding to the parking space occupancy information is occupied. The parking space vertex visibility information in the aforementioned parking space vertex visibility information group can characterize whether the parking space vertex corresponding to the parking space vertex visibility information is visible.

[0072] In some optional implementations of certain embodiments, based on the parking space topology information set, the aforementioned execution entity can generate the parking space information set through the following steps:

[0073] The first step is to determine the set of parking space topology regions based on the aforementioned set of parking space topology information. In practice, the executing entity can determine the region corresponding to each parking space topology information in the aforementioned set of parking space topology information as the set of parking space topology regions. Here, the region corresponding to the parking space topology information can be the area enclosed by the parking space topology information (i.e., a planar graphic). For example, the parking space topology information can be a rectangular region.

[0074] The second step is to generate a set of parking space topology regions based on the above parking space type information, preset parking space area conditions, and the above parking space topology structure area set.

[0075] In some optional implementations of certain embodiments, based on the parking space type information, preset parking space area conditions, and the parking space topology area set, the executing entity can generate the parking space topology area set through the following steps. The preset parking space area conditions may include preset vertical parking space area conditions, preset horizontal parking space area conditions, and preset angled parking space area conditions.

[0076] The first step involves performing the following steps for each parking space topology region in the set of parking space topology regions:

[0077] The first sub-step involves, in response to determining that the parking space type information corresponding to the aforementioned parking space topology region is of the vertical type, deleting parking space topology regions that do not meet the aforementioned preset vertical parking space region conditions from the parking space topology region set. The aforementioned preset vertical parking space region conditions can be pre-defined conditions that satisfy the requirements for a vertical parking space region. The aforementioned vertical parking space region can be a parking space region whose entrance is perpendicular to the road through which the target vehicle travels. The aforementioned entrance position can be the position corresponding to the shorter side of the rectangle surrounding the parking space. For example, the aforementioned preset vertical parking space region conditions can be that the four corners of the parking space region are all right angles, the parking space length is 5.5 meters, and the parking space width is 2.5 meters. Here, the parking space region corners can be the angles formed by adjacent sides enclosing the parking space region.

[0078] The second sub-step, in response to determining that the parking space type information corresponding to the aforementioned parking space topology area is horizontal, removes parking space topology areas that do not meet the aforementioned preset horizontal parking space area conditions from the parking space topology area set. The aforementioned preset horizontal parking space area conditions can be pre-defined conditions that satisfy the requirements for a horizontal parking space area. The aforementioned horizontal parking space area can be a parking space area whose entrance is parallel to the road used by the target vehicle. For example, the aforementioned preset horizontal parking space area conditions can be that the four corners of the corresponding parking space area are all right angles, the parking space length is 6 meters, and the parking space width is 2.5 meters.

[0079] The third sub-step, in response to determining that the parking space type information corresponding to the aforementioned parking space topology area is of the slanted type, removes parking space topology areas that do not meet the aforementioned preset slanted parking space area conditions from the parking space topology area set. The aforementioned preset slanted parking space area conditions can be pre-defined conditions that satisfy the requirements for a slanted parking space area. The aforementioned slanted parking space area can be a parking space area where the entrance location and the road used by the target vehicle meet a preset angle condition. The aforementioned preset angle condition can be that the angle formed by the road used by the target vehicle and the boundary line of the parking space area corresponding to the length of the parking space is greater than or equal to 30 degrees and less than 60 degrees. For example, the aforementioned preset slanted parking space area conditions can be that two of the four angles of the corresponding parking space area are acute angles, the other two are obtuse angles, and the parking space length is 6 meters and the parking space width is 2.8 meters.

[0080] The second step is to define each parking space topology region that satisfies the above-mentioned preset vertical parking space region conditions, the above-mentioned preset horizontal parking space region conditions, and the above-mentioned preset oblique parking space region conditions as a set of parking space topology regions.

[0081] The third step involves performing distortion region masking on each parking space topology region in the aforementioned parking space topology region set, resulting in a set of distorted parking space topology regions. In practice, the execution entity can perform distortion region masking on each parking space topology region in the aforementioned parking space topology region set according to a preset mask region, resulting in a set of distorted parking space topology regions. The preset mask region can be a pre-defined area for storing parking space topology regions.

[0082] The fourth step is to combine each parking space region in the above parking space region set with the corresponding parking space occupancy information and parking space vertex visibility information to obtain a parking space information set.

[0083] The first and second steps and related content described above, as an inventive point of this disclosure, solve the second technical problem mentioned in the background art: "In the process of generating parking space information, the predicted parking spaces are not subdivided by type and the parking space area conditions are not judged, and the relevant parking space information is not filtered based on the judgment results, resulting in a low accuracy rate of the generated parking space information, which in turn leads to a large number of parking accidents and low vehicle parking safety." The factors that lead to a low accuracy rate of the generated parking space information, resulting in a large number of parking accidents and low vehicle parking safety are often as follows: In the process of generating parking space information, the predicted parking spaces are not subdivided by type and the parking space area conditions are not judged, and the relevant parking space information is not filtered based on the judgment results, resulting in a low accuracy rate of the generated parking space information, which in turn leads to a large number of parking accidents and low vehicle parking safety. If the above factors are solved, the accuracy rate of the generated parking space information can be improved, the number of parking accidents can be reduced, and the safety of vehicle parking can be improved. To achieve this effect, firstly, for each parking space topology region in the set of parking space topology regions, the following steps are performed: Secondly, in response to determining that the parking space type information corresponding to the above parking space topology region is of the vertical type, parking space topology regions that do not meet the above preset vertical parking space region conditions are deleted from the set of parking space topology regions. This filters out parking space topology regions that do not meet the preset vertical parking space region conditions and are subdivided into vertical types within the parking space type. Then, in response to determining that the parking space type information corresponding to the above parking space topology region is of the horizontal type, parking space topology regions that do not meet the above preset horizontal parking space region conditions are deleted from the set of parking space topology regions. This filters out parking space topology regions that do not meet the preset horizontal parking space region conditions and are subdivided into horizontal types within the parking space type. Finally, in response to determining that the parking space type information corresponding to the above parking space topology region is of the diagonal type, parking space topology regions that do not meet the above preset diagonal parking space region conditions are deleted from the set of parking space topology regions. Therefore, parking space topology regions that do not meet the preset angled parking space area conditions can be filtered out from the parking space type subdivided into angled types. Finally, the parking space topology regions that meet the above-mentioned preset vertical parking space area conditions, preset horizontal parking space area conditions, and preset angled parking space area conditions are determined as a set of parking space topology regions. Thus, a set of parking space topology regions that meet the preset vertical parking space area conditions, preset horizontal parking space area conditions, and preset angled parking space area conditions can be obtained. This is because by subdividing parking spaces into types and judging parking space area conditions, parking space topology regions that do not meet the above-mentioned preset vertical parking space area conditions, preset horizontal parking space area conditions, and preset angled parking space area conditions can be filtered out.This can improve the accuracy of the generated parking space information, thereby reducing the number of parking accidents and improving vehicle safety when parking.

[0084] Optionally, the aforementioned executing entity may also perform the following steps: sending the aforementioned parking space information set to the aforementioned target vehicle, so that the aforementioned target vehicle performs a parking operation based on the aforementioned parking space information set.

[0085] In practice, the aforementioned executing entity can send the aforementioned set of parking space information to the aforementioned target vehicle, causing the target vehicle to select any parking space area in the set that meets the parking conditions as the target parking space area and perform a parking operation. The aforementioned parking conditions can be parking space occupancy information indicating that the parking space is not occupied.

[0086] The above-described embodiments of this disclosure have the following beneficial effects: The parking space information generation method based on a fisheye camera, as described in some embodiments of this disclosure, improves the completeness of the extracted parking space information, thereby increasing the accuracy of the generated parking space information, reducing the number of parking accidents, and improving vehicle parking safety. Specifically, the reason for the low completeness of the extracted parking space information, leading to a low accuracy of the generated parking space information, is that multi-scale feature extraction and multi-scale feature fusion were not performed on the parking space image during the generation process. This results in low completeness of the extracted parking space information when the parking space in the image is too large or too small. Consequently, the accuracy of the generated parking space information is low, leading to a higher number of parking accidents and lower vehicle parking safety. Therefore, the parking space information generation method based on a fisheye camera, as described in some embodiments of this disclosure, firstly acquires a target parking space fisheye image. This target parking space fisheye image is a parking space image captured by a fisheye camera mounted on the target vehicle. Thus, the target parking space fisheye image can be obtained and used to determine parking space information. Secondly, the fisheye image of the target parking space is input into the backbone network layer of a pre-trained parking space detection model to obtain a multi-scale feature map set. This parking space detection model also includes a feature fusion layer and a detection head layer. This multi-scale feature map set can be used to improve the completeness of the extracted parking space information. Then, the multi-scale feature map set is input into the feature fusion layer to obtain a multi-scale fused feature map set. This results in a multi-scale fused feature map set with high completeness of parking space information. Next, the multi-scale fused feature map set is input into the detection head layer to obtain a parking space detection information set. This results in a parking space detection information set representing the discriminative power of parking spaces. Subsequently, a parking space topology information set is generated based on the parking space detection information set. This results in a parking space topology information set representing the topology of each parking space area. Finally, a parking space information set is generated based on the parking space topology information set. This results in a parking space information set with high accuracy. Because fusing multi-scale feature map sets with different dimensions of representation data improves the completeness of the extracted parking space information. This improves the accuracy of generated parking space information, thereby reducing the number of parking accidents and enhancing vehicle safety when parking.

[0087] Further reference Figure 2 As an implementation of the methods shown in the above figures, this disclosure provides some embodiments of a parking space information generation device based on a fisheye camera. These device embodiments are similar to... Figure 1 Corresponding to the method embodiments shown, the device can be specifically applied to various electronic devices.

[0088] like Figure 2As shown, a parking space information generation device 200 based on a fisheye camera in some embodiments includes: an acquisition unit 201, a first input unit 202, a second input unit 203, a third input unit 204, a first generation unit 205, and a second generation unit 206. The acquisition unit 201 is configured to acquire a fisheye image of the target parking space, wherein the fisheye image of the target parking space is a parking space image captured by a fisheye camera mounted on the target vehicle; the first input unit 202 is configured to input the fisheye image of the target parking space into the backbone network layer of a pre-trained parking space detection model to obtain a multi-scale feature map set, wherein the parking space detection model further includes a feature fusion layer and a detection head layer; the second input unit 203 is configured to input the multi-scale feature map set into the feature fusion layer to obtain a multi-scale fused feature map set; the third input unit 204 is configured to input the multi-scale fused feature map set into the detection head layer to obtain a parking space detection information set; the first generation unit 205 is configured to generate a parking space topology information set based on the parking space detection information set; and the second generation unit 206 is configured to generate a parking space information set based on the parking space topology information set.

[0089] It is understandable that the units described in the fisheye camera-based parking space information generation device 200 are related to the reference... Figure 1 The steps in the described method correspond to each other. Therefore, the operations, features, and beneficial effects described above for the method also apply to the parking space information generation device 200 based on a fisheye camera and the units contained therein, and will not be repeated here.

[0090] The following is for reference. Figure 3 This document illustrates a schematic diagram of an electronic device 300 (e.g., a computing device) suitable for implementing some embodiments of the present disclosure. The electronic devices in some embodiments of the present disclosure may include, but are not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 3 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments of this disclosure.

[0091] like Figure 3As shown, the electronic device 300 may include a processing unit 301 (e.g., a central processing unit, a graphics processor, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory 302 (ROM) or a program loaded from a storage device 308 into a random access memory 303 (RAM). The RAM 303 also stores various programs and data required for the operation of the electronic device 300. The processing unit 301, ROM 302, and RAM 303 are interconnected via a bus 304. An input / output (I / O) interface 305 is also connected to the bus 304.

[0092] Typically, the following devices can be connected to I / O interface 305: input devices 306 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 307 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 308 including, for example, magnetic tapes, hard disks, etc.; and communication devices 309. Communication device 309 allows electronic device 300 to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 3 An electronic device 300 with various devices is shown; however, it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed alternatively. Figure 3 Each box shown can represent a device or multiple devices as needed.

[0093] In particular, according to some embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, some embodiments of this disclosure include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication device 309, or installed from storage device 308, or installed from ROM 302. When the computer program is executed by processing device 301, it performs the functions defined in the methods of some embodiments of this disclosure.

[0094] It should be noted that, in some embodiments of this disclosure, the computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium may be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In some embodiments of this disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In some embodiments of this disclosure, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0095] In some implementations, clients and servers can communicate using any currently known or future-developed network protocol such as HTTP (Hypertext Transfer Protocol), and can interconnect with digital data communication (e.g., communication networks) of any form or medium. Examples of communication networks include local area networks (“LANs”), wide area networks (“WANs”), the Internet (e.g., the Internet of Things), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks), as well as any currently known or future-developed networks.

[0096] The aforementioned computer-readable medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device. The aforementioned computer-readable medium carries one or more programs that, when executed by the electronic device, cause the electronic device to: acquire a fisheye image of a target parking space, wherein the target parking space fisheye image is a parking space image captured by a fisheye camera mounted on a target vehicle; input the target parking space fisheye image into the backbone network layer of a pre-trained parking space detection model to obtain a multi-scale feature map set, wherein the parking space detection model further includes a feature fusion layer and a detection head layer; input the multi-scale feature map set into the feature fusion layer to obtain a multi-scale fused feature map set; input the multi-scale fused feature map set into the detection head layer to obtain a parking space detection information set; generate a parking space topology information set based on the parking space detection information set; and generate a parking space information set based on the parking space topology information set.

[0097] Computer program code for performing operations of some embodiments of this disclosure can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0098] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0099] The units described in some embodiments of this disclosure can be implemented in software or hardware. The described units can also be housed in a processor; for example, a processor may be described as including an acquisition unit, a first input unit, a second input unit, a third input unit, a first generation unit, and a second generation unit. The names of these units do not necessarily limit the specific unit; for example, the acquisition unit may also be described as a "unit for acquiring a fisheye image of a target parking space."

[0100] The functions described above in this document can be performed at least in part by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), system-on-a-chip (SoCs), complex programmable logic devices (CPLDs), and so on.

[0101] The above description is merely a selection of preferred embodiments of this disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of the invention involved in the embodiments of this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described inventive concept. For example, technical solutions formed by substituting the above-described features with (but not limited to) technical features with similar functions disclosed in the embodiments of this disclosure.

Claims

1. A method for generating parking space information based on a fisheye camera, comprising: Acquire a fisheye image of the target parking space, wherein the fisheye image of the target parking space is a parking space image captured by a fisheye camera mounted on the target vehicle; The fisheye image of the target parking space is input into the backbone network layer of the pre-trained parking space detection model to obtain a multi-scale feature map set. The parking space detection model also includes a feature fusion layer and a detection head layer. The multi-scale feature map set is input into the feature fusion layer to obtain a multi-scale fused feature map set; The multi-scale fused feature map set is input into the detection head layer to obtain the parking space detection information set; Based on the parking space detection information set, a parking space topology information set is generated, including: For each parking space detection information in the parking space detection information set, perform the following steps: Based on the parking space detection information, including the parking space center grid point coordinate information and the parking space center grid point coordinate offset information, the parking space center point coordinate information is generated; Based on the parking space detection information including the relative offset information group of parking space vertices and the coordinate information of the parking space center point, a first parking space vertex information group is generated; Based on the parking space detection information including the parking space vertex grid coordinate information group and the parking space vertex grid coordinate offset information group, a second parking space vertex information group is generated, wherein the second parking space vertex information in the second parking space vertex information group corresponds to the first parking space vertex information in the first parking space vertex information group. In response to determining that at least one parking space vertex confidence information in the parking space vertex confidence information group included in the parking space detection information is less than a preset confidence threshold, the first parking space vertex information group is determined as the parking space vertex information group. In response to determining that the confidence information of each parking space vertex in the parking space vertex confidence information group is greater than or equal to the preset confidence threshold, the following steps are performed: Based on the first parking space vertex information group, a circular region set is generated, wherein the first parking space vertex information in the first parking space vertex information group corresponds to the circular region in the circular region set, and the second parking space vertex information in the second parking space vertex information group corresponds to the circular region in the circular region set. The coordinate positions corresponding to each second parking space vertex information in the second parking space vertex information group are determined as the second parking space vertex coordinate position group; In response to determining that each of the second parking space vertex coordinate positions in the second parking space vertex coordinate position group is within the corresponding circular area, the second parking space vertex information group is determined as the parking space vertex information group. In response to determining that at least one of the second parking space vertex coordinate positions in the second parking space vertex coordinate position group is not within the corresponding circular area, the first parking space vertex information group is determined as the parking space vertex information group. The topology corresponding to each parking space vertex information included in the determined parking space vertex information group is determined as the parking space topology. The obtained parking space topology structures are defined as a set of parking space topology structure information. A parking space information set is generated based on the parking space topology information set.

2. The method according to claim 1, wherein, The method further includes: The parking space information set is sent to the target vehicle, so that the target vehicle performs a parking operation based on the parking space information set.

3. The method according to claim 1, wherein, The parking space detection model was trained in the following way: Obtain a sample set, wherein the samples in the sample set include fisheye images of sample parking spaces and a set of sample parking space detection information corresponding to the fisheye images of sample parking spaces; Perform the following training steps based on the sample set: The fisheye image of the parking space of at least one sample in the sample set is input into the initial parking space detection model to obtain the parking space detection information set corresponding to each sample in the at least one sample. Compare the parking space detection information set corresponding to each of the at least one samples with the corresponding sample parking space detection information set; Based on the comparison results, determine whether the initial parking space detection model has achieved the preset optimization target; In response to the determination that the initial parking space detection model has achieved the optimization objective, the initial parking space detection model is determined as the trained parking space detection model.

4. The method according to claim 3, wherein, The steps for training the parking space detection model also include: In response to the determination that the initial parking space detection model has not achieved the optimization objective, the network parameters of the initial parking space detection model are adjusted, and a sample set is formed using unused samples. The adjusted initial parking space detection model is then used as the initial parking space detection model, and the training steps are executed again.

5. The method according to claim 1, wherein, The parking space detection information set further includes: parking space type information, parking space occupancy information, and a parking space vertex visibility information group, wherein the parking space vertex visibility information in the parking space vertex visibility information group corresponds to the parking space vertex grid coordinate information in the parking space vertex grid coordinate information group; and The step of generating a parking space information set based on the parking space topology information set includes: Based on the parking space topology information set, determine the parking space topology region set; Based on the parking space type information, the preset parking space area conditions, and the parking space topology area set, a parking space topology area set is generated; Each parking space topology region in the parking space topology region set is subjected to distortion region masking processing to obtain each parking space topology region after distortion region masking processing as a parking space region set. For each parking space region in the set of parking space regions, the parking space region is combined with the corresponding parking space occupancy information and parking space vertex visibility information to obtain a set of parking space information.

6. The method according to claim 1, wherein, The multi-scale feature map set includes a first-scale feature map, a second-scale feature map, and a third-scale feature map. The data dimension of the first-scale feature map is greater than the data dimension of the second-scale feature map, and the data dimension of the second-scale feature map is greater than the data dimension of the third-scale feature map. as well as The step of inputting the multi-scale feature map set into the feature fusion layer to obtain a multi-scale fused feature map set includes: The first-scale feature map is convolved to obtain the second-scale convolved feature map. The second-scale convolutional feature map is fused with the second-scale feature map to obtain the second-scale convolutional fused feature map; The second-scale convolutional fusion feature map is convolved to obtain the third-scale convolutional feature map. The third-scale convolutional feature map is fused with the third-scale feature map to obtain a third-scale convolutional fused feature map as the third fused feature map. The third-scale feature map is upsampled to obtain the second-scale upsampled feature map; The second scale upsampled feature map, the second scale feature map, and the second scale convolutional feature map are fused to obtain the second fused feature map. The upsampled feature map at the second scale is upsampled to obtain the upsampled feature map at the first scale. The upsampled feature map at the first scale is fused with the feature map at the first scale to obtain a first fused feature map. The first fusion feature map, the second fusion feature map, and the third fusion feature map are determined as a multi-scale fusion feature map set.

7. A parking space information generation device based on a fisheye camera, comprising: The device can implement the method as described in any one of claims 1-6; The acquisition unit is configured to acquire a fisheye image of a target parking space, wherein the fisheye image of the target parking space is a parking space image captured by a fisheye camera mounted on a target vehicle. The first input unit is configured to input the fisheye image of the target parking space into the backbone network layer of the pre-trained parking space detection model to obtain a multi-scale feature map set, wherein the parking space detection model further includes a feature fusion layer and a detection head layer; The second input unit is configured to input the multi-scale feature map set into the feature fusion layer to obtain a multi-scale fused feature map set. The third input unit is configured to input the multi-scale fused feature map set into the detection head layer to obtain a parking space detection information set. The first generation unit is configured to generate a parking space topology information set based on the parking space detection information set; The second generation unit is configured to generate a parking space information set based on the parking space topology information set.

8. An electronic device, comprising: One or more processors; Storage device, on which one or more programs are stored, When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any one of claims 1-6.

9. A computer-readable medium having a computer program stored thereon, wherein, When the computer program is executed by a processor, it implements the method as described in any one of claims 1-6.

Images