A parking space occupancy state detection method, device and electronic equipment

Abstract

The present application belongs to the field of image processing, and relates to a parking space occupation state detection method, device and electronic equipment, the method comprising: acquiring multiple AVM sample images under different parking scenes; determining and labeling multiple first-type key points of each visible parking space in each AVM sample image, determining and labeling multiple second-type key points corresponding to each visible vehicle, and obtaining an image sample set; training a deep learning model to obtain a key point detection model; inputting a real-time AVM image of a vehicle to be parked into the key point detection model to obtain a key point detection result; and judging and detecting the occupation state of the visible parking space according to whether the second-type key points in the key point detection result fall within the visible parking space area composed of the first-type key points to obtain a parking space occupation state detection result. The present application judges whether a parking space is occupied by whether the second-type key points fall within the visible parking space area composed of the first-type key points, thereby improving the accuracy and reliability of the parking space occupation state detection.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, and in particular to a parking space occupancy state detection method and device and electronic equipment. BACKGROUND

[0002] In recent years, with the rapid development of intelligent vehicle technology, automatic parking function is more and more favored by drivers. The automatic parking system can help the driver to park the vehicle more conveniently and safely, greatly improving the driving experience. However, accurately detecting the occupancy state of the parking space is a key prerequisite for the effective operation of the automatic parking system.

[0003] In related technologies, the image captured by the surround-view fisheye camera is usually used to detect whether the parking space is occupied during automatic parking. Generally, a deep learning model is used to detect the location of the parking space from the AVM (Around View Monitor) image composed of multiple images captured by the surround-view fisheye camera, and then a traditional image processing algorithm such as binary method is used to determine whether the parking space is idle, or a deep learning model is used to directly detect whether the parking space in the AVM image is occupied. In actual application, the first method using binary image processing cannot be applied to images captured and processed in different scenes, for example, the brightness of light can cause the previously debugged threshold parameters to fail. The second method has design limitations such as improper labeling of sample data used to train the deep learning model (for example, labeling a visually obstructed parking space as an idle parking space or a non-idle parking space completely depends on the judgment of the labeling personnel, without reliable judgment rules that conform to the actual occupancy of the parking space) or using only one task head to detect multiple categories of objects (for example, one detection head is used to detect the front, rear or side of the vehicle, and the different parts of the vehicle are all detected as a vehicle, which actually has a large difference in visual features, making it difficult for the model to learn common features). The trained deep learning model has large differences in accuracy when detecting different objects, and it is difficult to detect or easily misidentify objects with a small number of samples, resulting in low accuracy and reliability of the parking space occupancy state detection result, which is difficult to meet the actual detection requirements. SUMMARY

[0004] The present application provides a parking space occupancy state detection method, device and electronic equipment to solve the problem of low accuracy and reliability of traditional parking space occupancy state detection schemes.

[0005] In one aspect, the present application provides a parking space occupancy state detection method, comprising:

[0006] Obtaining multiple AVM sample images under different parking scenarios;

[0007] determine and label a plurality of first type key points corresponding to each visible parking space in each AVM sample image, and determine and label a plurality of second type key points corresponding to each visible vehicle in each AVM sample image according to the visible part of each visible vehicle in each AVM sample image, to obtain an image sample set;

[0008] train a pre-constructed deep learning model through the image sample set to obtain a key point detection model;

[0009] input a real-time AVM image of a vehicle to be parked into the key point detection model to obtain a key point detection result output by the key point detection model;

[0010] determine and label a plurality of first type key points corresponding to each visible parking space in each AVM sample image, and determine and label a plurality of second type key points corresponding to each visible vehicle in each AVM sample image according to the visible part of each visible vehicle in each AVM sample image, to obtain an image sample set;

[0011] According to the parking space occupancy state detection method provided by the application, the plurality of second type key points corresponding to each visible vehicle in each AVM sample image are determined according to the visible part of each visible vehicle in each AVM sample image, which comprises:

[0012] For any visible vehicle in the AVM sample image, if the visible part contains two wheels of the side surface, the second type key point is the wheel contact point corresponding to each wheel of the side surface;

[0013] If the visible part contains a vehicle head and at most contains one wheel of the side surface, the second type key point is the vehicle head visual contact point corresponding to the vehicle head;

[0014] If the visible part contains a vehicle tail and at most contains one wheel of the side surface, the second type key point is the vehicle tail visual contact point corresponding to the vehicle tail.

[0015] According to the parking space occupancy state detection method provided by the application, the deep learning model comprises a feature processing network and a plurality of key point detection heads;

[0016] The feature processing network is used for feature processing of an input image to obtain feature enhanced data;

[0017] The plurality of key point detection heads are used for detecting different types of key points in the input image according to the feature enhanced data, and the number of key points detected by at least part of the key point detection heads in the plurality of key point detection heads is different.

[0018] According to the parking space occupancy state detection method provided by the application, the plurality of key point detection heads comprise:

[0019] The first key point detection head is configured to detect a plurality of first type key points of a visible parking space in the input image.

[0020] The second key point detection head is configured to detect a vehicle head and a corresponding head visual grounding point of the vehicle head in the input image.

[0021] The third key point detection head is configured to detect a vehicle tail and a corresponding tail visual grounding point of the vehicle tail in the input image.

[0022] The fourth key point detection head is configured to detect two wheels on a visible side of a vehicle in the input image and corresponding wheel grounding points of the two wheels.

[0023] According to the parking space occupancy state detection method provided in the application, the feature processing network comprises:

[0024] The backbone network is configured to perform feature extraction on the input image to obtain initial feature data.

[0025] The neck network is configured to perform enhancement processing on the initial feature data to obtain feature enhancement data.

[0026] According to the parking space occupancy state detection method provided in the application, in the key point detection result, the first type key points comprise visible parking space key points, and the second type key points comprise key points of target visible parts.

[0027] According to whether the second type key points in the key point detection result fall in the parking space visible area composed of the first type key points, the occupancy state of the visible parking space in the real-time AVM image is detected to obtain a parking space occupancy state detection result, which comprises:

[0028] According to the visible parking space key points, the parking space visible area of each visible parking space in the real-time AVM image is determined.

[0029] For any target visible parking space, if all target visible parts do not overlap with the parking space visible area of the target visible parking space, the parking space occupancy state detection result is that the target visible parking space is in an idle state.

[0030] If at least one target visible part overlaps with the parking space visible area of the target visible parking space, and the key points of the visible part are in the parking space visible area of the target visible parking space, the parking space occupancy state detection result is that the target visible parking space is in an occupied state.

[0031] If at least one target visible part overlaps with the parking space visible area of the target visible parking space, and the key points of all target visible parts are not in the parking space visible area of the target visible parking space, the parking space occupancy state detection result is that the target visible parking space is in an idle state.

[0032] According to the parking space occupancy state detection method provided by the application, each group of first type key points contains four visible parking space key points.

[0033] According to the parking space occupancy state detection method provided by the application, the parking space visible area of each visible parking space in the real-time AVM image is determined according to the visible parking space key points, including:

[0034] The four visible parking space key points corresponding to each visible parking space are sequentially connected to obtain a quadrilateral with the four visible parking space key points as vertices.

[0035] The quadrilateral is used as the parking space visible area of each visible parking space in the real-time AVM image.

[0036] On the other hand, the application also provides a parking space occupancy state detection device, including:

[0037] The acquisition module is configured to acquire a plurality of AVM sample images under different parking scenes.

[0038] The labeling module is configured to determine and label a plurality of first type key points corresponding to each visible parking space in each AVM sample image, and determine and label a plurality of second type key points corresponding to each visible vehicle in each AVM sample image according to the visible part of each visible vehicle in each AVM sample image, to obtain an image sample set.

[0039] The training module is configured to train a pre-constructed deep learning model through the image sample set to obtain a key point detection model.

[0040] The input module is configured to input a real-time AVM image of a vehicle to be parked into the key point detection model to obtain a key point detection result output by the key point detection model.

[0041] The detection module is configured to determine and detect the occupancy state of a visible parking space in the real-time AVM image according to whether the second type key points in the key point detection result fall in the parking space visible area composed of the first type key points, to obtain a parking space occupancy state detection result.

[0042] On the other hand, the application also provides an electronic device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the parking space occupancy state detection method according to any one of the above when executing the program.

[0043] The application provides a parking space occupancy state detection method, device and electronic equipment, which comprises the following steps: acquiring a plurality of AVM sample images in different parking scenarios, determining and labeling a plurality of first type key points corresponding to each visible parking space in each AVM sample image, and determining and labeling a plurality of second type key points corresponding to each visible vehicle in each AVM sample image according to the visible part of each visible vehicle in each AVM sample image, so as to obtain an image sample set; training a pre-constructed deep learning model through the image sample set, so as to obtain a key point detection model; subsequently inputting a real-time AVM image of a to-be-parked vehicle into the key point detection model, so as to obtain a key point detection result; and finally determining and detecting the occupancy state of the visible parking space in the real-time AVM image according to whether the second type key points in the key point detection result fall in the parking space visible area formed by the first type key points, so as to obtain a parking space occupancy state detection result. Due to the sample labeling link, the plurality of first type key points corresponding to the visible parking space and the plurality of second type key points corresponding to the visible vehicle are used for data labeling, and whether the parking space is occupied is determined according to whether the second type key points fall in the parking space visible area formed by the first type key points, so that the accuracy and reliability of the parking space occupancy state detection result are improved. BRIEF DESCRIPTION OF DRAWINGS

[0044] In order to more clearly illustrate the technical solutions of the present application or prior art, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings in the following description are some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained without creative labor.

[0045] Figure 1 is a flowchart of the parking space occupancy state detection method provided by the embodiment of the present application;

[0046] Figure 2 is a schematic diagram of parking space structures of various shapes;

[0047] Figure 3 is a schematic diagram of parking space key points distribution;

[0048] Figure 4 is a schematic diagram of the position of the parking space visible area and the area where the vehicle head is located when the vehicle head is visible;

[0049] Figure 5 is a structural schematic diagram of the parking space occupancy state detection device provided by the embodiment of the present application;

[0050] Figure 6 is a structural schematic diagram of the electronic equipment provided by the embodiment of the present application. DETAILED DESCRIPTION

[0051] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0052] The following is combined Figures 1 to 6 This invention describes in detail the parking space occupancy status detection method, device, and electronic equipment provided in embodiments of the present invention.

[0053] Figure 1 This is a flowchart illustrating the parking space occupancy status detection method provided in an embodiment of the present invention.

[0054] like Figure 1 As shown, the parking space occupancy status detection method provided in this embodiment of the invention can be executed by an electronic device with data transmission and reception capabilities and data processing capabilities. The method mainly includes the following steps:

[0055] Step 110: Obtain multiple AVM sample images under different parking scenarios.

[0056] AVM stands for Around View Monitor, a panoramic monitoring imaging system. This system uses multiple ultra-wide-angle fisheye cameras installed around the vehicle to capture images, then performs distortion correction and stitching data processing on the captured images to ultimately create a panoramic image of the vehicle's surroundings. In this embodiment, the AVM sample images cover image samples from various parking scenarios, involving parking spaces of different shapes and parking situations under various conditions. Figure 2 Examples of several common parking space shapes are shown.

[0057] It should be noted that the AVM sample images are obtained by stitching together multiple original images taken by multiple fisheye cameras on both sides of the vehicle after distortion correction.

[0058] Step 120: Determine and label multiple first-type key points corresponding to each visible parking space in each AVM sample image, and based on the visible parts of each visible vehicle in each AVM sample image, determine and label multiple second-type key points corresponding to each visible vehicle in each AVM sample image to obtain an image sample set.

[0059] In this embodiment, the plurality of first type key points corresponding to the parking space can be determined in the determining link, and the two closest corner points of the vehicle in the parking space can be determined and labeled first, and then the remaining two points are determined and labeled according to the situation. Specifically, when a vehicle is in the parking space, the two intersection points of the vehicle and the parking line are labeled. When the parking space is in an idle state, the remaining two visible corner points of the parking space are labeled, so that the plurality of first type key points corresponding to each visible parking space can be obtained. In actual application, the number of first type key points can be four or more than four.

[0060] It should be noted that the plurality of second type key points corresponding to the visible vehicle need to be determined according to the visible part of the visible vehicle, which can be the contact points of the two wheels on the side, or the two visual contact points of the vehicle head, or the two visual contact points of the vehicle tail.

[0061] In actual application, taking the labeling of four first type key points as an example, the four first type key points corresponding to the visible parking space can be numbered in the counterclockwise direction, and the numbers of the two first type key points corresponding to the edge of the parking space close to the vehicle are 1 and 4, and the numbers of the remaining two first type key points are 2 and 3, as shown in Figure 3 The left side of the parking space shows the labeling state of the four first type key points when the parking space is idle, and the right side shows the labeling state of the four first type key points when the vehicle is in the parking space.

[0062] When labeling the second type key points, the wheel, tail or head of the vehicle on which the second type key points are located can be labeled with a rectangular frame, and the corresponding two key points are labeled, which are used to train the model to detect key points on different objects. It can be understood that by labeling the wheel, tail or head of the vehicle with a rectangular frame, the model can be trained to detect different objects. In actual detection process, the model first detects different objects, and then detects the key points on each object. This task decomposition type top-down design helps to improve the ability of the model to detect key points on different objects.

[0063] Step 140: training the pre-constructed deep learning model through the image sample set to obtain a key point detection model.

[0064] After the data labeling is completed, an image sample set with labels can be obtained, and then a key point detection model can be trained. It can be understood that the input data of the key point detection model is an input image, that is, a real-time AVM image, and the output data is a key point detection result containing key parts and key points.

[0065] Step 150: inputting the real-time AVM image of the vehicle to be parked into the key point detection model to obtain the key point detection result output by the key point detection model.

[0066] It can be understood that the real-time AVM image is an image obtained after splicing an image captured by a fisheye camera installed on the vehicle to be parked.

[0067] Step 160: According to the key point detection result, the occupancy state of the visible parking space in the real-time AVM image is detected to obtain a parking space occupancy state detection result.

[0068] The scheme provided in this embodiment can obtain more accurate key point detection results because the key point detection model applied in the key point detection link is trained by the image sample set generated according to the improved data labeling scheme. Subsequently, whether the second type of key point falls within the parking space visible area formed by the first type of key point is used to determine whether the parking space is occupied. Compared with the technical scheme of directly labeling the parking space as idle or occupied in the image data to train the model to detect the parking space occupancy state, the embodiment can avoid the misjudgment that the target parking space is detected as occupied when the target parking space is blocked by a vehicle in the adjacent parking space, and the detection result is more accurate and reliable.

[0069] In an embodiment, according to the visible part of each visible vehicle in each AVM sample image, a plurality of second type key points corresponding to each visible vehicle in each AVM sample image are determined, specifically including:

[0070] For any visible vehicle in the AVM sample image, if the visible part contains two wheels on the side, the second type key point is the wheel contact point corresponding to each wheel on the side.

[0071] If the visible part contains a vehicle head and at most one wheel on the side, the second type key point is the vehicle head visual contact point corresponding to the vehicle head.

[0072] If the visible part contains a vehicle tail and at most one wheel on the side, the second type key point is the vehicle tail visual contact point corresponding to the vehicle tail.

[0073] It can be understood that for any visible vehicle in the AVM sample image, if the visible part contains two wheels on the side, that is, both wheels on the side of the visible vehicle are visible, then the two wheels on the side and the wheel contact points corresponding to the two wheels can be labeled subsequently. If the visible part contains only one wheel on the side or does not contain a wheel on the side, that is, only one of the two wheels on the side is visible or both are not visible, then the visible vehicle head and the vehicle head visual contact point can be labeled, or the visible vehicle tail and the vehicle tail visual contact point can be labeled.

[0074] It can be found that the key point labeling method adopted in the embodiment can adapt to parking spaces of different shapes and has stronger generalization ability. As long as a certain number of AVM sample images of each shape of parking space are collected for labeling and model training, one model can stably detect parking spaces of multiple shapes without the need to spend a lot of effort to label details such as different types of line segments and line segment angles of the parking space, and the data labeling link is more convenient, efficient and accurate.

[0075] In an embodiment, the deep learning model specifically comprises a feature processing network and a plurality of key point detection heads.

[0076] The feature processing network is configured to perform feature processing on the input image to obtain feature enhancement data.

[0077] The plurality of key point detection heads are configured to detect different types of key points in the input image according to the feature enhancement data, and at least part of the key point detection heads in the plurality of key point detection heads detect different numbers of key points.

[0078] In a specific implementation, the plurality of key point detection heads specifically comprises:

[0079] The first key point detection head is configured to detect a plurality of first type key points of a visible parking space in the input image.

[0080] The second key point detection head is configured to detect a vehicle head and a corresponding vehicle head visual contact point in the input image.

[0081] The third key point detection head is configured to detect a vehicle tail and a corresponding vehicle tail visual contact point in the input image.

[0082] The fourth key point detection head is configured to detect two vehicle wheels on a visible vehicle side in the input image and corresponding wheel contact points of the two vehicle wheels.

[0083] In actual application, considering that a traditional deep learning model only supports configuring one key point detection task head to detect key points of multiple different categories of objects, but the number of key points must remain consistent. However, the parking space to be detected in the embodiment is 4 key points, and the visual contact points of the vehicle head, the vehicle tail, or the vehicle wheel on the side to be detected are 2 key points, so the traditional deep learning model cannot be directly applied to the present scheme.

[0084] At the same time, considering that when one key point detection head detects multiple different categories of objects, there is a large difference in detection accuracy when the trained model detects different categories of objects due to the difference in the number of sample data of different categories of objects.

[0085] Accordingly, the embodiment modifies the network structure of the traditional deep learning model, and configures four key point detection heads for the model, which are respectively used for detecting the parking space, the vehicle head and the visual contact point of the vehicle head, the vehicle tail and the visual contact point of the vehicle tail, the two wheels on the side and the contact points of the wheels, solves the defect that the traditional deep learning model cannot configure multiple key point detection heads at the same time, and removes the constraint condition that the number of key points of different objects is required to be the same when detecting the key points of different categories of objects, realizes that multiple key point detection heads can be configured, each key point detection head can detect the key points of different categories of objects, and different numbers of key points can be detected. In this way, compared with the traditional deep learning model, while improving the detection accuracy of each object and its key points, the unreasonable restriction that the number of key points of different objects is required to be the same is removed, and the above modification improves the accuracy and flexibility of the key point detection link.

[0086] In one specific implementation, the feature processing network specifically includes:

[0087] a backbone network configured to extract features from the input image to obtain initial feature data;

[0088] a neck network configured to perform enhancement processing on the initial feature data to obtain feature enhancement data.

[0089] In actual application, the backbone network is usually composed of a convolutional neural network, such as ResNet (Residual Network), VGG (Visual Geometry Group Network) network structure, etc., and its function is to extract features from the input image, convert the input image into initial feature data, and extract basic features such as edges and textures in the input image.

[0090] The neck network is mainly used to receive the initial feature data output by the backbone network, and further fuse and enhance the features, such as using a feature pyramid network structure, to improve the expression ability of the features, and obtain feature enhancement data.

[0091] The embodiment adopts the method of labeling and recognizing the vehicle head, the vehicle tail or the two wheels on the side and their respective contact points, which can more stably and accurately detect the visible parts of the vehicle in the real-time AVM image and their contact points.

[0092] Firstly, compared with the way of marking and identifying the vehicle as a whole, the marking and identifying scheme of the vehicle head, tail, side two wheels of different categories of objects is more accurate and stable; secondly, after the subdivision of the vehicle head, tail and side two wheels in target detection, the deep learning model can better learn the common features of each category, and has better discovery ability and higher accuracy when detecting the corresponding grounding points, thereby improving the detection accuracy of the key point detection model.

[0093] In an embodiment, in the key point detection result, the first type of key points include visible parking space key points, and the second type of key points include key points of target visible parts.

[0094] According to whether the second type of key points in the key point detection result falls within the parking space visible area composed of the first type of key points, the occupancy state of the visible parking space in the real-time AVM image is detected to obtain a parking space occupancy state detection result, which specifically includes:

[0095] First, according to the visible parking space key points, the parking space visible area of each visible parking space in the real-time AVM image is determined.

[0096] In this embodiment, each group of first type of key points includes four visible parking space key points.

[0097] In one specific implementation, according to the visible parking space key points, the parking space visible area of each visible parking space in the real-time AVM image is determined, which specifically includes:

[0098] The four visible parking space key points corresponding to each visible parking space are sequentially connected to obtain a quadrilateral with the four visible parking space key points as vertices, and the quadrilateral is taken as the parking space visible area of each visible parking space in the real-time AVM image.

[0099] Figure 4 An example is shown to show the case where the vehicle head is visible, and the parking space visible area of the quadrilateral enclosed by the four connecting lines corresponding to the four visible parking space key points P1, P2, P3, P4 of the visible parking space in the real-time AVM image is shown.

[0100] In some embodiments, the visible parking space key points can also be more than four, and accordingly, the parking space visible area is a polygon composed of multiple parking space key points, and the number of sides of the polygon corresponds to the number of parking space key points.

[0101] In one case, for any target visible parking space, if all target visible parts do not overlap with the parking space visible area of the target visible parking space, the parking space occupancy state detection result is that the target visible parking space is in an idle state.

[0102] In another case, if at least one target visible part overlaps with the parking space visible area of the target visible parking space, and the key points of the visible part are within the parking space visible area of the target visible parking space, the parking space occupancy state detection result is that the target visible parking space is in an occupied state.

[0103] In another case, if at least one target visible part overlaps with the parking space visible area of the target visible parking space, and the key points of all target visible parts are not within the parking space visible area of the target visible parking space, the parking space occupancy state detection result is that the target visible parking space is in an idle state.

[0104] As shown in Figure 4 Taking the target visible part as a vehicle head as an example, the rectangular frame is the area where the vehicle head is located, and can represent the position of the vehicle head. The four key points P1, P2, P3, and P4 correspond to four lines that form a trapezoidal parking space visible area 210. At this time, the target visible part shown by the rectangular frame overlaps with the parking space visible area 210, and the two vehicle head visual contact points P5 and P6 are within the parking space visible area 210. Therefore, it is concluded that the current parking space is in an occupied state.

[0105] It is considered that the distortion and stretching in the AVM image synthesized by the fisheye camera often causes a vehicle to block several adjacent parking spaces, resulting in a false impression that the adjacent parking spaces are all detected as occupied. After training a deep learning model using the two categories of labeled empty parking space and non-empty parking space data, the deep learning model cannot solve the above problems because the image features of a parking space partially blocked by a vehicle in an adjacent parking space are similar to those of a parking space with a vehicle actually parked on it. The model cannot distinguish the feature differences between the two, and the two categories are easily confused with each other during detection. The probability of false detection is higher when using the traditional binary image processing and pixel value judgment to determine whether a parking space is empty.

[0106] Therefore, the embodiment determines whether the visible parking space is actually occupied by a vehicle by judging whether the contact points of the vehicle head or tail or wheel are within the quadrilateral of the visible part of the parking space. Since the positions of the detected contact points of the vehicle head, tail, and wheel are relatively accurate, the state of the vehicle occupying the parking space can be accurately detected, and the occupancy state of the adjacent parking space partially blocked by the vehicle can be accurately determined.

[0107] In some embodiments, when it is detected in the parking space occupancy state detection result that there is a parking space in an idle state, the parking space in the idle state can be taken as a target parking space, and it is further detected whether there is other obvious foreign matter in the target parking space visible area, such as a ground lock in a car blocking state, a large stone, a garbage can and the like. Since the parking space visible area in the embodiment is obtained by a quadrangle composed of four visible parking space key points, the parking space visible range can be accurately positioned from the perspective of the vehicle to be parked, and foreign matter detection is further performed in the parking space visible range, which reduces the detection operation amount while improving the safety of the parking link.

[0108] In actual application, an algorithm based on analysis of image pixel values can be specifically used to identify foreign matter that may occupy the parking space. For example, the pixel mean and variance of all pixel points in the parking space visible area of the target parking space can be compared with the pixel mean and variance in the corresponding area of the parking space image in the normal idle state. If it is detected that the pixel mean and variance in the visible area of the target parking space are higher than the difference threshold compared with the pixel mean and variance of the idle parking space, it is indicated that the color distribution of part of the area in the parking space visible area of the target parking space is abnormal, and the image color distribution of the target parking space is greatly different from that of the normal idle parking space. Therefore, the target parking space can be defined as an abnormal parking space, and a prompt information can be sent to prompt the driver to further investigate whether there is foreign matter other than the vehicle in the target parking space.

[0109] In summary, the parking space occupancy detection method provided by the embodiment can detect the key points of the visible part of each parking space in the AVM measured image and the tail or head or side wheel of each vehicle and the corresponding grounding point at one time through the key point detection model capable of detecting multiple targets and their respective key points, and then determine whether each parking space is occupied by the algorithm for judging whether each parking space is occupied. The entire execution process is more efficient and fast, the detection result is more accurate than that of the usual detection method, and the detection efficiency is effectively improved.

[0110] Based on the same inventive concept, the application also protects a parking space occupancy state detection device. The parking space occupancy state detection device provided by the application is described below, and the parking space occupancy state detection device described below can be correspondingly referred to the parking space occupancy state detection method described above.

[0111] As shown in Figure 5 The parking space occupancy state detection device provided by the embodiment of the application specifically includes:

[0112] The acquisition module 310 is configured to acquire a plurality of AVM sample images under different parking scenarios.

[0113] The labeling module 320 is configured to determine and label a plurality of first type key points corresponding to each visible parking space in each AVM sample image, and determine and label a plurality of second type key points corresponding to each visible vehicle in each AVM sample image according to visible parts of each visible vehicle in each AVM sample image, to obtain the image sample set.

[0114] The training module 330 is configured to train a pre-constructed deep learning model by using the image sample set, to obtain a key point detection model.

[0115] The input module 340 is configured to input a real-time AVM image of a vehicle to be parked into the key point detection model, to obtain a key point detection result output by the key point detection model.

[0116] The detection module 350 is configured to determine and detect an occupancy state of a visible parking space in the real-time AVM image according to whether the second type key points in the key point detection result fall in a parking space visible area formed by the first type key points, to obtain a parking space occupancy state detection result.

[0117] As to the device in the above-mentioned embodiments, the specific manner in which each module performs operations has been described in detail in the embodiments of the method, and will not be described in detail here.

[0118] Figure 6 FIG. 1 is a structural schematic diagram of an electronic device provided by an embodiment of the present application.

[0119] As Figure 6As shown, the electronic device can include a processor 410, a communications interface 420, a memory 430, and a communications bus 440, wherein the processor 410, the communications interface 420, and the memory 430 complete mutual communication through the communications bus 440. The processor 410 can invoke a logic instruction in the memory 430 to execute the parking space occupancy state detection method provided by each of the above embodiments, including: acquiring a plurality of AVM sample images under different parking scenarios; determining and labeling a plurality of first type key points corresponding to each visible parking space in each AVM sample image, and determining and labeling a plurality of second type key points corresponding to each visible vehicle in each AVM sample image according to the visible parts of each visible vehicle in each AVM sample image, to obtain an image sample set; training a pre-constructed deep learning model through the image sample set to obtain a key point detection model; inputting a real-time AVM image of a vehicle to be parked into the key point detection model to obtain a key point detection result output by the key point detection model; and according to whether the second type key points in the key point detection result fall in a parking space visible area composed of the first type key points, detecting and judging the occupancy state of the visible parking space in the real-time AVM image to obtain a parking space occupancy state detection result.

[0120] In addition, the logic instruction in the memory 430 described above can be implemented in the form of a software function unit and sold or used as an independent product, and can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the present application essentially or the part that contributes to the prior art or part of the technical solutions can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a plurality of instructions to make a computer device (which can be a personal computer, a server, or a network device, etc.) execute all or part of the steps of the method described in each embodiment of the present application. The foregoing storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various program code storage media.

[0121] In another aspect, the present application also provides a computer program product comprising a computer program, which can be stored on a non-transitory computer-readable storage medium, and the computer program can be executed by a processor to enable a computer to perform the parking space occupancy state detection method provided by any of the above embodiments, including: obtaining a plurality of AVM sample images under different parking scenarios; determining and labeling a plurality of first type key points corresponding to each visible parking space in each AVM sample image, and determining and labeling a plurality of second type key points corresponding to each visible vehicle in each AVM sample image according to the visible parts of each visible vehicle in each AVM sample image, to obtain an image sample set; training a pre-constructed deep learning model through the image sample set to obtain a key point detection model; inputting a real-time AVM image of a vehicle to be parked into the key point detection model to obtain a key point detection result output by the key point detection model; and determining and detecting the occupancy state of a visible parking space in the real-time AVM image according to whether the second type key points in the key point detection result fall within a parking space visible area formed by the first type key points, to obtain a parking space occupancy state detection result.

[0122] In another aspect, the present application also provides a non-transitory computer-readable storage medium having a computer program stored thereon, and the computer program can be executed by a processor to implement the parking space occupancy state detection method provided by any of the above embodiments, including: obtaining a plurality of AVM sample images under different parking scenarios; determining and labeling a plurality of first type key points corresponding to each visible parking space in each AVM sample image, and determining and labeling a plurality of second type key points corresponding to each visible vehicle in each AVM sample image according to the visible parts of each visible vehicle in each AVM sample image, to obtain an image sample set; training a pre-constructed deep learning model through the image sample set to obtain a key point detection model; inputting a real-time AVM image of a vehicle to be parked into the key point detection model to obtain a key point detection result output by the key point detection model; and determining and detecting the occupancy state of a visible parking space in the real-time AVM image according to whether the second type key points in the key point detection result fall within a parking space visible area formed by the first type key points, to obtain a parking space occupancy state detection result.

[0123] The device embodiments described above are only schematic, wherein the units shown as separate components can or can not be physically separate, and the components shown as units can or can not be physical units, i.e., can be located in one place or distributed on a plurality of network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the present embodiment. Those skilled in the art can understand and implement without creative labor.

[0124] Those skilled in the art can clearly understand the technical solutions of the various embodiments from the above description of the embodiments, and the various embodiments can be implemented by means of software with the necessary general hardware platforms, and of course, can also be implemented by hardware. Based on such understanding, the above technical solutions, essentially or in other words, the part of the prior art that makes a contribution, can be embodied in the form of a software product, which can be stored in a computer readable storage medium, such as a ROM / RAM, a magnetic disk, an optical disk, and the like, and includes a number of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0125] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, rather than limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement for some technical features therein; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A method of detecting a parking space occupancy status, characterized by, The method comprises the following steps: acquiring a plurality of AVM sample images in different parking scenarios; determining and labeling a plurality of first-type key points corresponding to each visible parking space in each AVM sample image, and determining and labeling a plurality of second-type key points corresponding to each visible vehicle in each AVM sample image according to the visible parts of each visible vehicle in each AVM sample image, to obtain an image sample set; in the determining step, first, only the two closest corner points of the vehicle in the parking space are determined and labeled, and then the remaining two points are determined and labeled; for any visible vehicle in the AVM sample image, if the visible parts include two wheels on the side, the second-type key points are the wheel contact points corresponding to the two wheels on the side; if the visible parts include a vehicle head and at most one wheel on the side, the second-type key points are the vehicle head visual contact points corresponding to the vehicle head; if the visible parts include a vehicle tail and at most one wheel on the side, the second-type key points are the vehicle tail visual contact points corresponding to the vehicle tail; training a pre-constructed deep learning model through the image sample set, to obtain a key point detection model; inputting a real-time AVM image of a vehicle to be parked into the key point detection model, to obtain a key point detection result output by the key point detection model; determining and detecting the occupancy state of a visible parking space in the real-time AVM image according to whether the second-type key points in the key point detection result fall in the visible area of the parking space composed of the first-type key points, to obtain a parking space occupancy state detection result.

2. The parking space occupancy state detection method according to claim 1, characterized in that, The deep learning model comprises a feature processing network and a plurality of key point detection heads. The feature processing network is configured to perform feature processing on an input image, to obtain feature enhanced data. The plurality of key point detection heads are configured to detect different types of key points in an input image according to the feature enhanced data, and at least part of the key point detection heads in the plurality of key point detection heads detect different numbers of key points.

3. The parking space occupancy state detection method according to claim 2, characterized in that, The plurality of key point detection heads comprise: a first key point detection head configured to detect a plurality of first-type key points of a visible parking space in an input image; a second key point detection head configured to detect a vehicle head and vehicle head visual contact points corresponding to the vehicle head in an input image; a third key point detection head configured to detect a vehicle tail and vehicle tail visual contact points corresponding to the vehicle tail in an input image; a fourth key point detection head configured to detect two wheels on the side of a visible vehicle in an input image and wheel contact points corresponding to the two wheels.

4. The parking space occupancy state detection method according to claim 2, characterized by, The feature processing network comprises: a backbone network configured to perform feature extraction on an input image, to obtain initial feature data; a neck network configured to perform enhancement processing on the initial feature data, to obtain feature enhanced data.

5. The parking space occupancy detection method according to claim 1, characterized in that, In the key point detection result, the first-type key points comprise visible parking space key points, and the second-type key points comprise key points of target visible parts; determining and detecting the occupancy state of a visible parking space in the real-time AVM image according to whether the second-type key points in the key point detection result fall in the visible area of the parking space composed of the first-type key points, to obtain a parking space occupancy state detection result, which comprises: According to the visible parking space key points, a parking space visible area of each visible parking space in the real-time AVM image is determined; For any target visible parking space, if all target visible parts do not overlap with the parking space visible area of the target visible parking space, the parking space occupancy state detection result is that the target visible parking space is in an idle state; If at least one target visible part overlaps with the parking space visible area of the target visible parking space, and the key points of the target visible parts are in the parking space visible area of the target visible parking space, the parking space occupancy state detection result is that the target visible parking space is in an occupied state; If at least one target visible part overlaps with the parking space visible area of the target visible parking space, and the key points of all target visible parts are not in the parking space visible area of the target visible parking space, the parking space occupancy state detection result is that the target visible parking space is in an idle state.

6. The parking space occupancy detection method according to claim 5, characterized in that Each first-type key point group contains four visible parking space key points.

7. The parking space occupancy status detection method according to claim 6, characterized in that According to the visible parking space key points, a parking space visible area of each visible parking space in the real-time AVM image is determined, including: Connecting the four visible parking space key points corresponding to each visible parking space in sequence to obtain a quadrilateral with the four visible parking space key points as vertices; The quadrilateral is taken as the parking space visible area of each visible parking space in the real-time AVM image.

8. A parking space occupancy state detection apparatus characterized by comprising: It includes: An acquisition module is configured to acquire a plurality of AVM sample images under different parking scenarios; An annotation module is configured to determine and annotate a plurality of first-type key points corresponding to each visible parking space in each AVM sample image, and determine and annotate a plurality of second-type key points corresponding to each visible vehicle in each AVM sample image according to visible parts of each visible vehicle in each AVM sample image to obtain an image sample set; in the determining link, first, only the two visible corner points closest to the vehicle in the parking space are determined and annotated, and then the remaining two points are determined and annotated; for any visible vehicle in the AVM sample image, if the visible part contains two wheels on the side, the second-type key point is a wheel contact point corresponding to each wheel on the side; if the visible part contains a vehicle head and at most one wheel on the side, the second-type key point is a vehicle head visual contact point corresponding to the vehicle head; if the visible part contains a vehicle tail and at most one wheel on the side, the second-type key point is a vehicle tail visual contact point corresponding to the vehicle tail; A training module is configured to train a pre-constructed deep learning model through the image sample set to obtain a key point detection model; An input module is configured to input a real-time AVM image of a vehicle to be parked into the key point detection model to obtain a key point detection result output by the key point detection model; A detection module is configured to determine and detect an occupancy state of a visible parking space in the real-time AVM image according to whether the second-type key points in the key point detection result fall in a parking space visible area composed of first-type key points to obtain a parking space occupancy state detection result.

9. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor implements the parking space occupancy state detection method in any one of claims 1 to 7 when executing the program.

Images