A parking space recognition method and device based on a multi-task model

Abstract

The application relates to a parking space recognition method and device based on a multitask model, which comprises the following steps: acquiring a parking lot image of a parking space to be recognized; using a target neural network model to perform target detection on each object contained in the parking lot image, so as to obtain a plurality of target object regions; using the target neural network model to perform instance segmentation and feature extraction on each target object region, so as to obtain target parking space entrance information; the target neural network model is a multitask model, and is trained by using a multitask adaptive loss function based on classification and regression; and the target parking space entrance information is marked on the parking lot image, so as to guide a vehicle to park. According to the scheme, the two tasks of target detection and feature extraction are integrated in the target neural network, and the multitask adaptive loss function is used to effectively balance the gradient and weight between the tasks, so that the model precision is improved without increasing the time complexity of the model algorithm.

Description

Technical Field

[0001] This invention relates to the field of vehicle technology, and in particular to a parking space recognition method and apparatus based on a multi-task model. Background Technology

[0002] As autonomous driving matures, automatic parking technology is finding increasingly important applications in the field of intelligent driving. Automatic parking refers to a vehicle automatically parking itself in a parking space without human intervention, using perception and control algorithms.

[0003] Parking space detection technology typically employs an end-to-end single-stage parking space detection neural network. This method uses a one-stage detection network to detect parking entrances, while simultaneously utilizing VGG-16 as the backbone network for feature extraction. Specifically, the backbone network simultaneously acquires both global and local semantic features of the parking spaces, and the fusion of these two pieces of information is used to obtain parking space information from the image.

[0004] However, this model has two problems: First, parking space detection includes segmentation and detection tasks. A single-task model only has a detection algorithm. Given a fixed underlying chip computing power, adding a segmentation task requires building a new segmentation network. Second, the model uses multiple model heads for task processing, but the weights of the loss function differ significantly during the training of each sub-task head. This increases the model's time complexity and reduces its accuracy. Summary of the Invention

[0005] This invention aims to at least solve one of the technical problems existing in the prior art. To this end, a first aspect of this invention proposes a parking space identification method based on a multi-task model, the method comprising:

[0006] Acquire parking lot images of the parking spaces to be identified;

[0007] The target neural network model is used to detect each object contained in the parking lot image, resulting in multiple target object regions;

[0008] The target neural network model is used to perform instance segmentation and feature extraction on each target object region to obtain the target parking space entrance information; the target parking space is an vacant parking space used for parking; the target neural network model is a multi-task model, trained using a multi-task adaptive loss function based on classification and regression.

[0009] The entrance information of the target parking space is marked on the parking lot image to guide vehicles to park.

[0010] Optionally, the target neural network model is trained using the following method:

[0011] The parking lot sample image dataset is input into the initial neural network model. The parking lot sample images are labeled with real object areas and real parking space entrance information.

[0012] Using the initial neural network model, target detection is performed on each object included in the parking lot sample image to obtain multiple predicted object regions;

[0013] Using the initial neural network model, global and local feature extraction is performed on each of the predicted object regions to obtain the predicted parking space entrance information;

[0014] Determine the difference between the predicted object region and the real object region to obtain the first residual;

[0015] The difference between the predicted parking space entrance information and the actual parking space entrance information is determined to obtain the second residual;

[0016] Using a pre-determined multi-task adaptive loss function, determine the target residuals corresponding to the first residual and the second residual;

[0017] Based on the target residual, the parameters of the initial neural network model are modified, and the initial neural network model is trained again until the target residual is less than a preset threshold. Then the training ends and the target neural network model is obtained.

[0018] Optionally, determining the target residuals corresponding to the first residual and the second residual using a pre-determined multi-task adaptive loss function includes:

[0019] Substituting the first residual and the second residual into a pre-determined multi-task adaptive loss function, the multi-task adaptive loss function is as follows:

[0020]

[0021] Where, σ i For the measurement noise of each task, L i (W) represents the square of the residuals corresponding to each task. For L i The weight term of (W), The weight term is a regularization term, and its magnitude is inversely proportional to the regularization term.

[0022] Optionally, the predicted parking space entrance information includes the predicted entrance corner direction and the predicted entrance width, and determining the difference between the predicted parking space entrance information and the actual parking space entrance information includes:

[0023] Determine the directional difference between the predicted entrance corner direction and the actual entrance corner direction in the actual parking space entrance information;

[0024] Determine the width difference between the predicted entrance width and the actual entrance width in the actual parking space entrance information;

[0025] The direction difference and the width difference are weighted and fused to obtain the difference between the predicted parking space entrance information and the actual parking space entrance information.

[0026] Optionally, the predicted entrance corner direction includes a predicted left corner direction and a predicted right corner direction, and the actual entrance corner direction includes a true left corner direction and a true right corner direction. Determining the direction difference between the predicted entrance corner direction and the actual entrance corner direction in the actual parking space entrance information includes:

[0027] Determine the left corner point difference between the predicted left corner point direction and the actual left corner point direction, and determine the right corner point difference between the predicted right corner point direction and the actual right corner point direction;

[0028] Determine the product of the L1 loss function in the direction of the left corner point and the difference of the left corner point, and determine the product of the L1 loss function in the direction of the right corner point and the difference of the right corner point, respectively, to obtain the first product and the second product;

[0029] The sum of the first product and the second product is determined to obtain the initial direction difference;

[0030] The sum of the initial direction difference and the preset constraint term is determined to obtain the direction difference between the predicted entrance corner direction and the actual entrance corner direction in the actual parking space entrance information;

[0031] The constraint term is used to constrain the predicted left corner point direction and the predicted right corner point direction to be approximately the same.

[0032] Optionally, the preset constraint is the sum of the absolute values ​​of the differences in the horizontal and vertical coordinates, where the difference in the horizontal coordinates is the difference between the horizontal coordinates of the predicted left corner point and the horizontal coordinates of the predicted right corner point, and the difference in the vertical coordinates is the difference between the vertical coordinates of the predicted left corner point and the vertical coordinates of the predicted right corner point.

[0033] A second aspect of the present invention provides a parking space recognition device based on a multi-task model, the device comprising:

[0034] The image acquisition module is used to acquire parking lot images of the parking spaces to be identified.

[0035] The target detection module is used to perform target detection on each object contained in the parking lot image using a target neural network model, and obtain multiple target object regions;

[0036] The instance segmentation module is used to segment and extract features from each target object region using the target neural network model to obtain target parking space entrance information; the target parking space is an vacant parking space used for parking; the target neural network model is a multi-task model, trained using a multi-task adaptive loss function based on classification and regression;

[0037] The labeling module is used to label the entrance information of the target parking space on the parking lot image to guide vehicles to park.

[0038] Optionally, the device further includes a model training module, the model training module being used for:

[0039] The parking lot sample image dataset is input into the initial neural network model. The parking lot sample images are labeled with real object areas and real parking space entrance information.

[0040] Using the initial neural network model, target detection is performed on each object included in the parking lot sample image to obtain multiple predicted object regions;

[0041] Using the initial neural network model, global and local feature extraction is performed on each of the predicted object regions to obtain the predicted parking space entrance information;

[0042] Determine the difference between the predicted object region and the real object region to obtain the first residual;

[0043] The difference between the predicted parking space entrance information and the actual parking space entrance information is determined to obtain the second residual;

[0044] Using a pre-determined multi-task adaptive loss function, determine the target residuals corresponding to the first residual and the second residual;

[0045] Based on the target residual, the parameters of the initial neural network model are modified, and the initial neural network model is trained again until the target residual is less than a preset threshold. Then the training ends and the target neural network model is obtained.

[0046] Optionally, the model training module is specifically used for:

[0047] Substituting the first residual and the second residual into a pre-determined multi-task adaptive loss function, the multi-task adaptive loss function is as follows:

[0048]

[0049] Where, σ i For the measurement noise of each task, L i (W) represents the square of the residuals corresponding to each task. For L i The weight term of (W), The weight term is a regularization term, and its magnitude is inversely proportional to the regularization term.

[0050] Optionally, the model training module is specifically used for:

[0051] Determine the directional difference between the predicted entrance corner direction and the actual entrance corner direction in the actual parking space entrance information;

[0052] Determine the width difference between the predicted entrance width and the actual entrance width in the actual parking space entrance information;

[0053] The direction difference and the width difference are weighted and fused to obtain the difference between the predicted parking space entrance information and the actual parking space entrance information.

[0054] Optionally, the predicted entry corner direction includes the predicted left corner direction and the predicted right corner direction, and the true entry corner direction includes the true left corner direction and the true right corner direction. The model training module is specifically used for:

[0055] Determine the left corner point difference between the predicted left corner point direction and the actual left corner point direction, and determine the right corner point difference between the predicted right corner point direction and the actual right corner point direction;

[0056] Determine the product of the L1 loss function in the direction of the left corner point and the difference of the left corner point, and determine the product of the L1 loss function in the direction of the right corner point and the difference of the right corner point, respectively, to obtain the first product and the second product;

[0057] The sum of the first product and the second product is determined to obtain the initial direction difference;

[0058] The sum of the initial direction difference and the preset constraint term is determined to obtain the direction difference between the predicted entrance corner direction and the actual entrance corner direction in the actual parking space entrance information;

[0059] The constraint term is used to constrain the predicted left corner point direction and the predicted right corner point direction to be approximately the same.

[0060] A third aspect of the present invention provides an electronic device comprising a processor and a memory, wherein the memory stores at least one instruction, at least one program, a code set, or an instruction set, wherein the at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by the processor to implement the parking space recognition method based on a multi-task model as described in the first aspect.

[0061] A fourth aspect of the present invention provides a computer-readable storage medium storing at least one instruction, at least one program, a code set, or an instruction set, wherein the at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by a processor to implement the parking space recognition method based on a multi-task model as described in the first aspect.

[0062] The embodiments of the present invention have the following beneficial effects:

[0063] This invention provides a parking space recognition method based on a multi-task model. The method acquires a parking lot image containing the parking space to be identified; it uses a target neural network model to detect objects within the parking lot image, obtaining multiple target object regions; it then uses the target neural network model to perform instance segmentation and feature extraction on each target object region to obtain target parking space entrance information; the target parking space is an available parking space for parking; the target neural network model is a multi-task model, trained using a classification-based and regression-based multi-task adaptive loss function; and the target parking space entrance information is labeled on the parking lot image to guide vehicle parking. This solution integrates target detection and feature extraction tasks into the target neural network, eliminating the need to build a new network. Furthermore, the use of a multi-task adaptive loss function effectively helps the model better balance gradients and weights between tasks, improving model accuracy without increasing the time complexity of the algorithm.

[0064] Additional aspects and advantages of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0065] To more clearly illustrate the technical solutions of the present invention, the accompanying drawings used in the description of the embodiments or prior art will be briefly introduced below. Obviously, the drawings described below are merely some embodiments of the present invention, and those skilled in the art can obtain other drawings based on these drawings without any creative effort.

[0066] Figure 1 A flowchart illustrating the steps of a first parking space identification method based on a multi-task model provided in an embodiment of the present invention;

[0067] Figure 2 This is a schematic diagram of a parking space marking image provided in an embodiment of the present invention;

[0068] Figure 3 A flowchart illustrating the steps of a target neural network model training method provided in an embodiment of the present invention;

[0069] Figure 4 A flowchart illustrating the steps of a second parking space identification method based on a multi-task model provided in an embodiment of the present invention;

[0070] Figure 5 This is a structural block diagram of a parking space recognition device based on a multi-task model, provided in an embodiment of the present invention. Detailed Implementation

[0071] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0072] This specification provides the operational steps for the methods described in the embodiments or flowcharts, but may include more or fewer operational steps based on conventional or non-inventive labor. In actual system or server product execution, the methods shown in the embodiments or drawings may be executed sequentially or in parallel (e.g., in a parallel processor or multi-threaded processing environment).

[0073] Figure 1 A flowchart illustrating the steps of a first parking space identification method based on a multi-task model provided in an embodiment of the present invention. The method may include the following steps:

[0074] Step 101: Obtain the parking lot image of the parking space to be identified.

[0075] After a vehicle enters the parking lot, its vehicle-mounted camera can capture real-time images of the parking lot surrounding the vehicle. These images may include both occupied and vacant parking spaces. Specifically, the parking lot image can be a bird's-eye view.

[0076] Step 102: Use a target neural network model to perform target detection on each object contained in the parking lot image to obtain multiple target object regions.

[0077] The target neural network model in this embodiment of the invention is a multi-task model capable of performing both target detection and instance segmentation tasks. This model integrates the network structures for both tasks, reducing the algorithmic complexity of model deployment within a given underlying chip computing power.

[0078] Parking lot images often contain many objects, such as vehicles, people, buildings, traffic signs, parking lines, etc. The purpose of object detection is to determine the location and category of each object, mark each different object with a detection box, and label the object category.

[0079] In this embodiment of the invention, the image region included in the detection box is the target object region.

[0080] Step 103: Use the target neural network model to perform instance segmentation and feature extraction on each target object region to obtain target parking space entrance information; the target parking space is an vacant parking space used for parking; the target neural network model is a multi-task model, trained using a multi-task adaptive loss function based on classification and regression.

[0081] Image instance segmentation is a further refinement of object detection, separating the foreground and background of an object to achieve pixel-level object separation.

[0082] Instance segmentation involves representing each object in a target object region with a different color and separating each object from the background. For example, if the target object region includes multiple parking lines, the individual parking spaces formed by these lines are marked with different colors.

[0083] Then, for each parking space, a convolutional neural network is used to extract the corresponding global and local semantic features. The final detection result, after combining the global and local semantic features, yields the entrance information of the vacant parking space. This entrance information is then deduplicated and output to obtain the target parking space entrance information. Specifically, the target parking space entrance information includes the direction and width of the entrance corner.

[0084] The target neural network model is a multi-task model. Compared with the single-task model, the multi-task model has the following advantages: multiple tasks share a single model, reducing memory usage; multiple tasks perform forward computations in one go, increasing inference speed; related tasks can complement each other by sharing information, thus improving each other's performance.

[0085] Furthermore, the target neural network model needs to handle instance segmentation and parking space target detection. Therefore, the model needs to perform multiple regression and classification tasks simultaneously. Observation revealed that assigning a loss weight to different tasks significantly affects the overall task performance. However, manually searching for these weights is extremely difficult and resource-intensive. To address this, this invention proposes a multi-task adaptive loss function based on classification and regression to learn these weights. This allows for the simultaneous learning of different parameters at different units or scales in classification and regression settings, thereby resolving cognitive uncertainty caused by a lack of training data and accidental uncertainty caused by the inability of training data to interpret information.

[0086] Experiments show that this model can learn multi-task weights and outperforms single-task independent modeling and training.

[0087] Step 104: Mark the entrance information of the target parking space on the parking lot image to guide vehicles to park.

[0088] Specifically, the target parking space entrance information includes the entrance corner direction and entrance width. The entrance corner direction and entrance width are marked in the target parking space in the parking lot image so that the vehicle's automatic parking system can use it as a parking position reference and automatically park the vehicle in the target parking space.

[0089] Figure 2 This is a schematic diagram of a parking space marking image provided in an embodiment of the present invention.

[0090] Reference Figure 2 The target neural network model identified two target parking spaces: parking space 1 is an angled parking space, and parking space 2 is a horizontal parking space.

[0091] A and B represent the left and right corner points of the entrance to parking space 1, respectively. The distance between the left corner point A and the right corner point B is the entrance width.

[0092] In summary, the parking space recognition method based on a multi-task model provided in this invention acquires a parking lot image of the parking space to be identified; extracts global and local semantic features contained in the parking lot image using a target neural network model; the global semantic features are used to determine the initial entrance corner information of the target parking space; the target parking space is an available parking space; based on the prior parking space entrance information included in the target neural network model and the local semantic features, the initial entrance corner information is corrected to obtain target entrance corner information, which includes the target entrance corner direction and target entrance width; the prior parking space entrance information includes the standard entrance corner direction and standard entrance width of a standard parking space; the target entrance corner direction and target entrance width are marked in the target parking space to guide vehicle parking. This solution uses prior parking space entrance information as guidance for parking space entrance recognition, reducing the computational load of the target neural network model and enabling the model to achieve real-time computing speed; furthermore, constraining the prediction results in the form of prior knowledge can improve the accuracy of parking space entrance recognition and effectively avoid the problems of false parking space detection and inaccurate direction.

[0093] Figure 3 A flowchart illustrating the steps of a target neural network model training method provided in an embodiment of the present invention. The method may include the following steps:

[0094] Step 201: Input the parking lot sample image dataset into the initial neural network model. The parking lot sample images are labeled with real object areas and real parking space entrance information.

[0095] The parking lot sample images can be real-world parking lot photos or images downloaded from online databases. This dataset includes annotation information for actual parking spaces, which can include the actual object area and entrance information for each parking space.

[0096] The initial neural network model uses MobileNetV2 as the feature backbone. During downsampling at 2x, 4x, 8x, 16x, and 32x, the model learns image features from shallow texture, color, and boundary information, while deeper layers learn semantic features. Since parking space entrance information is considered high-level feature information, this model uses a feature map downsampled to 32x size for feature extraction.

[0097] Of course, this feature backbone network can also be replaced according to the specific application scenario, for example, using VGG16 as the feature backbone network.

[0098] Step 202: Using the initial neural network model, target detection is performed on each object included in the parking lot sample image to obtain multiple predicted object regions.

[0099] The initial neural network model in this embodiment of the invention is a multi-task model, capable of performing object detection and instance segmentation tasks.

[0100] Parking lot images often contain many objects, such as vehicles, people, buildings, traffic signs, parking lines, etc. The purpose of object detection is to determine the location and category of each object, mark each different object with a detection box, and label the object category.

[0101] In this embodiment of the invention, the image region contained in the detection box output by the initial neural network model is the predicted object region.

[0102] Step 203: Using the initial neural network model, perform global feature extraction and local feature extraction on each of the predicted object regions to obtain the predicted parking space entrance information.

[0103] In this embodiment of the invention, the predicted object area is first divided into cells. A convolutional neural network can simultaneously obtain the global semantic features and local semantic features of each cell. By fusing the information of the two, the parking space information on the bird's-eye view can be obtained.

[0104] The output of the global semantic feature extraction branch consists of four parts. The first part represents whether each cell is an internal cell of a parking space. The second part represents the predicted category of the parking space for the internal cells. The third part represents the offset of the two corner points of a parking space entrance relative to the center point of each internal cell of the parking space. The fourth part represents whether the predicted parking space for the internal cells is occupied by another vehicle.

[0105] The parking space type and whether the parking space is occupied obtained by the global semantic extraction branch are accurate, but the prediction of the parking space entrance edge is not necessarily accurate. This is because the distance between the cell at different locations of different parking spaces and the corner point of the parking space entrance is different. In order to further improve the positioning accuracy of the corner point of the parking space entrance, we also need to use the local semantic information of the parking space and the prior parking space entrance information to adjust the initial entrance corner point information.

[0106] The output of the local semantic extraction branch consists of three parts: the first part represents whether each cell contains a parking space entrance corner point; the second part represents the offset of a parking space entrance corner point relative to the center point of its cell; and the third part represents the direction of the entrance corner point.

[0107] By fusing local semantic information, we can obtain the predicted parking space entrance information.

[0108] Step 204: Determine the difference between the predicted object region and the real object region to obtain the first residual.

[0109] Specifically, the difference between the detection box corresponding to the predicted object region and the detection box corresponding to the real object region is determined to obtain the first residual.

[0110] Step 205: Determine the difference between the predicted parking space entrance information and the actual parking space entrance information to obtain the second residual.

[0111] Specifically, the predicted parking space entrance information includes the predicted entrance corner direction and the predicted entrance corner width. The second residual is obtained based on the difference between the two dimensions of entrance corner direction and entrance corner width.

[0112] In one possible implementation, the predicted parking space entrance information includes the predicted entrance corner direction and the predicted entrance width, and step 205 includes:

[0113] Step 2051: Determine the direction difference between the predicted entrance corner direction and the actual entrance corner direction in the actual parking space entrance information;

[0114] Step 2052: Determine the width difference between the predicted entrance width and the actual entrance width in the actual parking space entrance information;

[0115] Step 2053: Weight and fuse the direction difference and the width difference to obtain the difference between the predicted parking space entrance information and the actual parking space entrance information.

[0116] In steps 2051-2053, the difference between the predicted entrance corner direction and the actual entrance corner direction, and the difference between the predicted entrance width and the actual entrance width are determined. The two differences are then weighted and fused to obtain the second residual.

[0117] The weights corresponding to the two differences can be preset based on experience.

[0118] Step 206: Using a pre-determined multi-task adaptive loss function, determine the target residuals corresponding to the first residual and the second residual.

[0119] Substituting the first and second residuals into the multi-task adaptive loss function allows for the simultaneous learning of different parameters at different units or scales in classification and regression settings. This addresses cognitive uncertainty caused by a lack of training data, as well as random uncertainty caused by the inability of training data to interpret information.

[0120] Step 207: Modify the parameters of the initial neural network model based on the target residual, and continue training the initial neural network model until the target residual is less than a preset threshold, then the training ends and the target neural network model is obtained.

[0121] Determine the relationship between the target residual and the preset residual threshold. If the target residual is greater than the residual threshold, modify the model parameters using the target residual. Then continue training using the dataset until the final target residual is less than the residual threshold, or until the predicted entry corner information obtained in the final prediction meets the preset accuracy. At this point, the training is complete, and the target neural network model is obtained.

[0122] In one possible implementation, step 206 includes:

[0123] Substituting the first residual and the second residual into a pre-determined multi-task adaptive loss function, the multi-task adaptive loss function is as follows:

[0124]

[0125] Where, σ i For the measurement noise of each task, L i (W) represents the square of the residuals corresponding to each task. For L i The weight term of (W), The weight term is a regularization term, and its magnitude is inversely proportional to the regularization term.

[0126] In this embodiment of the invention, the derivation of the multi-task adaptive loss function is as follows:

[0127] The multi-task loss function is determined based on Gaussian likelihood estimation that maximizes homoscedasticity uncertainty. For regression tasks, we have:

[0128] p(y|f w (x))=N(f w (x), σ 2 (2)

[0129] Where f w y represents the network output, i.e., the predicted value of the model output, w represents the weight of this output, and y represents the model task.

[0130] In this embodiment of the invention, the model task includes a classification task and a regression task, wherein instance segmentation is a classification task and object detection is a regression task.

[0131] For classification tasks:

[0132] p(y|f w (x))=Softmax(fw (x), σ 2 (3)

[0133] Furthermore, the Gaussian likelihood estimate for multi-task operations is:

[0134] p(y1…y k |f w (x))=p(y1|f w (x))…p(y k |f w (x)) (4)

[0135] For example, for regression tasks, maximum likelihood estimation is transformed into minimizing the negative logarithm:

[0136]

[0137] Among them, L i (W)=||y i -f w (x)|| 2 y i f is the true value corresponding to the model task. w (x) represents the predicted value for this task output by the model.

[0138] Based on (2), (3), (4), and (5), the probabilities of y1 and y2 for instance segmentation and parking space target detection tasks in this paper are as follows:

[0139]

[0140] Therefore, the multi-task adaptive loss function for parking space detection is designed as follows:

[0141]

[0142] in The weights of the loss corresponding to the task. This is the regularization term for the loss. When When it gets too big, The weight will decrease, and vice versa. When I was a child, The weight will increase. At the same time, The regularization settings can effectively address issues such as branch overfitting and severe training imbalance.

[0143] A key challenge in multi-task learning comes from the negative transfer problem, which means that blindly sharing and transferring knowledge across tasks can force inappropriate knowledge transfer between unrelated tasks, ultimately leading to overfitting.

[0144] In summary, this scheme employs the aforementioned multi-task adaptive loss function, effectively addressing the negative transfer problem in multi-task models, improving model accuracy, ensuring model robustness and effectiveness, and resulting in superior multi-task accuracy and efficiency compared to single-task models. Furthermore, when... When it gets too big, The weight will decrease when When I was a child, The weights will increase, thus achieving weight adaptation and greatly reducing the workload of manually searching for weights during network training.

[0145] In one possible implementation, such as Figure 4 As shown, the predicted entrance corner direction includes the predicted left corner direction and the predicted right corner direction, and the actual entrance corner direction includes the actual left corner direction and the actual right corner direction. Determining the direction difference between the predicted entrance corner direction and the actual entrance corner direction in the actual parking space entrance information includes the following steps 301-304:

[0146] Step 301: Determine the left corner point difference between the predicted left corner point direction and the true left corner point direction, and determine the right corner point difference between the predicted right corner point direction and the true right corner point direction;

[0147] Step 302: Determine the product of the L1 loss function in the direction of the left corner point and the difference of the left corner point, and determine the product of the L1 loss function in the direction of the right corner point and the difference of the right corner point, respectively, to obtain the first product and the second product;

[0148] Step 303: Determine the sum of the first product and the second product to obtain the initial direction difference;

[0149] Step 304: Determine the sum of the initial direction difference and the preset constraint term to obtain the direction difference between the predicted entrance corner direction and the actual entrance corner direction in the actual parking space entrance information; wherein, the constraint term is used to constrain the predicted left corner direction and the predicted right corner direction to be approximately the same.

[0150] The preset constraint term is the sum of the absolute values ​​of the difference between the horizontal coordinates and the difference between the vertical coordinates. The difference between the horizontal coordinates is the difference between the horizontal coordinate of the predicted left corner point and the horizontal coordinate of the predicted right corner point, and the difference between the vertical coordinates is the difference between the vertical coordinate of the predicted left corner point and the vertical coordinate of the predicted right corner point.

[0151] In steps 301-304, during the training process, two sets of directions need to be predicted for the two front corner points of each parking space (i.e., the left and right corner points of the parking space entrance). However, due to model prediction errors and viewpoint shifts, errors will occur in the two sets of directions. Therefore, it is necessary to constrain the direction errors of the two front corner points to improve the accuracy of the model.

[0152] Specifically, the direction difference between the predicted entrance corner direction and the actual entrance corner direction is determined according to the following formula:

[0153]

[0154] Where L is the L1 loss function, which minimizes the error, which is the sum of all absolute differences between the true and predicted values. pred y pred These are the predicted left and right corner direction vectors for the parking space entrance, respectively. true y true These are the true left corner direction vector and the true right corner direction vector of the parking space entrance, respectively.

[0155] This is a constraint term used to ensure that the predicted left corner point direction and the predicted right corner point direction are approximately the same. Specifically, It is represented as |jx1-jx2|+|jy1-jy2|, where (jx1,jy1) and (jx2,jy2) represent the directions of the left and right corner points, respectively.

[0156] In this embodiment of the invention, the aforementioned directional difference is added. The constraints ensure that the left and right corner directions of each parking space entrance are approximately the same, minimizing the difference between the two corner directions and reducing the error between the two corner directions of the parking space, thus improving the prediction accuracy of the model.

[0157] Figure 5 This is a structural block diagram of a parking space recognition device based on a multi-task model, provided in an embodiment of the present invention. The device 400 includes:

[0158] Image acquisition module 401 is used to acquire parking lot images of parking spaces to be identified;

[0159] The target detection module 402 is used to perform target detection on each object contained in the parking lot image using a target neural network model to obtain multiple target object regions;

[0160] The instance segmentation module 403 is used to perform instance segmentation and feature extraction on each of the target object regions using the target neural network model to obtain the target parking space entrance information; the target parking space is an vacant parking space used for parking; the target neural network model is a multi-task model, trained using a multi-task adaptive loss function based on classification and regression;

[0161] The annotation module 404 is used to annotate the entrance information of the target parking space on the parking lot image to guide vehicles to park.

[0162] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0163] In another embodiment of the present invention, a device is also provided, the device including a processor and a memory, the memory storing at least one instruction, at least one program, code set or instruction set, the at least one instruction, the at least one program, the code set or instruction set being loaded and executed by the processor to implement the parking space recognition method based on a multi-task model as described in the embodiment of the present invention.

[0164] In another embodiment of the present invention, a computer-readable storage medium is also provided, wherein at least one instruction, at least one program, code set or instruction set is stored in the storage medium, and the at least one instruction, the at least one program, the code set or instruction set is loaded and executed by a processor to implement the parking space recognition method based on a multi-task model as described in the embodiment of the present invention.

[0165] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of the present invention are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid state disk (SSD)).

[0166] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0167] The various embodiments in this specification are described in a related manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the system embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0168] The above description is merely a preferred embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention are included within the scope of protection of the present invention.

Claims

1. A parking space identification method based on a multi-task model, characterized in that, The method includes: Acquire parking lot images of the parking spaces to be identified; The target neural network model is used to detect each object contained in the parking lot image, resulting in multiple target object regions; The target neural network model is used to perform instance segmentation and feature extraction on each target object region to obtain the target parking space entrance information; the target parking space is an vacant parking space used for parking; the target neural network model is a multi-task model, trained using a multi-task adaptive loss function based on classification and regression. The entrance information of the target parking space is marked on the parking lot image to guide vehicles to park; The training process of the target neural network model includes: The difference between the predicted parking space entrance information and the actual parking space entrance information is determined to obtain the second residual; The predicted parking space entrance information includes the predicted entrance corner direction and the predicted entrance width. Determining the difference between the predicted parking space entrance information and the actual parking space entrance information includes: Determine the directional difference between the predicted entrance corner direction and the actual entrance corner direction in the actual parking space entrance information; Determine the width difference between the predicted entrance width and the actual entrance width in the actual parking space entrance information; The direction difference and the width difference are weighted and fused to obtain the difference between the predicted parking space entrance information and the actual parking space entrance information; The predicted entrance corner direction includes the predicted left corner direction and the predicted right corner direction, and the actual entrance corner direction includes the actual left corner direction and the actual right corner direction. Determining the direction difference between the predicted entrance corner direction and the actual entrance corner direction in the actual parking space entrance information includes: Determine the left corner point difference between the predicted left corner point direction and the actual left corner point direction, and determine the right corner point difference between the predicted right corner point direction and the actual right corner point direction; Determine the product of the L1 loss function in the direction of the left corner point and the difference of the left corner point, and determine the product of the L1 loss function in the direction of the right corner point and the difference of the right corner point, respectively, to obtain the first product and the second product; The sum of the first product and the second product is determined to obtain the initial direction difference; The sum of the initial direction difference and the preset constraint term is determined to obtain the direction difference between the predicted entrance corner direction and the actual entrance corner direction in the actual parking space entrance information; The constraint term is used to constrain the directions of the predicted left corner point and the predicted right corner point to be approximately the same; the preset constraint term is the sum of the absolute values ​​of the difference between the horizontal coordinates and the absolute values ​​of the difference between the vertical coordinates, wherein the difference between the horizontal coordinates is the difference between the horizontal coordinates of the predicted left corner point and the horizontal coordinates of the predicted right corner point, and the difference between the vertical coordinates is the difference between the vertical coordinates of the predicted left corner point and the vertical coordinates of the predicted right corner point.

2. The method according to claim 1, characterized in that, The target neural network model is trained using the following method: The parking lot sample image dataset is input into the initial neural network model. The parking lot sample images are labeled with real object areas and real parking space entrance information. Using the initial neural network model, target detection is performed on each object included in the parking lot sample image to obtain multiple predicted object regions; Using the initial neural network model, global and local feature extraction is performed on each of the predicted object regions to obtain the predicted parking space entrance information; Determine the difference between the predicted object region and the real object region to obtain the first residual; The difference between the predicted parking space entrance information and the actual parking space entrance information is determined to obtain the second residual; Using a pre-determined multi-task adaptive loss function, determine the target residuals corresponding to the first residual and the second residual; Based on the target residual, the parameters of the initial neural network model are modified, and the initial neural network model is trained again until the target residual is less than a preset threshold. Then the training ends and the target neural network model is obtained.

3. The method according to claim 2, characterized in that, The step of determining the target residuals corresponding to the first residual and the second residual using a pre-determined multi-task adaptive loss function includes: Substituting the first residual and the second residual into a pre-determined multi-task adaptive loss function, the multi-task adaptive loss function is as follows: in, Measurement noise for each task. Let be the square of the residuals corresponding to each task. for The weighting terms The weight term is a regularization term, and its magnitude is inversely proportional to the regularization term.

4. A parking space recognition device based on a multi-task model, characterized in that, The device includes: The image acquisition module is used to acquire parking lot images of the parking spaces to be identified. The target detection module is used to perform target detection on each object contained in the parking lot image using a target neural network model, and obtain multiple target object regions; The instance segmentation module is used to segment and extract features from each target object region using the target neural network model to obtain target parking space entrance information; the target parking space is an vacant parking space used for parking; the target neural network model is a multi-task model, trained using a multi-task adaptive loss function based on classification and regression; The labeling module is used to label the entrance information of the target parking space on the parking lot image to guide vehicles to park; The device further includes a model training module, which is used for: The difference between the predicted parking space entrance information and the actual parking space entrance information is determined to obtain the second residual; Also used for: Determine the directional difference between the predicted entrance corner direction and the actual entrance corner direction in the actual parking space entrance information; Determine the width difference between the predicted entrance width and the actual entrance width in the actual parking space entrance information; The direction difference and the width difference are weighted and fused to obtain the difference between the predicted parking space entrance information and the actual parking space entrance information; The predicted entry corner direction includes the predicted left corner direction and the predicted right corner direction, and the true entry corner direction includes the true left corner direction and the true right corner direction. The model training module is further used for: Determine the left corner point difference between the predicted left corner point direction and the actual left corner point direction, and determine the right corner point difference between the predicted right corner point direction and the actual right corner point direction; Determine the product of the L1 loss function in the direction of the left corner point and the difference of the left corner point, and determine the product of the L1 loss function in the direction of the right corner point and the difference of the right corner point, respectively, to obtain the first product and the second product; The sum of the first product and the second product is determined to obtain the initial direction difference; The sum of the initial direction difference and the preset constraint term is determined to obtain the direction difference between the predicted entrance corner direction and the actual entrance corner direction in the actual parking space entrance information; The constraint term is used to constrain the directions of the predicted left corner point and the predicted right corner point to be approximately the same; the preset constraint term is the sum of the absolute values ​​of the difference between the horizontal coordinates and the absolute values ​​of the difference between the vertical coordinates, wherein the difference between the horizontal coordinates is the difference between the horizontal coordinates of the predicted left corner point and the horizontal coordinates of the predicted right corner point, and the difference between the vertical coordinates is the difference between the vertical coordinates of the predicted left corner point and the vertical coordinates of the predicted right corner point.

5. The apparatus according to claim 4, characterized in that, The device further includes a model training module, which is used for: The parking lot sample image dataset is input into the initial neural network model. The parking lot sample images are labeled with real object areas and real parking space entrance information. Using the initial neural network model, target detection is performed on each object included in the parking lot sample image to obtain multiple predicted object regions; Using the initial neural network model, global and local feature extraction is performed on each of the predicted object regions to obtain the predicted parking space entrance information; Determine the difference between the predicted object region and the real object region to obtain the first residual; The difference between the predicted parking space entrance information and the actual parking space entrance information is determined to obtain the second residual; Using a pre-determined multi-task adaptive loss function, determine the target residuals corresponding to the first residual and the second residual; Based on the target residual, the parameters of the initial neural network model are modified, and the initial neural network model is trained again until the target residual is less than a preset threshold. Then the training ends and the target neural network model is obtained.

6. An electronic device, characterized in that, The electronic device includes a processor and a memory, wherein the memory stores at least one instruction, at least one program, a code set, or an instruction set, and the at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by the processor to implement the parking space recognition method based on a multi-task model as described in any one of claims 1-3.

7. A computer-readable storage medium, characterized in that, The storage medium stores at least one instruction, at least one program, code set, or instruction set, wherein the at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by a processor to implement the parking space recognition method based on a multi-task model as described in any one of claims 1-3.

Images