Model training and automatic parking method, automatic parking system and computer medium

Abstract

The invention discloses a model training and automatic parking method, an automatic parking system and a computer medium, and the training method comprises the steps: obtaining a source model trained on a common data set and a first model parameter of the source model, and adjusting a model structure of the source model and the first model parameter to construct a target model; respectively inputting the training data set into a target model and a source model to obtain first feature data output by the source model and second feature data output by the target model; and constructing a total loss function of the target model based on difference data of the first feature data and the second feature data, so as to train the target model based on the total loss function. Therefore, the total loss function is constructed through the difference data of the first feature data and the second feature data, so that the target model can learn partial feature processing capability of the source model, the target model can quickly adapt to the target task, data acquisition and annotation of the target model in a parking scene are reduced, and meanwhile, the robustness of the target model is improved.

Description

Technical Field

[0001] This application relates to the technical field of automatic parking, and particularly to model training and automatic parking methods, automatic parking systems, and computer media. Background Art

[0002] An Advanced Driver Assistance Systems (ADAS) is an active safety function integrated control system that can perceive and collect information about the vehicle's surrounding environment through in-vehicle sensors, analyze the system, and make decisions, which can pre-emptively avoid risks and improve driving safety and comfort. With the increasing demand for automobiles, the number of vehicles in use continues to grow, resulting in problems such as traffic congestion, shortage of urban parking space resources, and small parking space sizes. Therefore, it poses a greater challenge to automatic parking.

[0003] Existing automatic parking methods usually use the information collected from fisheye images captured by fisheye cameras for predictive analysis to obtain parkable spaces. Due to the ultra-wide-angle characteristics of fisheye cameras, fisheye images have a large distortion compared to conventional images. Therefore, if one wants to train a model in an automatic parking scenario, a large number of fisheye images need to be collected and labeled, resulting in a large amount of manpower and financial resources being spent. Summary of the Invention

[0004] To solve the above technical problems, this application provides model training and automatic parking methods, automatic parking systems, and computer media.

[0005] To solve the above problems, this application provides a first technical solution: providing a model training method, including: obtaining a source model trained on a public dataset and first model parameters of the source model, adjusting the model structure and the first model parameters of the source model to construct a target model to be trained; obtaining a training dataset, inputting the training dataset into the target model and the source model respectively to obtain first feature data output by the source model and second feature data output by the target model; constructing a total loss function of the target model based on the difference data between the first feature data and the second feature data, and training the target model based on the total loss function.

[0006] Optionally, the first feature data is the first feature map output by the feature processing network of the source model, and the second feature data is the second feature map output by the feature processing network of the target model; constructing the total loss function of the target model based on the difference data between the first feature data and the second feature data includes: calculating the relative entropy between the first feature map and the second feature map; taking the product of the relative entropy and the model transfer weight as the first loss function; constructing the total loss function based on the first loss function.

[0007] Optionally, before the step of inputting the training data set into the target model and the source model respectively to obtain the first feature data output by the source model and the second feature data output by the target model, it includes: obtaining the training rounds of the target model; when the training rounds are less than or equal to the first preset threshold, performing the step of inputting the training data set into the target model and the source model respectively to obtain the first feature data output by the source model and the second feature data output by the target model; when the training rounds are greater than the first preset threshold, obtaining the task loss function of the target model to construct the total loss function of the target model based on the task loss function.

[0008] Optionally, the training data set includes training image data and label data, and the target model includes a feature processing network and a task detection head; constructing the total loss function of the target model based on the difference data between the first feature data and the second feature data to train the target model based on the total loss function includes: constructing the first loss function of the feature processing network based on the difference data between the first feature data and the second feature data; obtaining the prediction data output by the target model; constructing the task loss function of the task detection head based on the difference data between the prediction data and the label data; constructing the total loss function based on the sum of the first loss function and the task loss function to train the target model based on the total loss function.

[0009] Optionally, the prediction data includes a parking space detection frame, and the task loss function includes a second loss function; constructing the task loss function of the task detection head based on the difference data between the prediction data and the label data includes: calculating the first offset between the first reference point of the parking space detection frame and the four parking space corner points of the parking space detection frame; obtaining the second offset between the corresponding first reference point and the parking space corner points of the parking space true value from the label data; constructing the second loss function based on the difference between the first offset and the second offset.

[0010] Optionally, the above prediction data includes a first prediction probability value of a parking space, and the above task loss function includes a third loss function; constructing the task loss function of the above task detection head based on the difference data between the above prediction data and the above label data includes: obtaining a first classification label value corresponding to the above parking space from the above label data; constructing the above third loss function based on the degree of difference between the above first prediction probability value and the above first classification label value.

[0011] Optionally, the above prediction data includes an external detection frame of an obstacle and a second prediction probability value; constructing the task loss function of the above task detection head based on the difference data between the above prediction data and the above label data includes: calculating a third offset between a second reference point of the above obstacle and four sides of the above external detection frame, and obtaining a fourth offset corresponding to the above second reference point of the above obstacle from the above label data, so as to construct a fourth loss function based on the difference between the above third offset and the above fourth offset; calculating a fifth offset of a grounding point of the above obstacle in the horizontal direction of the above external detection frame, and obtaining a sixth offset corresponding to the above grounding point of the above obstacle from the above label data, so as to construct a fifth loss function based on the difference between the above fifth offset and the above sixth offset; obtaining a second classification label value corresponding to the above obstacle from the above label data, and constructing the above sixth loss function based on the degree of difference between the above second prediction probability value and the above second classification label value; constructing the above task loss function based on the sum value of the above fourth loss function, the above fifth loss function, and the above sixth loss function.

[0012] To solve the above problems, the present application provides a second technical solution: providing an automatic parking method applied to an automobile, the above automatic parking method includes: acquiring a fisheye image of the above automobile; inputting the above fisheye image into a target model trained by the above model training method; acquiring the parkable space information of the above automobile output by the above target model; and controlling the above automobile to park in a parking space based on the parkable space information of the above target model.

[0013] To solve the above problems, the present application provides a third technical solution: providing an automatic parking system, including a processor and a memory, the above processor is connected to the above memory, wherein the above memory stores program instructions; the above processor is configured to execute the program instructions stored in the above memory to implement the above method.

[0014] To solve the above problems, the present application provides a fourth technical solution: providing a computer-readable storage medium, the above computer-readable storage medium stores program instructions, and the above program instructions can be executed by a processor to implement the above method.

[0015] The present application provides a model training and automatic parking method, an automatic parking system, and a computer medium. The model training method obtains a source model trained on a public data set and the first model parameters of the source model, and adjusts the model structure and the first model parameters of the source model to construct a target model to be trained; obtains a training data set, and inputs the training data set into the target model and the source model respectively to obtain first feature data output by the source model and second feature data output by the target model; constructs a total loss function of the target model based on the difference data between the first feature data and the second feature data, so as to train the target model based on the total loss function. Therefore, the model training method can construct a total loss function through the difference data between the first feature data and the second feature data, so that the target model can learn part of the feature processing ability of the source model, transfer a part of the ability of the source model to the target model, the target model can quickly adapt to the target task, reduce the data collection and annotation of the target model in the parking scenario, and improve the robustness of the target model at the same time. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following will briefly introduce the drawings required for the description of the embodiments. Obviously, the following drawings are only some embodiments of the present application. For those of ordinary skill in the art, without creative efforts, other drawings can be obtained according to these drawings. Among them: Figure 1 is a flowchart of the first embodiment of the model training method provided by the present application; Figure 2 is a flowchart of the second embodiment of the model training method provided by the present application; Figure 3 is a flowchart of the third embodiment of the model training method provided by the present application; Figure 4 is a flowchart of the fourth embodiment of the model training method provided by the present application; Figure 5 is a schematic diagram of the first reference point in the parking space detection frame; Figure 6 is a flowchart of the fifth embodiment of the model training method provided by the present application; Figure 7 is a flowchart of the sixth embodiment of the model training method provided by the present application; Figure 8 is a schematic diagram of the second reference point in the external detection frame; Figure 9 is a flowchart of the first embodiment of the automatic parking method provided by the present application; Figure 10It is a schematic flowchart of the second embodiment of the automatic parking method provided by this application; Figure 11 It is a schematic structural diagram of an embodiment of the automatic parking system provided by this application; Figure 12 It is a schematic structural diagram of an embodiment of the computer-readable storage medium provided by this application. Detailed implementation manners

[0017] Next, the technical solutions in the embodiments of this application will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of this application. Obviously, the described embodiments are only a part of the embodiments of this application, rather than all the embodiments. Based on the embodiments in this application, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of this application.

[0018] It should be noted that if there are directional indications (such as up, down, left, right, front, back,...) involved in the embodiments of this application, then the directional indications are only used to explain the relative positional relationship and movement conditions between components in a specific posture (as shown in the accompanying drawings). If this specific posture changes, the directional indications will also change accordingly.

[0019] In addition, if there are descriptions such as "first", "second", etc. involved in the embodiments of this application, then the descriptions of "first", "second", etc. are only for descriptive purposes, and cannot be understood as indicating or implying their relative importance or implicitly indicating the quantity of the indicated technical features. Thus, the features defined with "first" and "second" may explicitly or implicitly include at least one such feature. In addition, the technical solutions between various embodiments can be combined with each other, but it must be based on the fact that those of ordinary skill in the art can implement them. When the combination of technical solutions appears to be contradictory or unable to be implemented, it should be considered that such a combination of technical solutions does not exist and is not within the protection scope required by this application.

[0020] Since the publicly available datasets in the existing field of autonomous driving already have a very large amount of data volume and label information, and if we want to collect and label in the parking scenario, it will require a large amount of manpower and financial resources. Considering that the publicly available datasets in the field of autonomous driving and the images in the parking scenario have certain similarities. In view of this, the embodiments of this application first provide a model training method. The target model trained by this model training method is applied to the field of automatic parking and is specifically used to assist the vehicle in finding nearby parking spaces. The model training method of the embodiments of this application performs knowledge transfer on the public dataset, so that the target model can make full use of this knowledge, reduce the waste of data resources, reduce the data annotation cost, and improve the robustness of the model.

[0021] Please refer to Figure 1 ,Figure 1 It is a schematic flowchart of the first embodiment of the model training method provided by this application. As Figure 1 shown, in this embodiment, the model training method includes: Step S11: Obtain a source model trained on a public dataset and the first model parameters of the source model, and adjust the model structure and the first model parameters of the source model to construct a target model to be trained.

[0022] Specifically, the source model is a model pre-trained on a large-scale public dataset through a source task. The public dataset includes, but is not limited to, large-scale datasets such as ImageNet, COCO, Conceptual Captions, KITTI, nuScenes, Waymo OpenDataset, etc. By adjusting the model structure and the first model parameters of the source model, a target model to be trained can be constructed. It can be understood that the target task of the target model is similar to the source task of the source model, and the task data used by the source model and the task data used by the target model are related, so that the target model can use the general feature representation ability of the source model to perform feature processing on the images of the target task.

[0023] Exemplarily, the source model can be, but is not limited to, applied to source tasks such as 3D detection, target tracking, and behavior prediction in the field of autonomous driving. The source model needs to perform near-field perception on the surrounding environment images of the vehicle to execute the autonomous driving task. At this time, the surrounding environment images include, but are not limited to, at least one of fisheye images, ultra-wide-angle images, image acquisition data with a field of view angle greater than 170°, and image data conforming to a non-linear distortion model. The target model can be, but is not limited to, applied to parking space recognition and detection in an automatic parking scenario, and can specifically be used to find a parking space through the surrounding environment images. In this embodiment, the target model can be, but is not limited to, a multi-task joint training model, and the target tasks of the target model include, but are not limited to, at least one of a parking space detection task, a parking space classification task, and an obstacle detection task, so as to output a parking space through at least one task detection result.

[0024] Among them, after obtaining the source model and its first model parameters in this embodiment, the low-level parameters for extracting general features in the source model can be fixed to retain the ability of the source model to extract basic features such as edges and textures under the source task. And, adjust the output layer structure of the source model so that the target model obtained after adjustment can be used to output the number of categories matching the target task in terms of dimension; or, according to the complexity of the target task, add a task head after the fixed low-level layer so that the added task head can be applicable to the detection output of the target task. Through the above methods, a target model to be trained can be constructed based on the source model.

[0025] Step S12: Obtain a training data set, and input the training data set into the target model and the source model respectively to obtain first feature data output by the source model and second feature data output by the target model.

[0026] Further, obtain a training data set under the target task. The training data set includes surrounding environment images of a vehicle and label data. The surrounding environment images are near-field area images of the vehicle collected by the vehicle's perception system (such as a fish-eye camera). Exemplarily, the training data set includes multiple surrounding environment images, and each surrounding environment image has corresponding label data. The label data identifies at least one relevant parameter such as the position of a parking space detection frame, the size of the parking space, the category of the parking space, the category of obstacles, the position of the obstacle detection frame, and the position of the obstacle in the surrounding environment image.

[0027] After obtaining the training data set, input the training data set into the target model and the source model respectively to obtain the first feature data output by the source model and the second feature data output by the target model. Among them, after inputting the training data set into the source model, the feature processing network of the source model performs feature extraction and multi-scale fusion on the surrounding environment image to output the first feature data; after inputting the training data set into the target model, the feature processing network of the target model performs feature extraction and multi-scale fusion on the surrounding environment image to output the second feature data. Among them, the feature processing networks of the source model and the target model have a certain similarity, and the first feature data and the second feature data overlap at least partially. The first feature data and the second feature data include but are not limited to at least one of multi-scale feature maps, fusion feature maps, pyramid feature maps, etc.

[0028] Step S13: Construct a total loss function of the target model based on the difference data between the first feature data and the second feature data, and train the target model based on the total loss function.

[0029] After obtaining the first feature data and the second feature data, calculate the degree of difference between the first feature data and the second feature data to obtain difference data. Among them, the difference data can be calculated by at least one method such as calculating the relative entropy (KL divergence loss), mean square error, mean absolute error, structural similarity index, mutual information, regional difference value grid method, Euclidean distance, VGG / ResNet features, etc. of the first feature data and the second feature data.

[0030] After obtaining the difference data between the first feature data and the second feature data, the total loss function of the target model can be constructed based on the difference data between the first feature data and the second feature data, so as to train the target model based on the total loss function. The specific training method can be but is not limited to: after the target model completes the image processing of the surrounding environment in one batch, that is, after one iteration, the loss is calculated based on the total loss function, and the gradient is backpropagated to update the model parameters of the target model. At this time, during the iterative update process, the first model parameters of the source model remain fixed.

[0031] In the embodiment of the present application, the model training method adjusts the model structure and the first model parameters of the source model by obtaining the source model trained on the public dataset and the first model parameters of the source model, so as to construct the target model to be trained; obtains the training dataset, and inputs the training dataset into the target model and the source model respectively to obtain the first feature data output by the source model and the second feature data output by the target model; constructs the total loss function of the target model based on the difference data between the first feature data and the second feature data, so as to train the target model based on the total loss function. Therefore, the model training method of this embodiment can construct the total loss function through the difference data between the first feature data and the second feature data, so that the target model can learn part of the feature processing ability of the source model, transfer a part of the ability of the source model to the target model, so that the target model can quickly adapt to the target task, reduce the data collection and annotation of the target model in the parking scenario, and improve the robustness of the target model at the same time.

[0032] In one embodiment, the first feature data is the first feature map output by the feature processing network of the source model, and the second feature data is the second feature map output by the feature processing network of the target model. Please refer to Figure 2 , Figure 2 is the schematic flowchart of the second embodiment of the model training method provided by the present application. As Figure 2 shown, in the step S13 of constructing the total loss function of the target model based on the difference data between the first feature data and the second feature data, the model training method of this embodiment further includes: Step S21: Calculate the relative entropy between the first feature map and the second feature map.

[0033] Specifically, the feature processing network of the source model is used to perform basic feature extraction and multi-scale feature fusion on the image data of the training dataset to obtain a multi-scale first feature map. Similarly, the feature processing network of the target model is used to perform basic feature extraction and multi-scale feature fusion on the image data of the training dataset to obtain a multi-scale second feature map, so as to represent the activation response of the input training dataset in a specific feature dimension through the first feature map and the second feature map. After obtaining the first feature map and the second feature map, the relative entropy between the first feature map and the second feature map is calculated.

[0034] Among them, relative entropy is a metric for measuring the difference between two probability distributions, also known as Kullback-Leibler Divergence. When calculating the relative entropy, the following steps can be included: normalizing all pixel values or channel values of the first feature map and the second feature map to convert the first feature map and the second feature map into probability distributions; calculating the relative entropy between the normalized first feature map and the second feature map. Specifically, the formula for relative entropy is as follows: ; Among them, is the relative entropy between the first feature map and the second feature map; is the first feature map output by the source model; is the second feature map of the target model trained based on the parking scenario; N is the corresponding channel of the first feature map and the second feature map.

[0035] Step S22: Take the product of the relative entropy and the model transfer weight as the first loss function.

[0036] Specifically, set a model transfer weight, which is used to represent the weight when migrating the source model to the target model. The larger the model transfer weight, the more migration, and the greater the similarity between the source model and the target model; otherwise, it is smaller. Among them, the formula for the first loss function is as follows: ; Among them, is the first loss function; is the relative entropy between the first feature map and the second feature map; is the model transfer weight.

[0037] Step S23: Construct the total loss function based on the first loss function.

[0038] After calculating the first loss function, construct the total loss function based on the first loss function. Specifically, it can be to add the first loss function and other loss functions to obtain the total loss function.

[0039] Therefore, the model training method of this embodiment can calculate the relative entropy of the first feature map and the second feature map, take the product of the relative entropy and the model migration weight as the first loss function, and construct a total loss function based on the first loss function. Therefore, the relative entropy can constrain the probability distribution of the feature maps of the source model and the target model, so that the first loss function can be used to close the similarity of the feature processing networks of the target model and the source model, ensure that the trained target model can learn part of the feature processing capabilities of the source model, improve the ability of the target model to adapt to the target task, reduce the data collection and labeling of the target model in the parking scenario, and improve the robustness of the target model.

[0040] Optionally, before the step of inputting the training data set into the target model and the source model respectively to obtain the first feature data output by the source model and the second feature data output by the target model in step S20, the model training method of this embodiment also includes: obtaining the training round of the target model; when the training round is less than or equal to a first preset threshold, executing the step of inputting the training data set into the target model and the source model respectively to obtain the first feature data output by the source model and the second feature data output by the target model; when the training round is greater than the first preset threshold, obtaining the task loss function of the target model to construct the total loss function of the target model based on the task loss function.

[0041] Specifically, the model training method of this embodiment includes multiple training rounds when training the target model. A training round is a period in which the target model completely traverses the training data set once. In each training round, the target model updates the weights through multiple iterations to gradually reduce the total loss function. Before the training data set is input into the target model and the source model respectively, the current training round of the target model is first obtained.

[0042] When the current training round is less than or equal to a first preset threshold, the step of inputting the training data set into the target model and the source model respectively to obtain the first feature data output by the source model and the second feature data output by the target model is executed, so that the total loss function can be constructed based on the first loss function determined based on the first feature data and the second feature data.

[0043] When the number of training rounds is greater than the first preset threshold, the task loss function of the target model is obtained. The task loss function is used to make the target model regress and learn the processing ability of the specific target task, and the total loss function of the target model is constructed based on the task loss function. Among them, the target model also includes a task detection head, which is used to receive the feature data output by the feature processing network and predict the feature data under the target task. The task loss function is used to measure the deviation between the prediction result of the task detection head and the label data of the training data set, so as to provide a clear optimization direction for the target model.

[0044] That is, before the specified number of training rounds, the model training method of this embodiment pulls in the similarity between the target model and the source model through the first loss function and completes the model transfer learning. After the specified number of training rounds, the task loss function is used to prompt the target model to learn independently, so as to improve the task prediction accuracy of the target model in the parking scenario and enhance the robustness of the target model.

[0045] In one embodiment, please refer to Figure 3 , Figure 3 which is a schematic flowchart of the third embodiment of the model training method provided by this application. As Figure 3 shown, the training data set includes training image data and label data, and the target model includes a feature processing network and a task detection head. In step S13, the model training method of this embodiment further includes: Step S41: Construct a first loss function for the feature processing network based on the difference data between the first feature data and the second feature data.

[0046] Specifically, the difference data between the first feature data and the second feature data can be represented by relative entropy or other parameters. Based on the difference data between the first feature data and the second feature data, a first loss function for the feature processing network is constructed.

[0047] Step S42: Obtain the prediction data output by the target model.

[0048] Specifically, the prediction data is the prediction-related data for the target task output by the task prediction head. When the target task is a parking space detection task, the prediction data can be, but is not limited to, prediction data such as the position of the parking space detection frame and the size of the parking space; when the target task is a parking space classification task, the prediction data can be, but is not limited to, prediction data of the parking space category. When the target task is an obstacle detection task, the prediction data includes, but is not limited to, data such as the obstacle category, the position of the obstacle detection frame, and the position of the obstacle.

[0049] Step S43: Construct a task loss function for the task detection head based on the difference data between the prediction data and the label data.

[0050] Specifically, after obtaining the prediction data of the target model, a task loss function for the task detection head can be constructed based on the difference data between the prediction data and the label data.

[0051] Step S44: Construct a total loss function based on the sum of the first loss function and the task loss function, and train the target model based on the total loss function.

[0052] A total loss function is constructed based on the sum of the first loss function and the task loss function, so as to train the target model based on the total loss function. In a possible implementation, when the current training round of the target model is less than or equal to the first preset threshold, the above steps S41-S44 are executed; when the current training round of the target model is greater than the first preset threshold, the prediction data output by the target model is directly obtained, and the task loss function of the task detection head is constructed based on the difference data between the prediction data and the label data, so as to train the target model using the task loss function as the total loss function.

[0053] In an embodiment of the present application, the model training method obtains the prediction data output by the target model, constructs the task loss function of the task detection head based on the difference data between the prediction data and the label data, and constructs the total loss function based on the sum of the first loss function and the task loss function, so as to train the target model based on the total loss function. Therefore, the model training method of this embodiment can pull in the similarity between the feature processing network of the target model and the feature processing network of the source model through the first loss function, and optimize the prediction / classification ability of the task detection head through the task loss function, thereby guiding the direction and amplitude of the parameter adjustment of the target model, and further improving the robustness of the target model.

[0054] Optionally, the prediction data includes a parking space detection box, and the task loss function includes a second loss function. Figure 4 and Figure 5 , Figure 4 is a flowchart of the fourth embodiment of the model training method provided by the present application, Figure 5 is a schematic diagram of the first reference point in the parking space detection frame. Figure 4 and Figure 5 As shown, in this embodiment, step S43 includes the following steps: S431: Calculate a first offset between a first reference point of the parking space detection frame and four parking space corner points of the parking space detection frame.

[0055] Specifically, in this embodiment, the target model includes a first detection head, which is used to implement the parking space detection task in the parking scenario. After the image of the training data set is input into the target model, the feature processing network of the target model extracts and fuses features from the image to obtain a second feature map. The first detection head receives the second feature map input by the feature processing network and performs object detection on all pixel points in the second feature map to predict and output the parking space detection frame in the second feature map. Among them, the parking space detection frame is usually related to the parking range outlined by the parking space lines. In the actual parking scenario, the shapes of parking spaces are diverse. For example, the shapes of parking spaces can include horizontal parking spaces, vertical parking spaces, or inclined parking spaces, etc. Since the fisheye image will cause distortion of the parking space shape when the vehicle captures the surrounding environment image through the fisheye lens of the perception system, the forms of the parking space detection frame in this embodiment are diverse, including but not limited to rectangular detection frames, long trapezoidal detection frames, or polygons, etc.

[0056] The first reference point of the parking space detection frame is a pixel point within the parking space detection frame. The first reference point can be a specified position pixel point set in advance. For example, the first reference point is the midpoint of the parking space detection frame, etc.; the first reference point can also be a random pixel point within the parking space detection frame. Among them, the parking space detection frame includes at least one first reference point.

[0057] After obtaining the first reference point of the parking space detection frame, calculate the first offset between the first reference point of the parking space detection frame and the four corner points of the parking space of the parking space detection frame. Exemplarily, a two-dimensional coordinate system can be defined with the first reference point as the origin. Under this two-dimensional coordinate system, the position coordinates of the four corner points of the parking space detection frame are respectively represented as (x1, y1), (x2, y2), (x3, y3), and (x4, y4). Among them, the first offset can include the offset of the four corner points in the x direction of this two-dimensional coordinate system from the first reference point and the offset of the four corner points in the y direction of this two-dimensional coordinate system from the first reference point. At this time, the first offset can include , , , , , , , .

[0058] S432: Obtain the second offset between the corresponding first reference point and the corner points of the ground truth of the parking space from the label data.

[0059] Specifically, the label data of the training image set includes the ground truth of the parking space, and the ground truth of the parking space indicates the position of the real parking space in the image. Therefore, the second offset between the corresponding first reference point and the corner points of the ground truth of the parking space can be obtained from the label data.

[0060] S433: Construct a second loss function based on the difference between the first offset and the second offset.

[0061] Specifically, after obtaining the first offset and the second offset, calculate the difference between the first offset and the second offset to obtain the gap between the parking space detection box predicted by the target model and the actual parking space. Specifically, the difference between the first offset and the second offset can be the difference in the offset in the x - direction or y - direction corresponding to the first reference point and the parking space detection box and the actual value of the parking space. The formula for the second loss function is as follows: ; where N represents the number of first reference points; represents the predicted offset in the x - direction between the i - th first reference point and the j - th corner point of the parking space detection box, represents the predicted offset in the y - direction between the i - th first reference point and the j - th corner point of the parking space detection box; represents the actual offset in the x - direction between the i - th first reference point and the j - th corner point of the actual parking space, represents the actual offset in the y - direction between the i - th first reference point and the j - th corner point of the actual parking space.

[0062] The model method of this embodiment calculates the first offset between the first reference point of the parking space detection box and the four corner points of the parking space detection box, obtains the second offset between the corresponding first reference point and the corner points of the actual parking space from the label data, and constructs a second loss function based on the difference between the first offset and the second offset. Different from the prior art which directly uses the method of circumscribed rectangle detection or only detects two corner points on the parking space entry line, this embodiment can describe the shape and size of the parking space by introducing the offset between the first reference point and the corner points of the parking space, thereby reducing the limitations on the shape of the parking space and the detection box, which is beneficial to improving the prediction accuracy of the parking space detection task and further enhancing the robustness of the target model.

[0063] Optionally, the prediction data includes the first prediction probability value of the parking space, and the task loss function includes a third loss function. Please refer to Figure 6 Figure 6 which is the flowchart of the fifth embodiment of the model training method provided by this application. As Figure 6 shown, in this embodiment, step S43 includes the following steps: S434: Obtain the first classification label value corresponding to the parking space from the label data.

[0064] ​Specifically, in this embodiment, the target model includes a second detection head, which is used to implement the parking space classification task in the parking scenario. After the image of the training dataset is input into the target model, the feature processing network of the target model extracts and fuses the features of the image to obtain a second feature map. The second detection head receives the second feature map input by the feature processing network and performs object detection on all pixel points in the second feature map to predict the first prediction probability value of at least one parking space in the second feature map.

[0065] Among them, the first prediction probability value is used to describe the probability that the parking space can be parked. The first prediction probability value can be confidence data from 0 to 1; when the first prediction probability value is 1, that is, the target model predicts that the category of this parking space is a parking space that can be parked; when the first prediction probability value is 0, that is, the target model predicts that the category of this parking space is a parking space that cannot be parked; the larger the first prediction probability value, the greater the probability that the target model predicts that this parking space can be parked.

[0066] After obtaining the first prediction probability value, further obtain the first classification label value of the corresponding parking space from the label data. The first classification label value is used to indicate whether this parking space is truly parkable. The first classification label value is 0 or 1. When the first classification label value is 0, it means that this parking space cannot be parked. When the first classification label value is 1, it means that this parking space can be parked.

[0067] S435: Construct a third loss function based on the difference degree between the first prediction probability value and the first classification label value.

[0068] Specifically, after obtaining the first prediction probability value and the first classification label value, calculate the difference degree between the first prediction probability value and the first classification label value, and construct a third loss function based on the difference degree between the first prediction probability value and the first classification label value. The formula of the third loss function is as follows: ; Among them, is the third loss function, M represents the number of parking spaces on the second feature map input to the task detection head, , represent the gradient backpropagation weights, which are used to measure the training effect, represents the first classification label value, represents the first prediction probability value.

[0069] Therefore, the model training method of this embodiment obtains the first classification label value of the corresponding parking space from the label data, constructs a third loss function based on the difference degree between the first prediction probability value and the first classification label value, and measures the difference degree between the prediction probability of the second detection head in the parking space classification task and the true label through the third loss function, which is convenient for subsequently minimizing this loss function, improving the classification accuracy of the model for whether the parking space can be parked, and further improving the robustness of the target model.

[0070] Optionally, the prediction data includes an external detection box of the obstacle and a second prediction probability value. Please refer to Figure 7 and Figure 8 , Figure 7 FIG. is a schematic flowchart of the sixth embodiment of the model training method provided by the present application, Figure 8 FIG. is a schematic diagram of the second reference point in the external detection box. As Figure 7 shown, in this embodiment, step S43 includes the following steps: S436: Calculate a third offset between the second reference point of the obstacle and the four sides of the external detection box, and obtain a fourth offset of the second reference point of the corresponding obstacle from the label data, so as to construct a fourth loss function based on the difference between the third offset and the fourth offset.

[0071] Specifically, in this embodiment, the target model includes a third detection head, and the third detection head is used to implement the obstacle detection task in the parking scenario. After the image of the training data set is input into the target model, the feature processing network of the target model extracts and fuses the features of the image to obtain a second feature map. The third detection head receives the second feature map input by the feature processing network and performs target detection on all pixel points in the second feature map to predict at least one obstacle in the second feature map, and outputs an external detection box of the obstacle and a second prediction probability value.

[0072] Among them, as Figure 8 shown, the second reference point of the obstacle can be defined. The selection method of the second reference point is similar to that of the above first reference point, and will not be elaborated here. Since the target model of this embodiment needs to obtain the fisheye image of the surrounding environment through the fisheye camera of the vehicle when applied to parking prediction and planning, and the shape of the obstacle in the fisheye image will be distorted, therefore, in this embodiment, a third offset between the second reference point of the obstacle and the four sides of the external detection box is calculated, and a fourth offset of the second reference point of the corresponding obstacle is obtained from the label data, so as to construct a fourth loss function based on the difference between the third offset and the fourth offset.

[0073] Specifically, taking the second reference point of the obstacle as the origin, a two-dimensional coordinate system is established within the external detection box, and then the third offset between the second reference point of the obstacle and the four sides can be represented by the distances on the coordinate axes between the second reference point and the four sides. Similarly, the label data includes the true external bounding box of the obstacle. Therefore, the fourth offset between the corresponding second reference point and the four sides of the true external bounding box can be obtained from the label data.

[0074] Construct a fourth loss function based on the difference between the third offset and the fourth offset. Specifically, the difference between the third offset and the fourth offset can be expressed as the difference between the distance from the second reference point to one side of the external detection box and the distance from the second reference point to the same side of the true external bounding box. The formula for the fourth loss function is as follows: ; Wherein, is the fourth loss function; U is the number of second reference points; , , , respectively represent the predicted offsets between one of the second reference points and the four sides of the external detection box, , , , respectively represent the predicted offsets between one of the second reference points and the four sides of the true external bounding box.

[0075] Therefore, in this embodiment, the shape and size of the obstacle can be described by introducing the offset between the second reference point and the external detection box of the obstacle, thereby reducing the restrictions on the shape of the obstacle and the detection box, reducing the influence of the distortion of the fisheye image on the detection, being beneficial to improving the prediction accuracy of the parking space detection task, and further enhancing the robustness of the target model.

[0076] S437: Calculate the fifth offset of the grounding point of the obstacle in the horizontal direction of the external detection box, and obtain the sixth offset of the grounding point of the corresponding obstacle from the label data, so as to construct a fifth loss function based on the difference between the fifth offset and the sixth offset.

[0077] Wherein, in this embodiment, the center point on the bottom edge of the external detection box is defined as the grounding point of the obstacle, and the fifth offset of the grounding point in the horizontal direction of the external detection box is calculated. Wherein, with the grounding point as the center, the direction of extending the grounding point along the bottom edge to one side is defined as the x direction, and the direction of extending the bottom edge of the grounding point to the other side is defined as the y direction. Then, the fifth offset of the grounding point of the obstacle in the horizontal direction of the external detection box can be expressed by the position of the side of the external detection box in the x direction and the position of the side in the y direction, that is, the fifth offset includes dx and dy. Alternatively, the fifth offset can also be expressed by the distances between the grounding point and the two sides of the external detection box.

[0078] Similarly, the label data includes the true external bounding box of the obstacle. Therefore, the offsets of the grounding point on the bottom edge of the true external bounding box relative to the sides in the x direction and the y direction can be obtained from the label data, that is, used as the sixth offset of the grounding point of the true external bounding box.

[0079] Construct a fourth loss function based on the difference between the fifth offset and the sixth offset. Specifically, the difference between the fifth offset and the sixth offset may include: the offset in the x direction between the grounding point and the external detection frame and the offset in the x direction between the grounding point and the true external bounding box; the offset in the y direction between the grounding point and the external detection frame and the offset in the y direction between the grounding point and the true external bounding box. The formula of the fifth loss function is as follows: ; where, is the fifth loss function; H is the number of external detection frames of the obstacle; is the predicted offset in the x direction between the grounding point and the external detection frame, represents the predicted offset in the y direction between the grounding point and the external detection frame; represents the true offset in the x direction between the grounding point and the true external bounding box, represents the true offset in the y direction between the grounding point and the true external bounding box.

[0080] Therefore, in this embodiment, the distance between the obstacle and the vehicle can be described by introducing the grounding point, so that the target model can evaluate an effective parking space based on the distance between the obstacle and the vehicle in the parking path; by introducing the fifth loss function, the influence of the distortion of the fisheye image on the judgment of the grounding point can be effectively reduced, which is beneficial to improving the prediction accuracy of the parking space detection task and further enhancing the robustness of the target model.

[0081] S438: Obtain the second classification label value corresponding to the obstacle from the label data, and construct a sixth loss function based on the degree of difference between the second prediction probability value and the second classification label value.

[0082] Among them, the second prediction probability value is used to describe the credibility of the classification prediction of the obstacle. The second prediction probability value can be confidence data from 0 to 1; when the second prediction probability value is 1, that is, the target model's prediction of the category of this obstacle is credible; when the second prediction probability value is 0, that is, the target model's prediction of the category of this obstacle is not credible. After obtaining the second prediction probability value, further obtain the second classification label value corresponding to the parking space from the label data, and the second classification label value is the true category label of the obstacle. Construct a sixth loss function based on the degree of difference between the second prediction probability value and the second classification label value. The formula of the sixth loss function is as follows: ; where, F represents the number of samples of the obstacle; , represent the gradient backpropagation weights, which are used to measure the training effect; represents the second classification label value, represents the second prediction probability value.

[0083] Therefore, in this embodiment, by introducing the sixth loss function, it is convenient to subsequently minimize the loss function to improve the accuracy of the target model in classifying obstacles and further enhance the robustness of the target model.

[0084] S439: Construct a task loss function based on the sum of the fourth loss function, the fifth loss function, and the sixth loss function.

[0085] Construct a task loss function based on the sum of the fourth loss function, the fifth loss function, and the sixth loss function, so that the task loss function can be used to balance the multitasking effect of the target model and further enhance the robustness of the target model.

[0086] Optionally, in one implementation, the above steps S431 - S433 can be executed after completing steps S434 - S435. Exemplarily, after executing steps S434 - S435, the model method further includes: obtaining parking spaces in a certain surrounding environment image where the first prediction probability value is greater than the second preset threshold, and obtaining the position parameters of the corresponding parking spaces in the feature map of the surrounding environment image; based on the position parameters, execute steps S431 - S433 to perform parking space detection on the corresponding parking spaces in the feature map of the surrounding environment image, reducing the amount of computation.

[0087] In another implementation, the above steps S431 - S433, steps S434 - S435, and steps S436 - S439 can be executed simultaneously. After completing steps S431 - S433, steps S434 - S435, and steps S436 - S439, the model method further includes: using the sum of the first loss function, the second loss function, the third loss function, the fourth loss function, the fifth loss function, and the sixth loss function as the total loss function; or, using the sum of the second loss function, the third loss function, the fourth loss function, the fifth loss function, and the sixth loss function as the total loss function.

[0088] The embodiment of the present application also proposes an automatic parking method, which is applied to an automobile. Please refer to Figure 9 , Figure 9 which is the flowchart of the first embodiment of the automatic parking method provided by the present application. As Figure 9 shown, in this embodiment, the automatic parking method includes: Step S51: Obtain the fisheye image of the vehicle.

[0089] Specifically, the vehicle may include a perception system, which includes a plurality of fisheye cameras disposed on the vehicle body or at different mirror positions. The vehicle is used to capture the surrounding environment through the fisheye cameras to obtain fisheye images. The automatic parking method of this embodiment obtains the fisheye images of the vehicle, so that the target model performs parking space detection, parking space classification, and obstacle detection on the fisheye images.

[0090] Step S52: Input the fisheye image into the target model trained by the model training method of any of the above embodiments.

[0091] Specifically, the target model may include a first detection head, a second detection head, and a third detection head. The first detection head is used to detect the parking spaces in the fisheye image to output parking space detection data; the second detection head is used to classify the parking spaces in the fisheye image to output parking space classification data; the third detection head is used to detect the obstacles in the fisheye image to output obstacle detection data.

[0092] It can be understood that in the prior art, when performing automatic parking, the parking spaces are usually detected through the images collected by the fisheye cameras, and the obstacles are detected through the information collected by the ultrasonic radars. By combining the information of the two sensors, namely the fisheye camera and the ultrasonic radar, the final available parking space position information is output. However, using ultrasonic sensors for detection requires adding an ultrasonic information processing module and an ultrasonic information and parking space detection information fusion module in the automatic parking system, which will make the entire solution more complex. Different from the prior art, the automatic parking method of this embodiment adopts a multi-task joint learning structure. By using the first detection head, the second detection head, and the third detection head to process the perceived fisheye images for different tasks respectively, it can effectively improve the utilization of the fisheye images and output more accurate obstacle information, so that the solution of this embodiment does not need to add redundant ultrasonic processing and simplifies the system structure.

[0093] Step S53: Obtain the available parking space information of the vehicle output by the target model.

[0094] Specifically, the target model can integrate the parking space detection data, the parking space classification data, and the obstacle detection data to output the available parking space information of the vehicle. Among them, the available parking space information includes, but is not limited to, the coordinate information of the available parking space, the path information of the available parking space, the vehicle control information during the path process, etc.

[0095] Step S54: Control the vehicle to park into the parking space based on the available parking space information of the target model.

[0096] After obtaining the available parking space information of the target model, control the vehicle to park into the parking space based on the available parking space information, so as to achieve automatic parking.

[0097] Optionally, please refer to Figure 10 ,Figure 10 is a schematic flowchart of the second embodiment of the automatic parking method provided by this application. As Figure 10 shown, this automatic parking method uses a target model trained by multi-task joint training, and at the same time adopts a model migration strategy to transfer part of the learning ability of the source model of the existing public dataset to the target model. After the fisheye data is input into the target model, the target model can output detection data including a parking space detection task, a parking space classification task, and an obstacle detection task, so as to effectively utilize the data of the public dataset of the existing scenario, and realize the recognition of parkable spaces end-to-end, which can effectively improve the robustness of the model. Moreover, it can overcome the problem that the existing automatic parking method requires a post-processing fusion module to integrate information and cannot achieve a unified end-to-end model to output all target information, and overcome the disadvantage that the multi-stage method is not friendly to storage devices.

[0098] Please refer to Figure 11 , Figure 11 is a schematic structural diagram of an embodiment of the automatic parking system provided by this application. As Figure 6 shown, the automatic parking system 50 of this embodiment includes a memory 52 and a processor 51, and the processor 51 is connected to the memory 52. The memory 52 is used to store program instructions. The processor 51 is used to execute the program instructions stored in the memory 52 to implement the method described in any of the above embodiments.

[0099] Among them, the processor 51 can also be called a CPU (Central Processing Unit, central processing unit). The processor 51 may be an integrated circuit chip with signaling processing capabilities. The processor 51 can also be a general-purpose processor, a digital signaling processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc.

[0100] The memory 52 can be a memory stick, a TF card, etc., and can store all the information in the automatic parking system 50. All the input raw data, computer programs, intermediate operation results, and final operation results are stored in the memory. It stores and retrieves information according to the positions specified by the controller. With the memory, the string matching prediction device has a memory function and can ensure normal operation. According to the purpose, the memory of the string matching prediction device can be divided into a main memory (memory) and an auxiliary memory (external memory), and there is also a classification method of dividing it into an external memory and an internal memory. The external memory is usually a magnetic medium or an optical disc, etc., which can store information for a long time. The memory refers to the storage component on the motherboard, which is used to store the data and programs being executed currently, but is only used to temporarily store programs and data. When the power is turned off or cut off, the data will be lost.

[0101] In several embodiments provided in the present application, it should be understood that the disclosed methods and apparatuses can be implemented in other ways. For example, the methods described above are merely illustrative. For example, the division of modules or units is only a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed couplings or direct couplings or communication connections between each other can be through some interfaces. The indirect couplings or communication connections of devices or units can be in electrical, mechanical or other forms.

[0102] The units described as separate components may or may not be physically separated. The components displayed as units may or may not be physical units, that is, they can be located in one place or distributed to multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0103] In addition, in each embodiment of the present application, each functional unit can be integrated in a processing unit, or each unit can exist physically alone, or two or more units can be integrated in one unit. The above integrated units can be implemented in the form of hardware or in the form of software functional units.

[0104] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the present application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which can be a personal computer, a system server, or a network device, etc.) or a processor to execute all or part of the steps of the methods in each embodiment of the present application.

[0105] Please refer to Figure 12 , Figure 12 which is a schematic structural diagram of an embodiment of the computer-readable storage medium provided by the present application. As Figure 12As shown, the computer-readable storage medium of the present application stores program instructions 61 capable of implementing all the above methods. Among them, the program instructions 61 can be stored in the above storage medium in the form of a software product, including several instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) or a processor to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage device includes: various media such as USB flash drives, external hard drives, read-only memories (ROMs), random access memories (RAMs), magnetic disks, or optical discs that can store program codes, or electronic devices such as computers, servers, mobile phones, or tablets.

[0106] The above are only the embodiments of the present application, and do not limit the patent scope of the present application. Any equivalent structural or equivalent process transformation made by using the content of the specification and drawings of the present application, or directly or indirectly applied in other related technical fields, shall be equally included in the patent protection scope of the present application.

Claims

1. A model training method, characterized in that, The model training method includes: Obtain a source model trained on a public dataset and first model parameters of the source model, and adjust the model structure and the first model parameters of the source model to construct a target model to be trained; Obtain a training dataset, and input the training dataset into the target model and the source model respectively to obtain first feature data output by the source model and second feature data output by the target model; Based on the difference data between the first feature data and the second feature data, construct a total loss function of the target model to train the target model based on the total loss function.

2. The model training method according to claim 1, wherein The first feature data is a first feature map output by a feature processing network of the source model, and the second feature data is a second feature map output by a feature processing network of the target model; The constructing the total loss function of the target model based on the difference data between the first feature data and the second feature data includes: Calculate the relative entropy between the first feature map and the second feature map; Take the product of the relative entropy and the model transfer weight as a first loss function; Construct the total loss function based on the first loss function.

3. The model training method according to claim 2, wherein Before the step of inputting the training dataset into the target model and the source model respectively to obtain first feature data output by the source model and second feature data output by the target model, it includes: Obtain the training round of the target model; When the training round is less than or equal to a first preset threshold, execute the step of inputting the training dataset into the target model and the source model respectively to obtain first feature data output by the source model and second feature data output by the target model; When the training round is greater than the first preset threshold, obtain the task loss function of the target model to construct the total loss function of the target model based on the task loss function.

4. The model training method according to claim 1, wherein The training dataset includes training image data and label data, and the target model includes a feature processing network and a task detection head; the constructing the total loss function of the target model based on the difference data between the first feature data and the second feature data to train the target model based on the total loss function includes: Based on the difference data between the first feature data and the second feature data, construct a first loss function of the feature processing network; Obtain prediction data output by the target model; Based on the difference data between the prediction data and the label data, construct a task loss function of the task detection head; Construct the total loss function based on the sum of the first loss function and the task loss function to train the target model based on the total loss function.

5. The model training method according to claim 4, characterized in that The prediction data includes a parking space detection frame, and the task loss function includes a second loss function; The constructing the task loss function of the task detection head based on the difference data between the prediction data and the label data includes: Calculate a first offset between a first reference point of the parking space detection frame and four parking space corner points of the parking space detection frame; Obtain a second offset between the corresponding first reference point and the corner point of the true parking space value from the label data; Construct the second loss function based on the difference between the first offset and the second offset.

6. The model training method according to claim 4, characterized in that The prediction data includes a first prediction probability value of the parking space, and the task loss function includes a third loss function; Constructing the task loss function of the task detection head based on the difference data between the prediction data and the label data includes: Obtain a first classification label value corresponding to the parking space from the label data; Construct the third loss function based on the difference degree between the first prediction probability value and the first classification label value.

7. The model training method according to claim 4, characterized in that The prediction data includes an external detection frame of the obstacle and a second prediction probability value; Constructing the task loss function of the task detection head based on the difference data between the prediction data and the label data includes: Calculate a third offset between the second reference point of the obstacle and the four sides of the external detection frame, and obtain a fourth offset corresponding to the second reference point of the obstacle from the label data, so as to construct a fourth loss function based on the difference between the third offset and the fourth offset; Calculate a fifth offset of the grounding point of the obstacle in the horizontal direction of the external detection frame, and obtain a sixth offset corresponding to the grounding point of the obstacle from the label data, so as to construct a fifth loss function based on the difference between the fifth offset and the sixth offset; Obtain a second classification label value corresponding to the obstacle from the label data, and construct a sixth loss function based on the difference degree between the second prediction probability value and the second classification label value; Construct the task loss function based on the sum value of the fourth loss function, the fifth loss function, and the sixth loss function.

8. An automatic parking method, characterized in that, Applied to an automobile, the automatic parking method includes: Obtain the fisheye image of the automobile; Input the fisheye image into a target model trained by the model training method according to any one of claims 1-7; Obtain the available parking space information of the automobile output by the target model; Control the automobile to park into the parking space based on the available parking space information of the target model.

9. An automatic parking system, characterized in that, Including a processor and a memory, the processor is connected to the memory, wherein, The memory stores program instructions; The processor is configured to execute the program instructions stored in the memory to implement the method according to any one of claims 1-8.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores program instructions, and the program instructions can be executed by a processor to implement the method according to any one of claims 1-8.

Images