Model training and automatic parking method, automatic parking system and computer medium

Abstract

The application discloses a model training and automatic parking method, an automatic parking system and a computer medium. The training method adjusts a model structure and first model parameters of a source model by acquiring the source model and the first model parameters trained on a public data set, to construct a target model. Training data sets are input into the target model and the source model respectively to acquire first feature data output by the source model and second feature data output by the target model. A total loss function of the target model is constructed based on difference data of the first feature data and the second feature data, to train the target model based on the total loss function. Therefore, the total loss function is constructed based on the difference data of the first feature data and the second feature data, so that the target model can learn part of feature processing capability of the source model, the target model can quickly adapt to a target task, data collection and labeling of the target model in a parking scene are reduced, and robustness of the target model is improved.

Description

Technical Field

[0001] The present application relates to the field of automatic parking technology, and in particular to model training and automatic parking methods, automatic parking systems, and computer media. Background Art

[0002] Advanced Driver Assistance Systems (ADAS) are integrated control systems for active safety features that use onboard sensors to perceive and collect information about the vehicle's surroundings, systematically analyze and make decisions, proactively avoiding danger and improving driving safety and comfort. With the growing demand for cars and the continuous increase in car ownership, traffic congestion, urban parking shortages, and limited parking spaces are all contributing to these challenges. This poses a greater challenge to automated parking.

[0003] Existing automated parking methods typically use information collected from fisheye images captured by fisheye cameras for predictive analysis to identify available parking spaces. Due to the ultra-wide-angle nature of fisheye cameras, fisheye images exhibit significant distortion compared to conventional images. Therefore, training a model for automated parking scenarios requires the collection and annotation of a large number of fisheye images, resulting in significant labor and financial costs. Summary of the Invention

[0004] To solve the above technical problems, the present application provides a model training and automatic parking method, an automatic parking system and a computer medium.

[0005] To solve the above problems, the present application provides a first technical solution: a model training method is provided, comprising: obtaining a source model trained on a public data set and a first model parameter of the source model, adjusting the model structure of the source model and the first model parameter to construct a target model to be trained; obtaining a training data set, and inputting the training data set into the target model and the source model respectively to obtain the first feature data output by the source model and the second feature data output by the target model; based on the difference data of the first feature data and the second feature data, constructing a total loss function of the target model to train the target model based on the total loss function.

[0006] Optionally, the first feature data is the first feature map output by the feature processing network of the source model, and the second feature data is the second feature map output by the feature processing network of the target model; the total loss function of the target model is constructed based on the difference data between the first feature data and the second feature data, including: calculating the relative entropy of the first feature map and the second feature map; taking the product of the relative entropy and the model migration weight as the first loss function; and constructing the total loss function based on the first loss function.

[0007] Optionally, before the step of inputting the above-mentioned training data set into the above-mentioned target model and the above-mentioned source model respectively to obtain the first feature data output by the above-mentioned source model and the second feature data output by the above-mentioned target model, the step includes: obtaining the training round of the above-mentioned target model; when the above-mentioned training round is less than or equal to the first preset threshold, executing the step of inputting the above-mentioned training data set into the above-mentioned target model and the above-mentioned source model respectively to obtain the first feature data output by the above-mentioned source model and the second feature data output by the above-mentioned target model; when the above-mentioned training round is greater than the above-mentioned first preset threshold, obtaining the task loss function of the above-mentioned target model to construct the total loss function of the above-mentioned target model based on the above-mentioned task loss function.

[0008] Optionally, the above-mentioned training data set includes training image data and label data, and the above-mentioned target model includes a feature processing network and a task detection head; the above-mentioned total loss function of the target model is constructed based on the difference data between the above-mentioned first feature data and the above-mentioned second feature data, so as to train the above-mentioned target model based on the above-mentioned total loss function, including: constructing the first loss function of the above-mentioned feature processing network based on the difference data between the above-mentioned first feature data and the above-mentioned second feature data; obtaining the predicted data output by the above-mentioned target model; constructing the task loss function of the above-mentioned task detection head based on the difference data between the above-mentioned predicted data and the above-mentioned label data; constructing the above-mentioned total loss function based on the sum of the above-mentioned first loss function and the above-mentioned task loss function, so as to train the above-mentioned target model based on the above-mentioned total loss function.

[0009] Optionally, the above-mentioned prediction data includes a parking space detection frame, and the above-mentioned task loss function includes a second loss function; the above-mentioned task loss function of the task detection head is constructed based on the difference data between the above-mentioned prediction data and the above-mentioned label data, including: calculating the first offset between the first reference point of the above-mentioned parking space detection frame and the four parking space corner points of the above-mentioned parking space detection frame; obtaining the corresponding second offset between the above-mentioned first reference point and the parking space corner point of the actual value of the parking space from the above-mentioned label data; and constructing the above-mentioned second loss function based on the difference between the above-mentioned first offset and the above-mentioned second offset.

[0010] Optionally, the above-mentioned prediction data includes a first prediction probability value of the parking space, and the above-mentioned task loss function includes a third loss function; the above-mentioned task loss function of the task detection head is constructed based on the difference data between the above-mentioned prediction data and the above-mentioned label data, including: obtaining the first classification label value corresponding to the above-mentioned parking space from the above-mentioned label data; and constructing the above-mentioned third loss function based on the degree of difference between the above-mentioned first prediction probability value and the above-mentioned first classification label value.

[0011] Optionally, the prediction data includes an external detection frame and a second predicted probability value of the obstacle; the task loss function of the task detection head is constructed based on the difference data between the prediction data and the label data, including: calculating a third offset between the second reference point of the obstacle and the four sides of the external detection frame, and obtaining a fourth offset corresponding to the second reference point of the obstacle from the label data, so as to construct a fourth loss function based on the difference between the third offset and the fourth offset; calculating a fifth offset of the ground point of the obstacle in the horizontal direction of the external detection frame, and obtaining a sixth offset corresponding to the ground point of the obstacle from the label data, so as to construct a fifth loss function based on the difference between the fifth offset and the sixth offset; obtaining a second classification label value corresponding to the obstacle from the label data, and constructing the sixth loss function based on the degree of difference between the second prediction probability value and the second classification label value; and constructing the task loss function based on the sum of the fourth loss function, the fifth loss function and the sixth loss function.

[0012] To solve the above problems, the present application provides a second technical solution: providing an automatic parking method, applied to a car, the above automatic parking method comprising: obtaining a fisheye image of the above car; inputting the above fisheye image into a target model trained by the above model training method; obtaining the parking space information of the above car output by the above target model; and controlling the above car to park in a parking space based on the parking space information of the above target model.

[0013] To solve the above problem, the present application provides a third technical solution: an automatic parking system is provided, comprising a processor and a memory, wherein the processor is connected to the memory, wherein the memory stores program instructions; the processor is used to execute the program instructions stored in the memory to implement the above method.

[0014] To solve the above problem, the present application provides a fourth technical solution: providing a computer-readable storage medium, wherein the computer-readable storage medium stores program instructions, and the program instructions can be executed by a processor to implement the above method.

[0015] The present application provides a model training and automatic parking method, an automatic parking system, and a computer medium. The model training method obtains a source model trained on a public data set and a first model parameter of the source model, adjusts the model structure and the first model parameter of the source model to construct a target model to be trained; obtains a training data set, inputs the training data set into the target model and the source model respectively to obtain the first feature data output by the source model and the second feature data output by the target model; constructs a total loss function of the target model based on the difference data of the first feature data and the second feature data, and trains the target model based on the total loss function. Therefore, the model training method can construct a total loss function through the difference data of the first feature data and the second feature data, so that the target model can learn part of the feature processing capabilities of the source model, and transfer part of the capabilities of the source model to the target model. The target model can quickly adapt to the target task, reduce the data collection and labeling of the target model in the parking scenario, and improve the robustness of the target model. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for describing the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without inventive efforts. Among them:

[0017] Figure 1 This is a flowchart of the first embodiment of the model training method provided by this application;

[0018] Figure 2 This is a flow chart of the second embodiment of the model training method provided by this application;

[0019] Figure 3 This is a flowchart of the third embodiment of the model training method provided by this application;

[0020] Figure 4 4 is a flowchart of the fourth embodiment of the model training method provided by this application;

[0021] Figure 5 is a schematic diagram of the first reference point in the parking space detection frame;

[0022] Figure 6 1 is a flowchart of the fifth embodiment of the model training method provided by this application;

[0023] Figure 7 1 is a flow chart of the sixth embodiment of the model training method provided by the present application;

[0024] Figure 8is a schematic diagram of the second reference point in the circumscribed detection frame;

[0025] Figure 9 is a flowchart of a first embodiment of the automatic parking method provided by the present application;

[0026] Figure 10 is a flowchart of a second embodiment of the automatic parking method provided by this application;

[0027] Figure 11 is a structural diagram of an embodiment of the automatic parking system provided by the present application;

[0028] Figure 12 It is a structural diagram of an embodiment of the computer-readable storage medium provided by the present application. DETAILED DESCRIPTION

[0029] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments of this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0030] It should be noted that if the embodiments of the present application involve directional indications (such as up, down, left, right, front, back, etc.), such directional indications are only used to explain the relative position relationship, movement status, etc. between the various components under a certain specific posture (as shown in the accompanying drawings). If the specific posture changes, the directional indication will also change accordingly.

[0031] In addition, if there are descriptions involving "first", "second", etc. in the embodiments of the present application, the descriptions of "first", "second", etc. are only for descriptive purposes and cannot be understood as indicating or implying their relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of such features. In addition, the technical solutions between the various embodiments can be combined with each other, but they must be based on the fact that they can be implemented by ordinary technicians in this field. When the combination of technical solutions is contradictory or cannot be implemented, it should be deemed that such a combination of technical solutions does not exist and is not within the scope of protection required by this application.

[0032] Since the existing public datasets in the field of autonomous driving already have a very large data volume and label information, and we need a lot of manpower and financial resources to collect and label them in parking scenes, considering that the public datasets in the field of autonomous driving and the images in parking scenes have certain similarities. In view of this, the embodiment of the present application first provides a model training method, and the target model obtained by training the model training method is applied to the field of automatic parking, specifically to assist cars in finding nearby parking spaces. The model training method of the embodiment of the present application transfers knowledge from the public dataset so that the target model can make full use of this knowledge, reduce the waste of data resources, reduce the cost of data labeling, and improve the robustness of the model.

[0033] See Figure 1 , Figure 1 This is a flow chart of the first embodiment of the model training method provided by this application. Figure 1 As shown, in this embodiment, the model training method includes:

[0034] Step S11: Obtain a source model trained on a public dataset and first model parameters of the source model, and adjust the model structure and first model parameters of the source model to construct a target model to be trained.

[0035] Specifically, the source model is a model that has been pre-trained for the source task on a large-scale public dataset, including but not limited to ImageNet, COCO, Conceptual Captions, KITTI, nuScenes, Waymo OpenDataset, and other large-scale datasets. By adjusting the model structure and first model parameters of the source model, a target model to be trained can be constructed. It is understandable that the target task of the target model is similar to the source task of the source model, and the task data used by the source model is related to the task data used by the target model, so that the target model can use the general feature representation capability of the source model to perform feature processing on the image of the target task.

[0036] For example, the source model can be applied to, but is not limited to, source tasks such as 3D detection, target tracking, and behavior prediction in the field of autonomous driving. The source model needs to perform autonomous driving tasks by performing near-field perception of the surrounding environment images of the car. In this case, the surrounding environment images include, but are not limited to, at least one of fisheye images, ultra-wide-angle images, image acquisition data with a field of view greater than 170°, and image data that conforms to a nonlinear distortion model. The target model can be applied to, but is not limited to, parking space recognition and detection in automatic parking scenarios, and can specifically be used to find available parking spaces through surrounding environment images. In this embodiment, the target model can be, but is not limited to, a multi-task joint training model. The target task of the target model includes, but is not limited to, at least one of a parking space detection task, a parking space classification task, and an obstacle detection task, so as to output available parking spaces based on the detection results of at least one task.

[0037] Among them, after obtaining the source model and its first model parameters, this embodiment can fix the low-level parameters for extracting common features in the source model to retain the source model's ability to extract basic features such as edges and textures under the source task. In addition, the output layer structure of the source model is adjusted so that the target model obtained after the adjustment can be used to match the number of categories of the target task with the output dimension; or, based on the complexity of the target task, a task head is added after the fixed low layer so that the newly added task head can be applied to the detection output of the target task. In this way, a target model to be trained can be constructed based on the source model.

[0038] Step S12: Obtain a training data set, and input the training data set into the target model and the source model respectively to obtain the first feature data output by the source model and the second feature data output by the target model.

[0039] Furthermore, a training dataset for the target task is obtained. The training dataset includes images of the vehicle's surroundings and labeled data. The surrounding images are images of the vehicle's near-field area captured by the vehicle's perception system (e.g., a fisheye camera). Exemplarily, the training dataset includes multiple surrounding images, each with corresponding labeled data. The labeled data identifies at least one relevant parameter in the surrounding image, such as the location of a parking space detection frame, parking space dimensions, parking space category, obstacle category, obstacle detection frame location, and obstacle location.

[0040] After obtaining the training data set, the training data set is input into the target model and the source model, respectively, to obtain first feature data output by the source model, and second feature data output by the target model. After the training data set is input into the source model, the feature processing network of the source model performs feature extraction and multi-scale fusion on the surrounding environment image to output the first feature data; after the training data set is input into the target model, the feature processing network of the target model performs feature extraction and multi-scale fusion on the surrounding environment image to output the second feature data. The feature processing networks of the source model and the target model have certain similarities, and the first feature data and the second feature data at least partially overlap. The first feature data and the second feature data include, but are not limited to, at least one of a multi-scale feature map, a fused feature map, and a pyramid feature map.

[0041] Step S13: constructing a total loss function of the target model based on the difference data between the first feature data and the second feature data, so as to train the target model based on the total loss function.

[0042] After obtaining the first feature data and the second feature data, the degree of differentiation between the first feature data and the second feature data is calculated to obtain difference data. The difference data can be obtained by, but is not limited to, calculating at least one of the following methods: relative entropy (KL divergence loss), mean square error, mean absolute error, structural similarity index, mutual information, regional difference value grid method, Euclidean distance, VGG / ResNet features, etc., of the first feature data and the second feature data.

[0043] After obtaining the difference data between the first feature data and the second feature data, a total loss function of the target model can be constructed based on the difference data between the first feature data and the second feature data, so as to train the target model based on the total loss function. The specific training method may include, but is not limited to: after the target model completes processing of a batch of surrounding environment images, that is, completes one iteration, loss calculation is performed based on the total loss function, and gradient backpropagation is performed to update the model parameters of the target model. In this case, during the iterative update process, the first model parameters of the source model remain fixed.

[0044] In an embodiment of the present application, the model training method obtains a source model trained on a public data set and the first model parameters of the source model, adjusts the model structure and the first model parameters of the source model to construct a target model to be trained; obtains a training data set, and inputs the training data set into the target model and the source model respectively to obtain the first feature data output by the source model and the second feature data output by the target model; constructs a total loss function for the target model based on the difference data between the first feature data and the second feature data, and trains the target model based on the total loss function. Therefore, the model training method of this embodiment can construct a total loss function through the difference data between the first feature data and the second feature data, so that the target model can learn part of the feature processing capabilities of the source model, and transfer part of the capabilities of the source model to the target model, so that the target model can quickly adapt to the target task, reduce the data collection and labeling of the target model in the parking scenario, and improve the robustness of the target model.

[0045] In one embodiment, the first feature data is a first feature map output by the feature processing network of the source model, and the second feature data is a second feature map output by the feature processing network of the target model. Figure 2 , Figure 2 This is a flow chart of the second embodiment of the model training method provided by this application. Figure 2 As shown, in step S13, in constructing the total loss function of the target model based on the difference data of the first feature data and the second feature data, the model training method of this embodiment further includes:

[0046] Step S21: Calculate the relative entropy of the first feature map and the second feature map.

[0047] Specifically, the feature processing network of the source model is used to perform basic feature extraction and multi-scale feature fusion on the image data of the training dataset to obtain a multi-scale first feature map. Similarly, the feature processing network of the target model is used to perform basic feature extraction and multi-scale feature fusion on the image data of the training dataset to obtain a multi-scale second feature map, so as to represent the activation response of the input training dataset on a specific feature dimension through the first feature map and the second feature map. After obtaining the first feature map and the second feature map, the relative entropy of the first feature map and the second feature map is calculated.

[0048] Relative entropy is an indicator that measures the difference between two probability distributions, also known as KL divergence. When calculating relative entropy, the following steps may be included: normalizing all pixel values ​​or channel values ​​of the first feature map and the second feature map to convert the first feature map and the second feature map into a probability distribution; and calculating the relative entropy of the normalized first feature map and the second feature map. Specifically, the formula for relative entropy is as follows:

[0049] ;

[0050] in, is the relative entropy of the first and second feature maps; is the first feature map output by the source model; is the second feature map of the target model trained based on the parking scenario; N is the corresponding channel of the first feature map and the second feature map.

[0051] Step S22: The product of the relative entropy and the model transfer weight is used as the first loss function.

[0052] Specifically, a model migration weight is set. The model migration weight is used to represent the weight when migrating the source model to the target model. The larger the model migration weight, the more migrations there are and the greater the similarity between the source model and the target model; otherwise, the smaller the model migration weight is. The formula of the first loss function is as follows:

[0053] ;

[0054] in, is the first loss function; is the relative entropy of the first and second feature maps; is the model transfer weight.

[0055] Step S23: constructing a total loss function based on the first loss function.

[0056] After calculating the first loss function, a total loss function is constructed based on the first loss function. Specifically, the first loss function and other loss functions may be added together to obtain the total loss function.

[0057] Therefore, the model training method of this embodiment can calculate the relative entropy of the first feature map and the second feature map, use the product of the relative entropy and the model migration weight as the first loss function, and construct a total loss function based on the first loss function. Therefore, the relative entropy can constrain the probability distribution of the feature maps of the source model and the target model, so that the first loss function can be used to close the similarity of the feature processing networks of the target model and the source model, ensuring that the trained target model can learn part of the feature processing capabilities of the source model, improve the ability of the target model to adapt to the target task, reduce the data collection and labeling of the target model in the parking scenario, and improve the robustness of the target model.

[0058] Optionally, before the step of inputting the training data set into the target model and the source model respectively to obtain the first feature data output by the source model and the second feature data output by the target model in step S20, the model training method of this embodiment also includes: obtaining the training round of the target model; when the training round is less than or equal to a first preset threshold, executing the step of inputting the training data set into the target model and the source model respectively to obtain the first feature data output by the source model and the second feature data output by the target model; when the training round is greater than the first preset threshold, obtaining the task loss function of the target model to construct the total loss function of the target model based on the task loss function.

[0059] Specifically, the model training method of this embodiment trains the target model through multiple training rounds, where a training round is a period during which the target model completely traverses the training dataset. During each training round, the target model updates its weights through multiple iterations, gradually reducing the total loss function. Before inputting the training dataset into the target model and the source model, the current training round of the target model is first obtained.

[0060] When the current training round is less than or equal to a first preset threshold, the steps of inputting the training data set into the target model and the source model respectively to obtain the first feature data output by the source model and the second feature data output by the target model, so that the total loss function can be constructed based on the first loss function determined based on the first feature data and the second feature data.

[0061] When the number of training rounds exceeds a first preset threshold, the task loss function of the target model is obtained. The task loss function is used to make the target model regress and learn the processing capabilities of the specific target task, and the total loss function of the target model is constructed based on the task loss function. The target model also includes a task detection head, which is used to receive feature data output by the feature processing network and predict the feature data under the target task. The task loss function is used to measure the deviation between the prediction results of the task detection head and the labeled data of the training dataset, so as to provide a clear optimization direction for the target model.

[0062] Specifically, the model training method of this embodiment uses a first loss function to increase the similarity between the target model and the source model before a specified training round, completing model transfer learning. After the specified training round, a task loss function is used to encourage autonomous learning in the target model, thereby improving the target model's task prediction accuracy in parking scenarios and enhancing its robustness.

[0063] In one embodiment, see Figure 3 , Figure 3 This is a flow chart of the third embodiment of the model training method provided by this application. Figure 3 As shown, the training data set includes training image data and label data, and the target model includes a feature processing network and a task detection head. In step S13, the model training method of this embodiment further includes:

[0064] Step S41: constructing a first loss function of a feature processing network based on the difference data between the first feature data and the second feature data.

[0065] Specifically, the difference data between the first feature data and the second feature data can be represented by relative entropy or other parameters. Based on the difference data between the first feature data and the second feature data, a first loss function of the feature processing network is constructed.

[0066] Step S42: Obtain the prediction data output by the target model.

[0067] Specifically, prediction data refers to data related to the target task's prediction output by the task prediction head. For parking space detection, the prediction data may include, but is not limited to, prediction data on parking space detection box positions and parking space dimensions. For parking space classification, the prediction data may include, but is not limited to, prediction data on parking space categories. For obstacle detection, the prediction data may include, but is not limited to, obstacle categories, obstacle detection box positions, and obstacle locations.

[0068] Step S43: Based on the difference data between the predicted data and the label data, construct the task loss function of the task detection head.

[0069] Specifically, after obtaining the predicted data of the target model, the task loss function of the task detection head can be constructed based on the difference data between the predicted data and the label data.

[0070] Step S44: constructing a total loss function based on the sum of the first loss function and the task loss function, so as to train the target model based on the total loss function.

[0071] A total loss function is constructed based on the sum of the first loss function and the task loss function, so as to train the target model based on the total loss function. In a possible implementation, when the current training round of the target model is less than or equal to a first preset threshold, the above steps S41-S44 are executed; when the current training round of the target model is greater than the first preset threshold, the predicted data output by the target model is directly obtained, and based on the difference data between the predicted data and the label data, the task loss function of the task detection head is constructed, so as to train the target model using the task loss function as the total loss function.

[0072] In an embodiment of the present application, the model training method obtains the prediction data output by the target model, constructs the task loss function of the task detection head based on the difference data between the prediction data and the label data, and constructs the total loss function based on the sum of the first loss function and the task loss function, so as to train the target model based on the total loss function. Therefore, the model training method of this embodiment can use the first loss function to draw in the similarity between the feature processing network of the target model and the feature processing network of the source model, and optimize the prediction / classification ability of the task detection head through the task loss function, thereby guiding the direction and magnitude of the parameter adjustment of the target model and further improving the robustness of the target model.

[0073] Optionally, the predicted data includes a parking space detection box, and the task loss function includes a second loss function. Figure 4 and Figure 5 , Figure 4 is a flowchart of the fourth embodiment of the model training method provided by this application, Figure 5 is a schematic diagram of the first reference point in the parking space detection frame. Figure 4 and Figure 5 As shown, in this embodiment, step S43 includes the following steps:

[0074] S431: Calculating a first offset between a first reference point of the parking space detection frame and four parking space corner points of the parking space detection frame.

[0075] Specifically, in this embodiment, the target model includes a first detection head, which is used to implement the parking space detection task in the parking scenario. After the image of the training data set is input into the target model, the feature processing network of the target model extracts and fuses the features of the image to obtain a second feature map. The first detection head receives the second feature map input by the feature processing network and performs target detection on all pixels in the second feature map to predict and output the parking space detection frame in the second feature map. Among them, the parking space detection frame is usually related to the parking range of the parking space selected by the parking space line. In actual parking scenarios, the shapes of parking spaces are diverse. For example, the shapes of parking spaces can include horizontal parking spaces, vertical parking spaces, or inclined parking spaces. Since when the car uses the fisheye lens of the perception system to capture the image of the surrounding environment, the fisheye image will cause the shape of the parking space to be distorted. Therefore, the parking space detection frame of this embodiment has various forms, including but not limited to rectangular detection frames, long trapezoidal detection frames, or polygons.

[0076] The first reference point of the parking space detection frame is a pixel point within the parking space detection frame. The first reference point can be a pre-set pixel point, such as the midpoint of the parking space detection frame. The first reference point can also be a random pixel point within the parking space detection frame. The parking space detection frame includes at least one first reference point.

[0077] After obtaining the first reference point of the parking space detection frame, calculate the first offset between the first reference point of the parking space detection frame and the four parking space corner points of the parking space detection frame. For example, a two-dimensional coordinate system can be defined with the first reference point as the origin, and the position coordinates of the four parking space corner points of the parking space detection frame are expressed as (x1, y1), (x2, y2), (x3, y3) and (x4, y4) in the two-dimensional coordinate system, wherein the first offset can include the offset of the four parking space corner points from the first reference point in the x direction of the two-dimensional coordinate system, and the offset of the four parking space corner points from the first reference point in the y direction of the two-dimensional coordinate system. At this time, the first offset can include , , , , , , , .

[0078] S432: Obtain a second offset between the corresponding first reference point and the parking space corner point of the actual parking space value from the label data.

[0079] Specifically, the label data of the training image set includes the true parking space value, which indicates the position of the actual parking space in the image. Therefore, the second offset between the corresponding first reference point and the parking space corner point of the true parking space value can be obtained from the label data.

[0080] S433: Construct a second loss function based on the difference between the first offset and the second offset.

[0081] Specifically, after obtaining the first offset and the second offset, the difference between the first offset and the second offset is calculated to obtain the difference between the parking space detection frame predicted by the target model and the actual parking space. Specifically, the difference between the first offset and the second offset can be the offset difference in the x-direction or y-direction between the first reference point and the parking space detection frame and the actual parking space value. The formula for the second loss function is as follows:

[0082] ;

[0083] Wherein, N represents the number of first reference points; It represents the predicted offset in the x direction between the i-th first reference point and the j-th parking space corner point of the parking space detection frame. represents the predicted offset in the y direction between the i-th first reference point and the j-th parking space corner point of the parking space detection frame; Indicates the actual offset in the x direction between the i-th first reference point and the j-th parking space corner point of the actual parking space value, It represents the actual offset in the y direction between the i-th first reference point and the j-th parking space corner point of the actual parking space value.

[0084] The model method of this embodiment calculates a first offset between the first reference point of the parking space detection frame and the four corner points of the parking space detection frame, obtains a second offset between the corresponding first reference point and the corner points of the parking space's true value from the label data, and constructs a second loss function based on the difference between the first and second offsets. Unlike the prior art methods that directly use bounding rectangle detection or only detect the two corner points on the parking space entry line, this embodiment describes the shape and size of the parking space by introducing the offset between the first reference point and the parking space corner points, thereby reducing restrictions on the parking space shape and detection frame shape, which is beneficial for improving the prediction accuracy of the parking space detection task and further enhancing the robustness of the target model.

[0085] Optionally, the prediction data includes a first predicted probability value of the parking space, and the task loss function includes a third loss function. Figure 6 , Figure 6 This is a flow chart of the fifth embodiment of the model training method provided by this application. Figure 6 As shown, in this embodiment, step S43 includes the following steps:

[0086] S434: Obtain a first category label value corresponding to the parking space from the label data.

[0087] Specifically, in this embodiment, the target model includes a second detection head, which is used to perform parking space classification tasks in parking scenarios. After images from the training dataset are input into the target model, the target model's feature processing network extracts and fuses features from the images to obtain a second feature map. The second detection head then receives the second feature map from the feature processing network and performs object detection on all pixels in the second feature map to predict a first predicted probability value for at least one parking space in the second feature map.

[0088] The first prediction probability value is used to describe the probability that the parking space is available. The first prediction probability value can be a confidence value between 0 and 1. When the first prediction probability value is 1, the target model predicts that the parking space is available for parking. When the first prediction probability value is 0, the target model predicts that the parking space is not available for parking. The larger the first prediction probability value, the greater the probability that the target model predicts that the parking space is available for parking.

[0089] After obtaining the first predicted probability value, a first classification label value for the corresponding parking space is further obtained from the label data. The first classification label value is used to indicate whether the parking space is actually available for parking. The first classification label value is 0 or 1. A first classification label value of 0 indicates that the parking space is not available for parking, and a first classification label value of 1 indicates that the parking space is available for parking.

[0090] S435: Constructing a third loss function based on the degree of difference between the first predicted probability value and the first classification label value.

[0091] Specifically, after obtaining the first predicted probability value and the first classification label value, the difference between the first predicted probability value and the first classification label value is calculated to construct a third loss function based on the difference between the first predicted probability value and the first classification label value. The formula of the third loss function is as follows:

[0092] ;

[0093] in, is the third loss function, M represents the number of parking spaces on the second feature map of the input task detection head, 、 Represents the gradient return weight, which is used to measure the training effect. Represents the first classification label value, Represents the first predicted probability value.

[0094] Therefore, the model training method of this embodiment obtains the first classification label value of the corresponding parking space from the label data, and constructs a third loss function based on the degree of difference between the first predicted probability value and the first classification label value. The third loss function is used to measure the degree of difference between the predicted probability and the true label of the second detection head in the parking space classification task, so that by subsequently minimizing the loss function, the model's classification accuracy of whether the parking space is available can be improved, thereby further improving the robustness of the target model.

[0095] Optionally, the prediction data includes the obstacle's bounding box and the second predicted probability value. Figure 7 and Figure 8 , Figure 7 This is a flow chart of the sixth embodiment of the model training method provided by this application. Figure 8 is a schematic diagram of the second reference point in the external detection frame. Figure 7 As shown, in this embodiment, step S43 includes the following steps:

[0096] S436: Calculate the third offset between the second reference point of the obstacle and the four sides of the circumscribed detection frame, and obtain the fourth offset of the second reference point corresponding to the obstacle from the label data to construct a fourth loss function based on the difference between the third offset and the fourth offset.

[0097] Specifically, in this embodiment, the target model includes a third detection head, which is used to perform obstacle detection in parking scenarios. After images from the training dataset are input into the target model, the target model's feature processing network extracts and fuses features from the images to produce a second feature map. The third detection head then receives the second feature map from the feature processing network and performs object detection on all pixels in the second feature map, predicting at least one obstacle and outputting a bounding box for the obstacle and a second predicted probability value.

[0098] Among them, such as Figure 8 As shown, a second reference point for the obstacle can be defined. The selection method for the second reference point is similar to that of the first reference point described above and will not be repeated here. Because the target model of this embodiment, when applied to parking prediction and planning, requires the vehicle's fisheye camera to obtain a fisheye image of the surrounding environment, the shape of the obstacle in the fisheye image will be distorted. Therefore, this embodiment calculates the third offset between the second reference point of the obstacle and the four sides of the circumscribed detection box, and obtains the fourth offset corresponding to the second reference point of the obstacle from the label data, to construct a fourth loss function based on the difference between the third and fourth offsets.

[0099] Specifically, using the obstacle's second reference point as the origin, a two-dimensional coordinate system is established within the bounding box. The third offset between the obstacle's second reference point and the four edges of the bounding box can be expressed by the distances between the second reference point and the four edges on the coordinate axes. Similarly, the label data includes the obstacle's true bounding box. Therefore, the fourth offset between the corresponding second reference point and the four edges of the true bounding box can be obtained from the label data.

[0100] A fourth loss function is constructed based on the difference between the third offset and the fourth offset. Specifically, the difference between the third offset and the fourth offset can be expressed as the difference between the distance between the second reference point and one of the edges of the circumscribed detection frame and the distance between the second reference point and the same edge on the true circumscribed frame. The formula of the fourth loss function is as follows:

[0101] ;

[0102] in, is the fourth loss function; U is the number of the second benchmark points; 、 、 、 Respectively represent the predicted offset between one of the second reference points and the four edges of the external detection box, 、 、 、 They represent the predicted offsets between one of the second reference points and the four sides of the true bounding box.

[0103] Therefore, this embodiment can describe the shape and size of the obstacle by introducing an offset between the second reference point and the external detection frame of the obstacle, thereby reducing the restrictions on the shape of the obstacle and the shape of the detection frame, and reducing the impact of the distortion of the fisheye image on the detection, which is conducive to improving the prediction accuracy of the parking space detection task and further enhancing the robustness of the target model.

[0104] S437: Calculate a fifth offset of the ground point of the obstacle in the horizontal direction of the external detection frame, and obtain a sixth offset of the ground point corresponding to the obstacle from the label data, so as to construct a fifth loss function based on the difference between the fifth offset and the sixth offset.

[0105] In this embodiment, the center point on the bottom edge of the external detection frame is defined as the grounding point of the obstacle, and the fifth offset of the grounding point in the horizontal direction of the external detection frame is calculated. With the grounding point as the center, the direction from the grounding point along the bottom edge to one side edge is defined as the x-direction, and the direction from the bottom edge of the grounding point to the other side edge is defined as the y-direction. The fifth offset of the grounding point of the obstacle in the horizontal direction of the external detection frame can be represented by the position of the side edge of the external detection frame in the x-direction and the position of the side edge in the y-direction, i.e., the fifth offset includes dx and dy. Alternatively, the fifth offset can also be represented by the distance between the grounding point and the two side edges of the external detection frame.

[0106] Similarly, the label data includes the actual bounding box of the obstacle. Therefore, the offset of the ground point on the bottom edge of the actual bounding box relative to the side edge in the x and y directions can be obtained from the label data, which is the sixth offset of the ground point of the actual bounding box.

[0107] A fourth loss function is constructed based on the difference between the fifth offset and the sixth offset. Specifically, the difference between the fifth offset and the sixth offset may include: the offset between the ground point and the external detection frame in the x direction and the offset between the ground point and the true external frame in the x direction; the offset between the ground point and the external detection frame in the y direction and the offset between the ground point and the true external frame in the y direction. The formula for the fifth loss function is as follows:

[0108] ;

[0109] in, is the fifth loss function; H is the number of external detection boxes of obstacles; To represent the predicted offset between the ground point and the external detection box in the x direction, Indicates the predicted offset between the ground point and the external detection box in the y direction; Indicates the actual offset between the ground point and the actual external frame in the x direction. Indicates the actual offset between the ground point and the actual bounding box in the y direction.

[0110] Therefore, this embodiment can introduce a grounding point to describe the distance between the obstacle and the car, so that the target model can evaluate the effective parking space based on the impact of the distance between the obstacle and the car on the obstacle in the parking path; by introducing the fifth loss function, the impact of the distortion of the fisheye image on the judgment of the grounding point can be effectively reduced, which is conducive to improving the prediction accuracy of the parking space detection task and further enhancing the robustness of the target model.

[0111] S438: Obtain a second classification label value corresponding to the obstacle from the label data, and construct a sixth loss function based on the difference between the second predicted probability value and the second classification label value.

[0112] Among them, the second predicted probability value is used to describe the credibility of the classification prediction of the obstacle. The second predicted probability value can be confidence data between 0 and 1; when the second predicted probability value is 1, the target model's prediction of the category of the obstacle is credible; when the second predicted probability value is 0, the target model's prediction of the category of the obstacle is unreliable. After obtaining the second predicted probability value, the second classification label value of the corresponding parking space is further obtained from the label data. The second classification label value is the true category label of the obstacle. The sixth loss function is constructed based on the degree of difference between the second predicted probability value and the second classification label value. The formula of the sixth loss function is as follows:

[0113] ;

[0114] Where F represents the number of obstacle samples; 、 Represents the gradient backpropagation weight, which is used to measure the training effect; Represents the second classification label value, represents the second predicted probability value.

[0115] Therefore, this embodiment can introduce the sixth loss function to facilitate the subsequent minimization of the loss function, thereby improving the accuracy of the target model in classifying obstacles and further improving the robustness of the target model.

[0116] S439: Construct a task loss function based on the sum of the fourth loss function, the fifth loss function, and the sixth loss function.

[0117] Based on the sum of the fourth loss function, the fifth loss function, and the sixth loss function, a task loss function is constructed so that the task loss function can be used to balance the multi-task processing effect of the target model and further improve the robustness of the target model.

[0118] Optionally, in one embodiment, steps S431-S433 may be performed after steps S434-S435 are completed. For example, after executing steps S434-S435, the model method further includes: obtaining parking spaces in a surrounding environment image whose first predicted probability value is greater than a second preset threshold, and obtaining position parameters of the corresponding parking spaces in a feature map of the surrounding environment image; and executing steps S431-S433 based on the position parameters to perform parking space detection for the corresponding parking spaces in the feature map of the surrounding environment image, thereby reducing the amount of computation.

[0119] In another embodiment, steps S431-S433, steps S434-S435, and steps S436-S439 can be performed simultaneously. After completing steps S431-S433, steps S434-S435, and steps S436-S439, the model method further includes: using the sum of the first loss function, the second loss function, the third loss function, the fourth loss function, the fifth loss function, and the sixth loss function as the total loss function; or using the sum of the second loss function, the third loss function, the fourth loss function, the fifth loss function, and the sixth loss function as the total loss function.

[0120] The present application also provides an automatic parking method, which is applied to automobiles. Figure 9 , Figure 9 This is a flow chart of the first embodiment of the automatic parking method provided by this application. Figure 9 As shown, in this embodiment, the automatic parking method includes:

[0121] Step S51: Acquire a fisheye image of the car.

[0122] Specifically, a vehicle may include a perception system comprising multiple fisheye cameras mounted on the vehicle body or in various mirror locations. The vehicle is configured to capture the surrounding environment through the fisheye cameras to obtain fisheye images. The automated parking method of this embodiment acquires the fisheye images of the vehicle and uses the target model to perform parking space detection, parking space classification, and obstacle detection on the fisheye images.

[0123] Step S52: input the fisheye image into the target model trained by the model training method of any of the above embodiments.

[0124] Specifically, the target model may include a first detection head, a second detection head, and a third detection head. The first detection head is used to detect parking spaces in the fisheye image and output parking space detection data; the second detection head is used to classify parking spaces in the fisheye image and output parking space classification data; and the third detection head is used to detect obstacles in the fisheye image and output obstacle detection data.

[0125] Understandably, the existing technology, when performing automatic parking, usually detects parking spaces using images captured by fisheye cameras, detects obstacles using information collected by ultrasonic radars, and outputs final parking space location information by combining the information from the two sensors: the fisheye camera and the ultrasonic radar. However, the use of ultrasonic sensors for detection requires the addition of an ultrasonic information processing module and an ultrasonic information and parking space detection information fusion module to the automatic parking system, which makes the entire solution more complex. Different from the existing technology, the automatic parking method of this embodiment adopts a multi-task joint learning structure, in which the first detection head, the second detection head, and the third detection head respectively perform different tasks on the perceived fisheye image, which can effectively improve the utilization of the fisheye image and output more accurate obstacle information. This eliminates the need for redundant ultrasonic processing and simplifies the system structure.

[0126] Step S53: Obtain the available parking space information of the car output by the target model.

[0127] Specifically, the target model can integrate parking space detection data, parking space classification data, and obstacle detection data to output available parking space information for the car. The available parking space information includes, but is not limited to, the coordinates of the available parking spaces, the path to the available parking spaces, and the car control information during the path.

[0128] Step S54: Based on the parking space information of the target model, control the car to park in the parking space.

[0129] After obtaining the parking space information of the target model, the car is controlled to park in the parking space based on the parking space information, thereby realizing automatic parking.

[0130] Alternatively, see Figure 10 , Figure 10 FIG. 1 is a flow chart of the second embodiment of the automatic parking method provided by this application. Figure 10 As shown, this automated parking method employs a target model trained on multiple tasks and employs a model transfer strategy to transfer some of the learning capabilities of the source model from an existing public dataset to the target model. This allows the target model to output detection data for tasks including parking space detection, parking space classification, and obstacle detection after inputting fisheye data. This effectively utilizes data from public datasets for existing scenarios, enabling end-to-end recognition of available parking spaces and significantly improving the robustness of the model. Furthermore, this method overcomes the problem of existing automated parking methods requiring a post-processing fusion module for information integration and the inability to achieve a unified end-to-end model output for all target information. This overcomes the storage-unfriendly nature of multi-stage methods.

[0131] See Figure 11 , Figure 11FIG. 1 is a schematic diagram of the structure of an embodiment of the automatic parking system provided by the present application. Figure 6 As shown, the automatic parking system 50 of this embodiment includes a memory 52 and a processor 51, wherein the processor 51 is connected to the memory 52. ​​The memory 52 is used to store program instructions. The processor 51 is used to execute the program instructions stored in the memory 52 to implement the method described in any of the above embodiments.

[0132] The processor 51 may also be referred to as a CPU (Central Processing Unit). The processor 51 may be an integrated circuit chip with signaling processing capabilities. The processor 51 may also be a general-purpose processor, a digital signaling processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic device, a discrete gate or transistor logic device, or a discrete hardware component. The general-purpose processor may be a microprocessor or any conventional processor.

[0133] The memory 52 can be a memory stick, a TF card, etc., which can store all the information in the automatic parking system 50, including the input raw data, computer programs, intermediate operation results and final operation results are all stored in the memory. It stores and retrieves information according to the location specified by the controller. With the memory, the string matching prediction device has a memory function and can ensure normal operation. The memory of the string matching prediction device can be divided into main memory (internal memory) and auxiliary memory (external memory) according to its purpose. There is also a classification method of dividing it into external memory and internal memory. External memory is usually a magnetic medium or an optical disk, etc., which can store information for a long time. Memory refers to the storage component on the motherboard, which is used to store the data and programs currently being executed, but is only used to temporarily store programs and data. If the power is turned off or the power is cut off, the data will be lost.

[0134] In the several embodiments provided in this application, it should be understood that the disclosed methods and devices can be implemented in other ways. For example, the methods described above are only schematic. For example, the division of modules or units is only a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0135] Units described as separate components may or may not be physically separate, and components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0136] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0137] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the existing technology, or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a number of instructions for causing a computer device (which can be a personal computer, system server, or network device, etc.) or a processor to execute all or part of the steps of the various implementation methods of the present application.

[0138] See also Figure 12 , Figure 12 This is a schematic diagram of the structure of an embodiment of the computer-readable storage medium provided by this application. Figure 12 As shown, the computer-readable storage medium of the present application stores program instructions 61 capable of implementing all of the above-mentioned methods. The program instructions 61 may be stored in the aforementioned storage medium in the form of a software product, including several instructions for causing a computer device (which may be a personal computer, server, or network device, etc.) or a processor to execute all or part of the steps of the various embodiments of the present application. The aforementioned storage device includes various media capable of storing program code, such as a USB flash drive, a mobile hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk, or electronic devices such as a computer, server, mobile phone, or tablet.

[0139] The above is only an implementation method of the present application and does not limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the description and drawings of this application, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A model training method, characterized in that: The model training method includes: Obtaining a source model trained on a public dataset and first model parameters of the source model, and adjusting the model structure of the source model and the first model parameters to construct a target model to be trained; Obtaining a training data set, and inputting the training data set into the target model and the source model respectively to obtain first feature data output by the source model and second feature data output by the target model; constructing a total loss function of the target model based on difference data between the first feature data and the second feature data, so as to train the target model based on the total loss function; The first feature data is a first feature map output by the feature processing network of the source model, and the second feature data is a second feature map output by the feature processing network of the target model; The constructing the total loss function of the target model based on the difference data between the first feature data and the second feature data includes: Calculating relative entropy between the first feature map and the second feature map; Using the product of the relative entropy and the model migration weight as a first loss function; Constructing the total loss function based on the first loss function; The training data set includes an image of the surrounding environment of the vehicle and label data. The surrounding environment image is an image of the near-field area of ​​the vehicle collected by the vehicle's perception system. The source model is used to perform the autonomous driving task by performing near-field perception on the surrounding environment image. The target model is used to find available parking spaces using the surrounding environment image. The surrounding environment image includes at least one of a fisheye image, an ultra-wide-angle image, image acquisition data with a field of view angle greater than 170°, and image data that conforms to a nonlinear distortion model.

2. The model training method according to claim 1, characterized in that Before the step of inputting the training data set into the target model and the source model respectively to obtain the first feature data output by the source model and the second feature data output by the target model, the method includes: Obtaining the training rounds of the target model; When the number of training rounds is less than or equal to a first preset threshold, performing the step of inputting the training data set into the target model and the source model respectively to obtain first feature data output by the source model and second feature data output by the target model; When the training round is greater than the first preset threshold, the task loss function of the target model is obtained to construct a total loss function of the target model based on the task loss function.

3. The model training method according to claim 1, characterized in that The training data set includes training image data and label data, and the target model includes a feature processing network and a task detection head; constructing a total loss function of the target model based on the difference data between the first feature data and the second feature data, and training the target model based on the total loss function, including: constructing a first loss function of the feature processing network based on difference data between the first feature data and the second feature data; Obtaining prediction data output by the target model; Constructing a task loss function of the task detection head based on difference data between the predicted data and the label data; The total loss function is constructed based on the sum of the first loss function and the task loss function, so as to train the target model based on the total loss function.

4. The model training method according to claim 3, characterized in that The predicted data includes a parking space detection frame, and the task loss function includes a second loss function; The step of constructing a task loss function of the task detection head based on difference data between the predicted data and the label data includes: Calculating a first offset between a first reference point of the parking space detection frame and four parking space corner points of the parking space detection frame; Obtaining a second offset between the corresponding first reference point and a parking space corner point of a true parking space value from the label data; The second loss function is constructed based on the difference between the first offset and the second offset.

5. The model training method according to claim 3, characterized in that: The predicted data includes a first predicted probability value of the parking space, and the task loss function includes a third loss function; The step of constructing a task loss function of the task detection head based on difference data between the predicted data and the label data includes: Obtaining a first classification label value corresponding to the parking space from the label data; The third loss function is constructed based on the degree of difference between the first predicted probability value and the first classification label value.

6. The model training method according to claim 3, characterized in that The prediction data includes an external detection frame of the obstacle and a second prediction probability value; The step of constructing a task loss function of the task detection head based on difference data between the predicted data and the label data includes: Calculating a third offset between a second reference point of the obstacle and four sides of the circumscribed detection box, and obtaining a fourth offset corresponding to the second reference point of the obstacle from the label data, so as to construct a fourth loss function based on a difference between the third offset and the fourth offset; Calculating a fifth offset of the ground point of the obstacle in the horizontal direction of the circumscribed detection frame, and obtaining a sixth offset of the ground point corresponding to the obstacle from the label data, so as to construct a fifth loss function based on a difference between the fifth offset and the sixth offset; Obtaining a second classification label value corresponding to the obstacle from the label data, and constructing a sixth loss function based on a degree of difference between the second predicted probability value and the second classification label value; The task loss function is constructed based on the sum of the fourth loss function, the fifth loss function, and the sixth loss function.

7. An automatic parking method, characterized in that: Applied to an automobile, the automatic parking method includes: Acquire a fisheye image of the car; Inputting the fisheye image into a target model trained by the model training method according to any one of claims 1 to 6; Acquiring available parking space information of the vehicle output by the target model; Based on the parking space information of the target model, the vehicle is controlled to park in the parking space.

8. An automatic parking system, characterized in that: The system comprises a processor and a memory, wherein the processor is connected to the memory, The memory stores program instructions; The processor is configured to execute program instructions stored in the memory to implement the method according to any one of claims 1 to 7.

9. A computer-readable storage medium, characterized in that The computer-readable storage medium stores program instructions, and the program instructions can be executed by a processor to implement the method according to any one of claims 1 to 7.

Images