Panoramic look-around calibration method and electronic equipment

Abstract

The invention relates to the field of data processing, and provides a panoramic look-around calibration method and electronic equipment. The method comprises the following steps: receiving an instruction of panoramic look-around calibration, obtaining panoramic look-around information, and carrying out panoramic look-around calibration on the panoramic look-around information by utilizing a corner calibration mode to obtain a first calibration result; in response to the fact that the first calibration result is a calibration error, a deep learning model is called, correct panoramic look-around calibration is completed on the panoramic look-around information through the deep learning model, and the deep learning model is a neural network model which is obtained through training in advance and can conduct panoramic look-around calibration. In this way, the deep learning model can be utilized to make up for the problem that a traditional angular point calibration mode is single in applicable scene, traditional angular point calibration and the deep learning model are matched with each other, then the panoramic look-around calibration conditions of various environment scenes can be met, the application range is wide, the accuracy is high, and the robustness of panoramic look-around calibration is improved.

Description

Technical Field

[0001] This application relates to the technical field of data processing, and in particular, to a calibration method for panoramic surround view and an electronic device. Background Art

[0002] At present, the panoramic surround view function of vehicles is becoming more and more popular. To accurately use the panoramic surround view function, it is necessary to perform calibration processing on the vehicle for panoramic surround view to ensure the accuracy of the angular position of the formed panoramic surround view image.

[0003] However, traditional calibration methods for panoramic surround view directly perform joint calibration based on the corner points of the vehicle. This calibration method is relatively single. If the vehicle is in a relatively complex scene, the traditional corner point calibration method is prone to calibration errors. This will make the position of the collected panoramic surround view image inaccurate and affect the driving experience of the vehicle. Summary of the Invention

[0004] In view of this, the purpose of this application is to propose a calibration method for panoramic surround view and an electronic device, so as to solve the problem that the current corner point calibration method for panoramic surround view is relatively single, cannot adapt to complex vehicle scenarios, and is prone to calibration errors.

[0005] Based on the above purpose, this application provides a calibration method for panoramic surround view, which is applied to the vehicle end. The method includes:

[0006] Receiving an instruction for panoramic surround view calibration, obtaining panoramic surround view information, and performing panoramic surround view calibration on the panoramic surround view information by using a corner point calibration method to obtain a first calibration result;

[0007] In response to the first calibration result being a calibration error, invoking a deep learning model, and using the deep learning model to complete correct panoramic surround view calibration on the panoramic surround view information, where the deep learning model is a neural network model that has been pre-trained to be able to perform panoramic surround view calibration.

[0008] Through the above scheme, since the traditional corner point calibration method is relatively simple and fast, after the vehicle receives the panoramic view calibration instruction, it will first use the traditional corner point calibration method to perform the traditional panoramic view calibration on the acquired panoramic view information, and obtain the first calibration result. If the first calibration result is wrong, it proves that the environmental scene calibrated by the traditional corner point calibration method is single and cannot adapt to the current environmental scene of the vehicle. The current environmental scene will be disturbed by light or surrounding background images, etc. The traditional corner point calibration method cannot adapt to various environmental scenes disturbed by light and surrounding background images. At this time, it is necessary to use a deep learning model with a wider range of applications to correctly calibrate the panoramic view information. Since the deep learning model is a neural network model obtained by pre-training the panoramic view information of various environmental scenes, the deep learning model is used to perform a panoramic view calibration on the corresponding panoramic view information when the first calibration result is wrong, and then the panoramic view calibration process is correctly completed. In this way, the deep learning model can be used to make up for the problem that the traditional corner point calibration method is only applicable to a single scenario. The traditional corner point calibration and the deep learning model work together to meet the panoramic view calibration needs of various environmental scenarios. It has a wide range of applications, high accuracy, and improves the robustness of the panoramic view calibration.

[0009] In some embodiments, the using of a deep learning model to correctly calibrate the panoramic view information includes:

[0010] Performing panoramic surround view calibration on the panoramic surround view information using a deep learning model to obtain a second calibration result;

[0011] In response to the second calibration result being correct, determining that the panoramic view calibration process is completed; or,

[0012] In response to the second calibration result being a calibration error, sending the panoramic surround view information with the calibration error to the service platform, so that the server platform trains the deep learning model according to the panoramic surround view information with the calibration error to obtain a trained deep learning model;

[0013] Receive the trained deep learning model sent by the service platform, use the trained deep learning model to process the panoramic view data, and complete the panoramic view calibration process.

[0014] Through the above solution, after the vehicle terminal uses the deep learning model to perform panoramic calibration processing on the panoramic surround view data, if a calibration error occurs, it proves that the processing accuracy of the deep learning model is relatively low. Therefore, in order to improve the accuracy of the deep learning model, the panoramic surround view data with calibration errors will be sent to the service platform; then the service platform will continue to train the deep learning model, and then obtain a trained deep learning model. After sending it to the vehicle terminal, the vehicle terminal can use the trained deep learning model to perform accurate panoramic surround view calibration. Since the trained deep learning model is obtained by the service platform continuing to train on the basis of the original deep learning model and continuously correcting the defects of the original deep learning model. The trained deep learning model obtained after defect correction can perform correct panoramic surround view calibration based on the current environment of the vehicle terminal. In this way, the trained deep learning model adds the current environment scene of the vehicle terminal on the basis of the original scenes that can perform panoramic surround view calibration, making the scenes where the trained deep learning model performs correct panoramic surround view calibration more.

[0015] Based on the same inventive concept, the present application also provides a calibration method for panoramic surround view, which is applied to a service platform. The method includes:

[0016] Receiving the panoramic surround view data with calibration errors fed back by the vehicle terminal. Among them, when the vehicle terminal uses the deep learning model to perform panoramic surround view calibration on the panoramic surround view data, the panoramic surround view data will be fed back after a calibration error occurs;

[0017] Preprocessing the panoramic surround view data, and using the preprocessed panoramic surround view data to train the deep learning model to obtain a trained deep learning model;

[0018] Sending the trained deep learning model to the vehicle terminal for the vehicle terminal to use the trained deep learning model to correctly calibrate the panoramic surround view data.

[0019] Through the above solution, when the deep learning model of the vehicle terminal has a panoramic surround view calibration error, the service platform can use the panoramic surround view data with calibration errors fed back by the vehicle terminal. After preprocessing it, make its calibration have correct panoramic surround view data, and use it as a training sample to retrain the original deep learning model, continuously correct the defects of the deep learning model, so that the trained deep learning model obtained through the correction process increases the ability to correctly calibrate the panoramic surround view data with calibration errors fed back by the vehicle terminal.

[0020] In some embodiments, the service platform includes: a data middle platform and a training device, where the deep learning model is stored in the training device;

[0021] The preprocessing of the panoramic view data and training of the deep learning model using the preprocessed panoramic view data to obtain a trained deep learning model includes:

[0022] Filtering and cleaning the panoramic view data using the data center, calibrating the cleaned panoramic view data according to the correct calibration data to obtain calibrated panoramic view data, and sending the calibrated panoramic view data to the training device;

[0023] The training device is used to train the deep learning model according to the calibrated panoramic view data, and according to the difference between the training output result and the correct calibration data, the parameters in the deep learning model are iteratively trained and adjusted until the training output result matches the correct calibration data, thereby obtaining a trained deep learning model.

[0024] Through the above scheme, the filtering, cleaning and calibration of the panoramic view data can be completed through the cooperation of the data middle platform and the training equipment, so as to avoid noise infection in the panoramic view data and ensure the accuracy of the calibrated panoramic view data. According to the calibrated panoramic view data, the deep learning model is trained and processed, so that the trained deep learning model has the ability to perform panoramic view calibrate on the panoramic view data of the current scene, effectively improving the accuracy.

[0025] In some embodiments, the deep learning model is a target detection network model;

[0026] The deep learning model is trained using the training device according to the calibrated panoramic view data, and the parameters in the deep learning model are iteratively trained and adjusted according to the difference between the training output result and the correct calibration data until the training output result matches the correct calibration data, thereby obtaining a trained deep learning model, including:

[0027] Using the training device, perform:

[0028] Resizing the calibrated panoramic view data, and inputting the resized panoramic view data into the target detection network model for panoramic view calibration processing to obtain a model processing result;

[0029] Using a non-maximum suppression algorithm, redundant detection frames in the model processing result are removed to obtain a trained angle calibration result as a training output result;

[0030] Comparing the training output result with the correct calibration data to determine the degree of difference;

[0031] The adjustment parameters of each network layer of the target detection network model are determined according to the degree of difference, and each network layer of the target detection network model is adjusted and corrected according to the adjustment parameters to obtain a trained deep learning model.

[0032] Through the above scheme, on the basis of the environment scene in which the original target detection network model (YOLO model) can perform panoramic view calibration, the process of retraining the calibrated panoramic view data is added. During the training process, the non-maximum suppression algorithm is used to remove the redundancy of the training output results of the target detection network model to improve the accuracy of the obtained training output results. Since the training output results are more accurate, the degree of difference between them and the correct calibration data is determined more accurately, and then according to the difference program, the parameters of each network layer of the target detection network model can be better adjusted, making the training process smoother and the accuracy of the deep learning model completed by training higher.

[0033] In some embodiments, sending the trained deep learning model to the vehicle side includes:

[0034] Using the training device, perform:

[0035] Generate a panoramic surround view calibration development tool based on the trained deep learning model;

[0036] Performing panoramic surround view calibration verification processing according to the panoramic surround view calibration development tool to obtain a calibration verification result;

[0037] In response to the calibration verification result being wrong, the verification panoramic surround view data used for calibration verification is fed back to the data center, and the data center preprocessing and the training device training processing process are repeated until the obtained calibration verification result is correct;

[0038] In response to the calibration verification result being correct, the panoramic view calibration development tool is sent to the vehicle side, so that the vehicle side can execute the function of the panoramic view calibration development tool to perform correct panoramic view calibration using the trained deep learning model.

[0039] Through the above scheme, the trained deep learning model will be generated into a corresponding panoramic view calibration development tool, which is convenient for using the panoramic view calibration development tool to perform the calibration verification process. In this way, the accuracy of the trained deep learning model can be determined in combination with the calibration verification result. After determining that the calibration verification result is correct, the panoramic view calibration development tool is sent to the vehicle end, thus ensuring the accuracy of the panoramic view calibration development tool sent to the vehicle end for panoramic view calibration.

[0040] In some embodiments, the service platform further includes: a cockpit simulation platform;

[0041] Performing panoramic surround calibration verification processing according to the panoramic surround calibration development tool to obtain a calibration verification result, including:

[0042] Using the cockpit simulation platform to simulate the vehicle and the camera installation positions on the vehicle, and cooperating with the training device to determine a virtual calibration scenario;

[0043] In the virtual calibration scenario, the training device uses the panoramic surround calibration development tool to control the cockpit simulation platform to perform virtual calibration adjustment of the panoramic surround, and displays the result obtained from the virtual calibration adjustment for the user to make a determination based on the displayed panoramic surround data after calibration verification;

[0044] Using the training device to receive the determination result for the panoramic surround data after calibration verification, and taking the determination result as the calibration verification result.

[0045] Through the above solution, it is possible to complete the process of simulating calibration verification of the panoramic surround calibration development tool through the mutual cooperation of the training device (for example, the computer side) and the cockpit simulation platform (cockpit bench, dynamic cockpit simulator, and calibration device), and then obtain an accurate calibration verification result. Only in this way can it be determined whether to continue training the deep learning model based on the accuracy of the calibration verification result, reducing the situation where the panoramic surround calibration development tool is directly sent to the vehicle end and calibration errors occur again at the vehicle end.

[0046] In some embodiments, performing panoramic surround calibration verification processing according to the panoramic surround calibration development tool to obtain a calibration verification result, including:

[0047] Using the training device to perform calibration verification on the panoramic surround data according to the panoramic surround calibration development tool, and displaying the panoramic surround data after calibration verification obtained for the user to make a determination based on the displayed panoramic surround data after calibration verification;

[0048] Using the training device to receive the determination result for the panoramic surround data after calibration verification, and taking the determination result as the calibration verification result.

[0049] Through the above solution, the process of performing calibration verification on the panoramic surround calibration development tool is executed in the training device, which is simple and convenient to operate. And only after the calibration verification process can it be determined whether to continue training the deep learning model based on the accuracy of the calibration verification result, reducing the situation where the panoramic surround calibration development tool is directly sent to the vehicle end and calibration errors occur again at the vehicle end.

[0050] In some embodiments, the using the training device to perform: in response to the calibration verification result being wrong, feeding back the verification panoramic surround view data used for calibration verification to the data center, includes:

[0051] In response to the calibration verification result being wrong, using the training device to expand the verification panoramic view data used for calibration verification into a plurality of different verification panoramic view data in an incremental expansion manner;

[0052] The training device feeds back a plurality of different verified panoramic view data to the data center, so that the data center performs iterative training processing based on the plurality of different verified panoramic view data in combination with the training device.

[0053] Through the above scheme, after the calibration verification result is wrong, the panoramic view data is verified by incremental expansion, and the data middle station and training equipment are used again to perform the training processing process, so that there can be multiple data training references, which improves the accuracy of training and makes the deep learning model obtained after training more accurate.

[0054] Based on the same inventive concept, the present application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable by the processor, wherein the processor implements the method as described above when executing the computer program.

[0055] From the above, it can be seen that the panoramic view calibration method and electronic device provided by the present application are relatively simple and quick because the traditional corner point calibration method is relatively simple and quick. Therefore, after the vehicle receives the panoramic view calibration instruction, it will first use the traditional corner point calibration method to perform the traditional panoramic view calibration on the acquired panoramic view information, and obtain the first calibration result. If the first calibration result is wrong, it proves that the environmental scene calibrated by the traditional corner point calibration method is single and cannot adapt to the environmental scene where the vehicle is currently located. The current environmental scene will be disturbed by light or surrounding background images, etc. The traditional corner point calibration method cannot adapt to various environmental scenes disturbed by light and surrounding background images. At this time, it is necessary to use a deep learning model with a wider range of applications to correctly calibrate the panoramic view information. Since the deep learning model is a neural network model obtained by pre-training the panoramic view information of various environmental scenes, the deep learning model is used to perform the panoramic view calibration again on the corresponding panoramic view information when the first calibration result is wrong, so as to correctly complete the panoramic view calibration process. In this way, the deep learning model can be used to make up for the problem that the traditional corner point calibration method is only applicable to a single scenario. The traditional corner point calibration and the deep learning model work together to meet the panoramic view calibration needs of various environmental scenarios. It has a wide range of applications, high accuracy, and improves the robustness of the panoramic view calibration. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] To more clearly illustrate the technical solutions in the present application or related technologies, the following will briefly introduce the drawings required for use in the embodiments or related technology descriptions. Obviously, the drawings in the following descriptions are only the embodiments of the present application. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings.

[0057] Figure 1 Schematic diagram for identifying the corner points of traditional calibration black blocks;

[0058] Figure 2 Schematic diagram for the situation of double image misalignment in the calibration effect;

[0059] Figure 3 Flowchart of the panoramic surround calibration method according to an embodiment of the present application;

[0060] Figure 4 Schematic diagram of the process of sending the panoramic surround information with calibration errors in the embodiments of the present application to the service platform;

[0061] Figure 5 Flowchart of the panoramic surround calibration method according to another embodiment of the present application;

[0062] Figure 6 Schematic diagram of the structure of the YOLO model in the embodiments of the present application;

[0063] Figure 7 Schematic diagram of the logic of the panoramic surround calibration method corresponding to Solution 1 of the present application;

[0064] Figure 8 Schematic diagram of the logic of the panoramic surround calibration method corresponding to Solution 2 of the present application;

[0065] Figure 9 Block diagram of the structure of the panoramic surround calibration device according to an embodiment of the present application;

[0066] Figure 10 Block diagram of the structure of the panoramic surround calibration device according to another embodiment of the present application;

[0067] Figure 11 Schematic diagram of the electronic device in the embodiments of the present application. Detailed implementation manners

[0068] To make the objectives, technical solutions, and advantages of the present application clearer and more understandable, the following further elaborates on the present application in detail with reference to specific embodiments and the accompanying drawings.

[0069] It should be noted that, unless otherwise defined, the technical terms or scientific terms used in the embodiments of the present application should have the ordinary meanings understood by those with ordinary skills in the field to which the present application belongs. The "first", "second" and similar terms used in the embodiments of the present application do not indicate any order, quantity or importance, but are only used to distinguish different components. Words such as "including" or "comprising" mean that the elements or objects appearing before the word cover the elements or objects listed after the word and their equivalents, without excluding other elements or objects. Words such as "connected" or "linked" are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. "Upper", "lower", "left", "right", etc. are only used to represent relative position relationships. When the absolute position of the object being described changes, the relative position relationship may also change accordingly.

[0070] Glossary:

[0071] AVM: Around View Monitor, panoramic surround view technology.

[0072] PC: Personal Computer, personal computer.

[0073] OTA: Over-The-Air technology, over-the-air technology.

[0074] SDK: Software Development Kit, software development kit.

[0075] YOLO: You Only Look Once, an algorithm for object detection using convolutional neural networks.

[0076] NMS, Non-Maximum Suppression, non-maximum suppression algorithm.

[0077] QNX, Quick UNIX, operating system.

[0078] The panoramic surround view system (AVM) consists of surround view cameras in the front, rear, left and right of the vehicle and a cockpit controller. After processing the images collected by each camera, the panoramic surround view image result of the vehicle is obtained for displaying the surrounding environment information of the vehicle.

[0079] During the development of the current panoramic surround view system (AVM) function, due to the assembly differences in the installation positions of the surround view cameras of the same vehicle model, it is necessary to perform calibration work on this vehicle model.

[0080] Traditional corner calibration algorithm process: calibration black block corner recognition → camera joint calibration → distortion removal → AVM function calibration.

[0081] Traditional calibration of black block corner recognition (such as Figure 1 The process is as follows:

[0082] Binarize the calibration scene data → polygon detection → quadrilateral screening → quadrilateral corner point extraction.

[0083] Specifically: binarization processing is performed on the image data collected from the calibration scene;

[0084] Perform polygon detection on the binarized image data to determine the polygonal outlines of each target in the image data;

[0085] The polygonal contours of each target are screened using the quadrilateral model, and the quadrilateral contours (i.e., black blocks) are screened out to obtain the quadrilateral targets;

[0086] The corner points of the quadrilateral target are extracted to obtain the black block corner points.

[0087] The calibration work is divided into vehicle off-line calibration and after-sales calibration. The calibration environment of off-line calibration is relatively simple, and it is carried out in the production line calibration room. However, due to the large production line schedule, the production line will be contaminated to varying degrees, which will lead to a complex calibration environment (light, brightness, degree of reflection, degree of dirtiness of the calibration block, etc. cannot be consistent with the new site); the calibration environment of after-sales calibration is relatively complex, and there is no standard calibration room for calibration. Usually, calibration boards are placed outdoors or indoors for calibration. The outdoor or indoor environment is relatively complex, and the light, brightness, degree of reflection, degree of dirtiness of the calibration block, etc. are relatively poor compared to the calibration room.

[0088] Due to the differences in calibration environments, large production line output, and a large number of after-sales maintenance vehicles, the traditional corner point calibration algorithm has low robustness and cannot fully cover all scenarios. It is easy for traditional algorithms in complex production line / after-sales scenarios to cause calibration black block detection offset / failure to recognize, which in turn leads to ghosting and misalignment of the calibration effect (such as Figure 2 As shown in the figure), the calibration may fail or the calibration effect may be poor.

[0089] The embodiments of the present application are described in detail below with reference to the accompanying drawings.

[0090] The panoramic view calibration method proposed in the embodiment of the present application is applied to a vehicle end, which is arranged on a vehicle.

[0091] like Figure 3 As shown, the method includes:

[0092] Step 101, receiving a panoramic view calibration instruction, acquiring panoramic view information, and performing panoramic view calibration on the panoramic view information by using a corner point calibration method to obtain a first calibration result.

[0093] During specific implementation, after the calibration personnel determine that the vehicle meets the calibration conditions (the vehicle is stationary and in the environment where each calibration corner point is located), the vehicle terminal issues an instruction for panoramic surround view calibration and starts the panoramic surround view calibration program. Among them, the specific start methods include at least one of the following: button start, voice start, gesture action start, image control start, touch shape start, and facial expression start.

[0094] Multiple cameras are installed on the vehicle, and image data in various directions for panoramic surround view calibration are collected using these cameras, and these image data are combined as panoramic surround view information.

[0095] A traditional corner point calibration program is configured on the vehicle terminal. Since the corner point calibration method is relatively simple and fast, in order to improve the efficiency of panoramic surround view calibration, the traditional corner point calibration method is preferentially selected for implementation. After calibrating the panoramic surround view information through the traditional corner point calibration method, if each calibration corner point (for example, the four corner points of a black block) can be accurately and clearly identified to determine that the calibration is completed, the corresponding first calibration result is correct calibration, and in this case, there is no need to execute the subsequent process, and it is directly determined that the calibration is completed and the calibration process ends. If it is determined that each calibration corner point cannot be recognized or the recognized corner points are misaligned, the corresponding first calibration result is calibration error, and the subsequent calibration process of the deep learning model needs to be continued.

[0096] Step 102, in response to the first calibration result being calibration error, retrieve the deep learning model, and use the deep learning model to complete correct panoramic surround view calibration for the panoramic surround view information, where the deep learning model is a neural network model that has been pre-trained to be able to perform panoramic surround view calibration.

[0097] During specific implementation, the initial neural network is pre-trained using a predetermined number (for example, 1000) of panoramic surround view information in various scenarios to obtain the deep learning model. In this way, the deep learning model can accurately calibrate the panoramic surround view information collected in various scenarios.

[0098] After the deep learning model training is completed, the software development kit (AVM SDK) corresponding to the deep learning model will be sent to the vehicle terminal through the OTA platform via the network. After the vehicle terminal receives this software development kit, it will be downloaded and installed and associated with the traditional corner point calibration method. Furthermore, when the first calibration result obtained by the corner point calibration method shows calibration error, the deep learning model will be started to execute, receive the panoramic surround view information corresponding to the calibration error, and re-perform panoramic surround view calibration on the panoramic surround view information with calibration error, and finally obtain an accurate panoramic surround view calibration result.

[0099] Through the above scheme, since the traditional corner point calibration method is relatively simple and fast, after the vehicle receives the panoramic view calibration instruction, it will first use the traditional corner point calibration method to perform the traditional panoramic view calibration on the acquired panoramic view information, and obtain the first calibration result. If the first calibration result is wrong, it proves that the environmental scene calibrated by the traditional corner point calibration method is single and cannot adapt to the current environmental scene of the vehicle. The current environmental scene will be disturbed by light or surrounding background images, etc. The traditional corner point calibration method cannot adapt to various environmental scenes disturbed by light and surrounding background images. At this time, it is necessary to use a deep learning model with a wider range of applications to correctly calibrate the panoramic view information. Since the deep learning model is a neural network model obtained by pre-training the panoramic view information of various environmental scenes, the deep learning model is used to perform a panoramic view calibration on the corresponding panoramic view information when the first calibration result is wrong, and then the panoramic view calibration process is correctly completed. In this way, the deep learning model can be used to make up for the problem that the traditional corner point calibration method is only applicable to a single scenario. The traditional corner point calibration and the deep learning model work together to meet the panoramic view calibration needs of various environmental scenarios. It has a wide range of applications, high accuracy, and improves the robustness of the panoramic view calibration.

[0100] In some embodiments, based on the specific content of step 101 above, the step 102 of using a deep learning model to correctly calibrate the panoramic view information includes:

[0101] Step 1021: calibrate the panoramic view information using a deep learning model to obtain a second calibration result.

[0102] In specific implementation, since the deep learning model has the function of accurately identifying and locating image targets, it can perform panoramic surround view calibration on panoramic surround view information in various scenarios after training.

[0103] However, since the panoramic view information used to train the deep learning model is limited, the deep learning model may also have defects and inaccurate panoramic view calibration may occur.

[0104] If the deep learning model can accurately identify each corner block in the panoramic view information, and can accurately identify and locate each corner block, the second calibration result obtained in this way is a correct calibration; if the deep learning model cannot accurately identify each corner block in the panoramic view information, and / or cannot accurately identify and locate each corner block, the second calibration result obtained in this way will be an incorrect calibration.

[0105] Step 1022: In response to the second calibration result being correct, determine that the panoramic view calibration process is completed.

[0106] During specific implementation, the corresponding second calibration result will be displayed at the vehicle end, and then it is manually determined whether it is correct based on the displayed second calibration result. If the obtained second calibration result is correct calibration, it proves that the deep learning model can accurately perform corner recognition and calibration on the panoramic surround view information collected in the current scene at the vehicle end, and then accurately complete the panoramic surround view calibration process, and then obtain an accurate panoramic surround view calibration result, and thus the calibration process ends.

[0107] Or, in step 1023, in response to the second calibration result being calibration error, the panoramic surround view information with calibration error is sent to the service platform for the server platform to train the deep learning model based on the panoramic surround view information with calibration error to obtain a trained deep learning model.

[0108] During specific implementation, after the panoramic surround view calibration performed by the deep learning model, if the obtained second calibration result is calibration error, it proves that there are defects in the process of determining the corner blocks and / or positioning the corner points of the corner blocks of the panoramic surround view information in the current scene at the vehicle end, and the panoramic surround view positioning function of the deep learning model needs to be further improved.

[0109] At this time, the panoramic surround view information with calibration error collected at the vehicle end will be sent to the service platform, so that the service platform can perform correct calibration based on the panoramic surround view information with calibration error and then continue to train the deep learning model. In this way, the trained deep learning model obtained after training can accurately perform panoramic surround view positioning on the panoramic surround view information in the current scene at the vehicle end.

[0110] Step 1024, receive the trained deep learning model sent by the service platform, and use the trained deep learning model to process the panoramic surround view data to complete the panoramic surround view calibration process.

[0111] During specific implementation, after the service platform completes the continuous training of the deep learning model, it will process it into a corresponding software development kit (SDK) according to the trained deep learning model. Then, through the OTA platform, the software development kit (SDK) is sent to the vehicle end. In this way, the vehicle end receives the software development kit (SDK) corresponding to the trained deep learning model, installs it, re-collects the panoramic surround view information, and uses the trained deep learning model to perform panoramic surround view calibration processing on the re-collected panoramic surround view information. If the calibration result obtained from this panoramic surround view calibration is correct, the corresponding calibration process ends; if the calibration result obtained from this panoramic surround view calibration is incorrect, the processes of steps 1023 and 1024 above will be repeated until the calibration result obtained from the panoramic surround view calibration is correct.

[0112] Through the above solution, after the vehicle terminal uses the deep learning model to perform panoramic calibration processing on the panoramic surround view data, if a calibration error occurs, it proves that the processing accuracy of the deep learning model is relatively low. Therefore, in order to improve the accuracy of the deep learning model, the panoramic surround view data with calibration errors will be sent to the service platform; then, the service platform will continue to train the deep learning model, and then obtain the trained deep learning model. After sending it to the vehicle terminal, the vehicle terminal can use the trained deep learning model to perform accurate panoramic surround view calibration. Since the trained deep learning model is obtained by the service platform continuing to train on the basis of the original deep learning model and continuously correcting the defects of the original deep learning model. The trained deep learning model obtained after defect correction can perform correct panoramic surround view calibration based on the current environment of the vehicle terminal. In this way, on the basis of the original scenarios where the trained deep learning model can perform panoramic surround view calibration, the current environment scenario of the vehicle terminal is added, so that the scenarios where the trained deep learning model performs correct panoramic surround view calibration are more.

[0113] Specifically, in step 1023, the process of sending the panoramic surround view information with calibration errors to the service platform is as follows:

[0114] There is a panoramic surround view calibration system, namely the AVM system, in the vehicle terminal. As Figure 4 shown, use the AVM system to execute:

[0115] S1, determine whether the panoramic surround view information with calibration errors is valid. After determining its validity, create a directory and copy file corresponding to the panoramic surround view information with calibration errors in the QNX layer of the AVM system.

[0116] Among them, the user of the vehicle terminal will judge the validity of the panoramic surround view information with calibration errors to determine whether it is valid. This is to ensure the accuracy of the data.

[0117] S2, the operating system of the vehicle terminal (for example, the Android system) compresses the panoramic surround view information with calibration errors, improves the upload configuration information, and adds it to the upload queue of the transmission platform (for example, Tencent Cloud).

[0118] Among them, since the panoramic surround view information with calibration errors is relatively scattered and some data is too large to be directly uploaded, the operating system of the vehicle terminal will compress the panoramic surround view information with calibration errors to form a smaller compressed package. Some configuration information of the vehicle terminal (specifically including the vehicle identification number, operating system version or vehicle terminal identity configuration, etc.) of the vehicle terminal is added to the compressed package. In this way, the compressed package carries the configuration information to form a compressed file, and then it is added to the upload queue of the transmission platform and waits to be uploaded.

[0119] S3, The transmission platform establishes a file link with the operating system of the vehicle terminal.

[0120] Among them, after the transmission platform receives the information of joining the upload queue sent by the operating system of the vehicle terminal, the transmission platform will establish a file connection with the compressed file carrying the configuration information in the operating system of the vehicle terminal. Since the transmission platform is connected to the service platform, the transmission platform can be used as a transmission medium to connect the operating system of the vehicle terminal and the service platform.

[0121] S4, The operating system of the vehicle terminal can directly send the file configuration information to the service platform and transmit the compressed file to the service platform communicating with the transmission platform.

[0122] Among them, the file configuration information includes: the size, name, file type, etc. of the compressed package carrying the configuration information. Based on the transmission platform as a transmission medium, the operating system of the vehicle terminal directly sends the file configuration information to the service platform, and the service platform prepares the corresponding storage space according to the file configuration information. Then the operating system of the vehicle terminal transmits the compressed package carrying the configuration information to the service platform through the transmission platform and stores the compressed file in the corresponding storage space prepared by the service platform.

[0123] S5, After the compressed file is successfully uploaded, the service platform feeds back the upload success information to the operating system of the vehicle terminal.

[0124] Among them, after all the compressed files are uploaded, the service platform determines that the upload is successful, generates the upload success information corresponding to the compressed file, and sends the upload success information to the operating system of the vehicle terminal, so that the operating system of the vehicle terminal can know the situation that the compressed file has been uploaded.

[0125] Based on the same inventive concept, the panoramic surround calibration method proposed in the embodiment of the present application is applied to a service platform that can communicate with the vehicle terminal directly or indirectly.

[0126] As Figure 5 shown, the method includes:

[0127] Step 201, Receive the panoramic surround data with calibration errors feedback from the vehicle terminal. Among them, when the vehicle terminal uses the deep learning model to perform panoramic surround calibration on the panoramic surround data, it will feedback the panoramic surround data after calibration errors occur.

[0128] In specific implementation, a traditional corner calibration program is configured on the vehicle terminal. Since the corner calibration method is relatively simple and fast, in order to improve the efficiency of panoramic surround calibration, the traditional corner calibration method is preferentially used for implementation. After calibrating the panoramic surround information by the traditional corner calibration method, a first calibration result is obtained.

[0129] If the first calibration result is correct, there is no need to execute the subsequent processes in this case. It is directly determined that the calibration is completed and the calibration process ends.

[0130] If the first calibration result is incorrect, the vehicle end will prove that the traditional corner calibration method cannot adapt to the environmental scenario where the vehicle end is currently located. It is necessary to use a deep learning model with a wider applicable range to continue calibrating the panoramic surround view information to obtain a second calibration result.

[0131] If the second calibration result is correct, it proves that the deep learning model can accurately perform corner recognition and calibration on the panoramic surround view information collected in the current scenario of the vehicle end. Furthermore, the panoramic surround view calibration process can be accurately completed, and an accurate panoramic surround view calibration result can be obtained. In this way, the calibration process ends.

[0132] If the obtained second calibration result is incorrect, it proves that the deep learning model cannot accurately perform corner recognition and calibration on the panoramic surround view information in the current scenario of the vehicle end. To improve the accuracy of the deep learning model, the vehicle end will collect the panoramic surround view data with calibration errors and send it to the service platform. In this way, when the service platform receives the panoramic surround view data with calibration errors, it will start the training process of the deep learning model.

[0133] Step 202: Preprocess the panoramic surround view data, and use the preprocessed panoramic surround view data to train the deep learning model to obtain a trained deep learning model.

[0134] In specific implementation, in order to enable the panoramic surround view data to be used as training samples for training, it is necessary to preprocess the panoramic surround view data. The preprocessing process at least includes the correct calibration process of the panoramic surround view data. In this way, the preprocessed panoramic surround view data has correctly calibrated content and can be used to train the deep learning model.

[0135] Since the deep learning model is a neural network model composed of multiple neural network layers, during the training process, the deep learning model will continuously adjust the parameters of each internal neural network layer, so that the panoramic surround view calibration result obtained after processing the preprocessed panoramic surround view data matches the correct calibration result, thereby completing the training process of the deep learning model. The trained deep learning model can accurately calibrate the panoramic surround view data of the current scenario of the vehicle end.

[0136] Step 203: Send the trained deep learning model to the vehicle end for the vehicle end to use the trained deep learning model to correctly calibrate the panoramic surround view data.

[0137] During specific implementation, after the service platform finishes training the deep learning model, it will send the software development kit (SDK) corresponding to the trained deep learning model to the vehicle terminal. Then the vehicle terminal installs the software development kit (SDK) and re-collects the panoramic surround view information for panoramic surround view calibration processing (specifically refer to the process of step 1024 in the above embodiment, which will not be elaborated here). The vehicle terminal will display the result of the panoramic surround view calibration processing of the trained deep learning model for the user to view. The user can make a judgment based on the displayed result to determine whether the current panoramic surround view calibration is correct and feedback to the service platform.

[0138] The service platform will wait for the result of the panoramic surround view calibration feedback from the vehicle terminal. If the calibration is correct, the service platform stops waiting; if the calibration is incorrect, the service platform will receive the panoramic surround view data with calibration error feedback from the vehicle terminal again and repeat the processes of steps 202 and 203 until the correct calibration is determined.

[0139] Through the above solution, when there is an error in the panoramic surround view calibration of the deep learning model on the vehicle terminal, the service platform can use the panoramic surround view data with calibration error feedback from the vehicle terminal. After preprocessing it, it can calibrate it with correct panoramic surround view data and use it as a training sample to retrain the original deep learning model, continuously correcting the defects of the deep learning model, so that the trained deep learning model obtained through the correction process has an increased ability to correctly calibrate the panoramic surround view data with calibration error feedback from the vehicle terminal.

[0140] In some embodiments, the service platform includes: a data middle platform and a training device, where the deep learning model is stored in the training device.

[0141] During specific implementation, there is an AI platform in the service platform for training the deep learning model. The AI platform includes: a data middle platform and a training device. The training device includes a physical server for storing and training the deep learning model. In addition, the AI platform also includes a cloud server for receiving the panoramic surround view data with calibration error.

[0142] Based on steps 201 and 203 described in the above embodiment, step 202 includes:

[0143] Step 2021, use the data middle platform to filter and clean the panoramic surround view data, and then calibrate the cleaned panoramic surround view data according to the correct calibration data to obtain the calibrated panoramic surround view data, and send the calibrated panoramic surround view data to the training device.

[0144] In specific implementation, after the cloud server receives the panoramic surround-view data with calibration errors fed back by the vehicle terminal and stores it, the data middle platform retrieves the panoramic surround-view data from the cloud server, filters and cleans the data to remove the noise data in the panoramic surround-view data, so as to prevent the noise data from affecting the training of the deep learning model. Then, the data middle platform displays the cleaned panoramic surround-view data on the screen for the corresponding personnel to correctly calibrate the cleaned panoramic surround-view data, obtaining the panoramic surround-view data with correct calibration data. After calibration, the data middle platform sends the calibrated panoramic surround-view data to the training device capable of executing the training process to train the deep learning model.

[0145] Step 2022: Use the training device to perform training processing on the deep learning model according to the calibrated panoramic surround-view data, and iteratively train and adjust the parameters in the deep learning model based on the difference between the training output result and the correct calibration data until the training output result matches the correct calibration data, obtaining the trained deep learning model.

[0146] In specific implementation, if the deep learning model is pre-stored in the training device, the training device directly performs training processing on the deep learning model according to the calibrated panoramic surround-view data. If there is no deep learning model, a retrieval instruction is sent to the vehicle terminal. After the vehicle terminal sends the deep learning model to the cloud server, the cloud server sends the deep learning model to the training device for storage, and then performs training processing on the deep learning model according to the calibrated panoramic surround-view data.

[0147] During the training process, the training device inputs the calibrated panoramic surround-view data into the deep learning model. After being processed by each network layer, a training output result can be obtained. Then, the training output result is compared with the correct calibration data to determine the degree of difference. According to the degree of difference, the parameter adjustment amount of each network layer is determined, and the parameters of each network layer in the deep learning model are adjusted according to the parameter adjustment amount. After the adjustment is completed, the adjusted deep learning model is used to process the calibrated panoramic surround-view data again to obtain a training output result, and then the training output result is compared with the correct calibration data to determine the degree of difference.

[0148] If the degree of difference is large (greater than the set threshold), it proves that further training is required. The parameters of each network layer in the deep learning model are re-adjusted according to the degree of difference, and this parameter adjustment process is continuously repeated until the determined degree of difference is less than or equal to the set threshold.

[0149] If the degree of difference is small (less than or equal to the set threshold), it proves that the training is completed, and the deep learning model at this time is used as the trained deep learning model.

[0150] The training device includes: a physical server, which uses the physical server to perform the training process of steps 2021 to 2022 above.

[0151] Through the above scheme, the filtering, cleaning and calibration of the panoramic view data can be completed through the cooperation of the data middle platform and the training equipment, so as to avoid noise infection in the panoramic view data and ensure the accuracy of the calibrated panoramic view data. According to the calibrated panoramic view data, the deep learning model is trained and processed, so that the trained deep learning model has the ability to perform panoramic view calibrate on the panoramic view data of the current scene, effectively improving the accuracy.

[0152] In a preferred embodiment, each network layer of the deep learning model includes: an input layer, multiple hidden layers and an output layer.

[0153] The training process of a deep learning model on calibrated panoramic view data includes:

[0154] (1) The input layer is used to process the calibrated panoramic view data and distribute it to each hidden layer for analysis and processing.

[0155] (2) Multiple hidden layers, including: a shape analysis hidden layer, a color analysis hidden layer, and a brightness analysis hidden layer. The shape analysis hidden layer is used to analyze the shape of each figure in the calibrated panoramic view data to determine the corresponding calibrated corner block position; the color analysis hidden layer is used to analyze the color of each figure in the calibrated panoramic view data to determine the color distribution area that best matches the corresponding calibrated corner block, and use it as the corresponding calibrated corner block position; the brightness analysis hidden layer is used to analyze the brightness of each figure in the calibrated panoramic view data to determine the brightness area that best matches the corresponding calibrated corner block, and use it as the corresponding calibrated corner block position; the shape analysis hidden layer, the color analysis hidden layer, and the brightness analysis hidden layer each send the obtained calibrated corner block position to the output layer.

[0156] (3) An output layer is used to compare and superimpose the calibrated corner block positions sent from the hidden layer, determine the corner block position with the most overlap after superposition, and output the corner block position with the most overlap as the training output result of the panoramic view calibration position.

[0157] (4) Compare the training output result of the output layer with the correct calibration data in the calibrated panoramic view data, determine the corresponding loss function, calculate the loss value, and adjust the parameters of the input layer, multiple hidden layers and the output layer according to the loss value.

[0158] After determining that all calibrated panoramic view data have completed training, the deep learning model with final parameter adjustment is used as the trained deep learning model.

[0159] In some embodiments, the deep learning model is an object detection network model.

[0160] As Figure 6 shown, the object detection network model is the YOLO model, and this YOLO model includes 24 convolutional layers, 4 pooling layers and 2 fully connected layers.

[0161] Among them, the convolutional layer is used to perform convolutional processing on the graphic data in the panoramic surround view data.

[0162] The pooling layer is arranged after one or more convolutional layers and is used to extract features from the results processed by the convolutional layer, including a max pooling layer and / or an average pooling layer.

[0163] The fully connected layer is used to integrate the previously extracted features into the required results.

[0164] Based on the processes of the above-mentioned step 201, step 2021 and step 203, the step 2022 executed by the training device specifically includes:

[0165] Step 20221: Adjust the size of the calibrated panoramic surround view data, input the panoramic surround view data after size adjustment into the object detection network model for panoramic surround view calibration processing, and obtain the model processing result.

[0166] In specific implementation, since the calibrated panoramic surround view data cannot be directly input into the object detection network model due to inconsistent sizes, it is necessary to adjust the size of the calibrated panoramic surround view data and then input it into the object detection network model. After the convolutional processing of the convolutional layer, the feature extraction processing of the pooling layer, and the feature integration processing of the fully connected layer, the model processing result is obtained.

[0167] Step 20222: Use the non-maximum suppression algorithm (NMS) to remove the redundant detection frames in the model processing result, and obtain the calibrated angle result after training as the training output result.

[0168] In specific implementation, since there are many redundancies in the model processing result, it is necessary to use the non-maximum suppression algorithm (NMS) to remove these redundant detection frames so as to obtain the training output result with accurate calibrated angle results.

[0169] Step 20223: Compare the training output result with the correct calibration data to determine the degree of difference.

[0170] Step 20224: Determine the adjustment parameters for each network layer of the target detection network model according to the degree of difference, and adjust and correct each network layer of the target detection network model according to the adjustment parameters to obtain a trained deep learning model.

[0171] In specific implementation, since the correctly calibrated data is carried in the marked panoramic surround view data, the correctly calibrated data will be extracted and compared with the training output result, and the degree of difference will be determined according to the comparison result. If the degree of difference is large (greater than the set threshold), it proves that further training is still needed. Then, the parameters of each network layer in the deep learning model will be adjusted again according to the degree of difference, and this process of parameter adjustment will be continuously repeated until the determined degree of difference is less than or equal to the set threshold. If the degree of difference is small (less than or equal to the set threshold), it proves that the training is over, and the deep learning model at this time will be used as the trained deep learning model.

[0172] Through the above solution, on the basis of the original environment scene where the target detection network model (YOLO model) can perform panoramic surround view calibration, the process of retraining the calibrated panoramic surround view data is added. During the training process, the non-maximum suppression algorithm is used to remove redundancy from the training output result of the target detection network model, improving the accuracy of the obtained training output result. Since the training output result is more accurate, the determined degree of difference between it and the correctly calibrated data is more accurate. Furthermore, according to the degree of difference, the parameters of each network layer of the target detection network model can be adjusted better, making the training process smoother and the accuracy of the trained deep learning model higher.

[0173] In some embodiments, based on the processes of steps 201 and 202 above, step 203 includes:

[0174] Step 2031: Generate a panoramic surround view calibration development tool according to the trained deep learning model.

[0175] In specific implementation, the training device can process the trained deep learning model into a corresponding panoramic surround view calibration development tool (AVM SDK), so that this panoramic surround view calibration development tool has the function of installation and execution.

[0176] Step 2032: Perform panoramic surround view calibration verification processing according to the panoramic surround view calibration development tool to obtain a calibration verification result.

[0177] In specific implementation, the training device needs to perform panoramic surround view calibration verification processing on the panoramic surround view calibration development tool and display the calibration verification result for personnel to view and analyze whether it is correct.

[0178] Step 2033, in response to the calibration verification result being wrong, the verification panoramic view data used for calibration verification is fed back to the data center, and the data center preprocessing and the training equipment training processing process are repeated until the calibration verification result is correct.

[0179] In specific implementation, if the personnel determine that the calibration verification result is wrong after analysis, they will extract the calibration verification verification panoramic view data and then feed it back to the data center. The data center will then filter and clean it, and after correct calibration, use the training equipment to continue training the trained deep learning model (i.e. repeat the above steps 2021 and 2022) until the calibration verification result is correct.

[0180] Step 2034, in response to the calibration verification result being correct, the panoramic view calibration development tool is sent to the vehicle side, so that the vehicle side can execute the function of the panoramic view calibration development tool to perform correct panoramic view calibration using the trained deep learning model.

[0181] In specific implementation, if the personnel determine that the calibration verification result is correct after analysis, proving that the training of the deep learning network has been completed, the corresponding panoramic view calibration development tool can be directly sent to the vehicle through the OTA platform. In this way, the vehicle will install it, re-collect the panoramic view information, and use the installed panoramic view calibration development tool to perform panoramic view calibration processing on the re-collected panoramic view information.

[0182] The training device in the service platform further includes: a computer terminal (PC terminal). The calibration and verification process of the above steps 2031 to 2034 is performed using the computer terminal (PC terminal).

[0183] Through the above scheme, the trained deep learning model will be generated into a corresponding panoramic view calibration development tool, which is convenient for using the panoramic view calibration development tool to perform the calibration verification process. In this way, the accuracy of the trained deep learning model can be determined in combination with the calibration verification result. After determining that the calibration verification result is correct, the panoramic view calibration development tool is sent to the vehicle end, thus ensuring the accuracy of the panoramic view calibration development tool sent to the vehicle end for panoramic view calibration.

[0184] The specific calibration and verification process can be divided into at least two situations:

[0185] The first one is to build a virtual cockpit environment and simulate the calibration process to perform calibration verification.

[0186] In some embodiments, the service platform further includes: a cockpit simulation platform, wherein the cockpit simulation platform includes: a PC terminal, a cockpit test bench, a dynamic cockpit simulator and a calibration device.

[0187] The PC side is used to process the trained deep learning model into the corresponding panoramic surround calibration development tool (AVMSDK).

[0188] The cockpit test bench is used to carry dynamic cockpit simulators of different vehicle models.

[0189] The dynamic cockpit simulator can be used to simulate the styling dimensions and camera installation positions of different vehicle models. The dynamic cockpit simulator is exactly the same as the overall vehicle dimensions and camera installation positions of various vehicle models, and can also simulate the calibration environment under different scenarios.

[0190] The calibration device is used to adjust the calibration site of calibration plates at different positions and verify the trained deep learning model. It can also directly perform local simulation verification through the PC side. After the verification is correct, the panoramic surround calibration development tool (AVM SDK) is integrated.

[0191] Based on the processes of Step 201, Step 202, Step 2031, Step 2033, and Step 2034 above, Step 2032 includes:

[0192] Step A1, using the cockpit simulation platform to simulate the vehicle and the camera installation positions on the vehicle, and cooperating with the training device to determine the virtual calibration scenario.

[0193] Specifically in implementation, the cockpit test bench in the cockpit simulation platform will be used to carry the dynamic cockpit simulator, the dynamic cockpit simulator will be used to simulate the vehicle and the camera installation positions on the vehicle, and the dynamic cockpit simulator will cooperate with the computer side in the training device to form a virtual calibration scenario.

[0194] Step A2, in the virtual calibration scenario, the training device uses the panoramic surround calibration development tool to control the cockpit simulation platform to perform virtual calibration adjustment of the panoramic surround, and display the results obtained from the virtual calibration adjustment for the user to determine based on the displayed panoramic surround data after calibration verification.

[0195] Specifically in implementation, the cockpit test bench in the cockpit simulation platform will be used to carry the dynamic cockpit simulator with the same vehicle model as that of the vehicle end. The dynamic cockpit simulator will be used to simulate the verified panoramic surround data collected from the scene where the vehicle end is located. Then, the computer side in the training device will execute the panoramic surround calibration development tool to control the calibration plate on the calibration device to simulate the verification process of panoramic surround calibration for the collected verified panoramic surround data, and thus obtain the calibration verification result.

[0196] Through the above solution, the simulation calibration verification process of the panoramic surround view calibration development tool can be completed through the mutual cooperation of the training device (e.g., computer terminal) and the cockpit simulation platform (cockpit bench, dynamic cockpit simulator, and calibration device), and then an accurate calibration verification result can be obtained. Only in this way can it be determined whether to continue training the deep learning model based on the accuracy of the calibration verification result, reducing the situation where the panoramic surround view calibration development tool is directly sent to the vehicle terminal and calibration errors occur again at the vehicle terminal.

[0197] The second method: directly perform calibration verification, and after calibration verification, incrementally expand the verification panoramic surround view data used for calibration verification errors and then send it to the data middle platform, and use the data middle platform and the training device (e.g., physical server) to continue the training process of the deep learning model.

[0198] In some embodiments, based on the processes of the above steps 201, 202, 2031, 2033, and 2034, step 2032 includes:

[0199] Step B1, using the training device to perform calibration verification on the panoramic surround view data according to the panoramic surround view calibration development tool, and displaying the calibrated panoramic surround view data obtained, so that the user can make a determination based on the displayed calibrated panoramic surround view data.

[0200] Specifically, during implementation, the training device will locally install the panoramic surround view calibration development tool, and after installation, an execution program corresponding to the trained deep learning model will be formed. Then, the obtained panoramic surround view data will be input into this execution program, and the calibration verification process of the panoramic surround view calibration will be carried out using the trained deep learning model to obtain the calibrated panoramic surround view data.

[0201] For easy viewing, the calibrated panoramic surround view data will be displayed. After seeing it, the user will make a manual judgment based on the displayed calibrated panoramic surround view data, and then obtain the corresponding determination result.

[0202] Step B2, using the training device to receive the determination result for the calibrated panoramic surround view data, and taking the determination result as the calibration verification result.

[0203] Specifically, during implementation, the training device can receive the determination result input by the user for the panoramic surround view data calibrated by the training device shown, and this determination result includes: correct calibration verification or incorrect calibration verification.

[0204] Through the above solution, the process of calibrating and validating the panoramic surround calibration development tool is executed in the training device, which is simple and convenient to operate. And only after the calibration verification process can it be determined whether to continue training the deep learning model based on the accuracy of the calibration verification result, reducing the situation where the panoramic surround calibration development tool is directly sent to the vehicle terminal and calibration errors occur again at the vehicle terminal.

[0205] In some embodiments, based on the processes of step 201, step 202, step 2031, step 2032, and step 2034 above, in step 2033, the training device is used to execute: in response to the calibration verification result being incorrect, feedback the verification panoramic surround data used for calibration verification to the data middle platform, including:

[0206] Step 20331, in response to the calibration verification result being incorrect, use the training device to expand the verification panoramic surround data used for calibration verification into multiple different verification panoramic surround data in an incremental expansion manner.

[0207] Step 20332, the training device feeds back multiple different verification panoramic surround data to the data middle platform for the data middle platform and the training device to perform training processing based on the multiple different verification panoramic surround data.

[0208] Specifically, when implementing, if the calibration verification result is incorrect in the calibration verification method of step B1 and step B2 of the second type above, it proves that the training of the deep learning model still needs to be continued. However, since the cockpit simulation is not performed in this case, more verification panoramic surround data cannot be obtained. Therefore, the verification panoramic surround data originally used for calibration verification will be incrementally expanded to obtain multiple different verification panoramic surround data. The specific incremental expansion methods include at least one of changing the color, changing the color difference, changing the brightness, deleting edge features, and adjusting the size.

[0209] Then, the data middle platform and the training device are used to continue the training process for these different verification panoramic surround data according to the processes of step 2021 and step 2022 above until the obtained calibration verification result is correct.

[0210] Through the above solution, after the calibration verification result is incorrect, by incrementally expanding the verification panoramic surround data and then using the data middle platform and the training device to execute the training process again, it is possible to have multiple data training references, improve the training accuracy, and make the accuracy of the trained deep learning model higher.

[0211] Based on the same inventive concept, the following describes the process of the vehicle terminal and the service platform executing the panoramic surround calibration method in a specific implementation scenario.

[0212] Solution 1:

[0213] As Figure 7 shown, after the vehicle end performs production line calibration or after-sales calibration using the traditional corner point calibration method, that is, the vehicle end transmits the panoramic surround calibration data corresponding to the calibration scenario to the vehicle's AVM, and then executes the traditional corner point calibration method to perform panoramic surround calibration to obtain the first calibration result.

[0214] If the first calibration result is correct, the vehicle end calibration process ends.

[0215] If the first calibration result fails or the calibration is poor (that is, the first calibration result is a calibration error), then continue to execute the panoramic surround calibration process using the deep learning model to obtain the second calibration result.

[0216] If the second calibration result is correct, the calibration ends.

[0217] If the second calibration result is incorrect, resulting in calibration failure or poor calibration effect, upload the panoramic surround data used for calibration to the cloud server, perform data cleaning and calibration (calibrate the correct calibration data) on the panoramic surround data through the data middle platform in the AI platform, and then use the calibrated panoramic surround data on the physical server to train the deep learning model to optimize the deep learning model. After optimization, obtain the trained deep learning model and send it to the PC side.

[0218] Generate the corresponding panoramic surround calibration development tool (AVM SDK) on the PC side according to the trained deep learning model.

[0219] By mounting and setting a dynamic cockpit simulator with the same vehicle model as the vehicle end on the cockpit bench, use the dynamic cockpit simulator to simulate the verification panoramic surround data collected from the scene where the vehicle end is located.

[0220] The computer side will execute the panoramic surround calibration development tool, control the calibration board on the calibration device to simulate the verification process of panoramic surround calibration for the collected verification panoramic surround data, and then obtain the calibration verification result.

[0221] If the calibration verification result is incorrect, the computer side will feedback the verification panoramic surround data used for calibration verification to the data middle platform, and repeat the data middle platform preprocessing and physical server training processing process until the obtained calibration verification result is correct.

[0222] If the calibration verification result is correct, the PC side will transmit the panoramic surround calibration development tool (AVM SDK) to the vehicle end through the OTA platform, and let the vehicle end re-execute the panoramic surround calibration process based on the panoramic surround calibration development tool (AVM SDK).

[0223] If the panoramic surround calibration performed again by the vehicle end based on the panoramic surround calibration development tool (AVM SDK) is correct, the calibration is completed.

[0224] If the panoramic surround calibration performed again by the vehicle end based on the panoramic surround calibration development tool (AVM SDK) is incorrect, repeat the process of training the above deep learning model until the finally obtained panoramic surround calibration development tool (AVM SDK) can correctly perform panoramic surround calibration and obtain a correct panoramic surround calibration result.

[0225] Solution 2:

[0226] As Figure 8 shown, after the vehicle end performs production line calibration or after-sales calibration using the traditional corner calibration method, that is, the vehicle end transmits the panoramic surround calibration data corresponding to the calibration scene to the AVM of the vehicle, and then performs panoramic surround calibration using the traditional corner calibration method to obtain the first calibration result.

[0227] If the first calibration result is correct, the vehicle end calibration process ends.

[0228] If the first calibration result fails or the calibration is not good (that is, the first calibration result is a calibration error), continue to perform the panoramic surround calibration process using the deep learning model to obtain the second calibration result.

[0229] If the second calibration result is correct, the calibration ends.

[0230] If the second calibration result is incorrect, showing a calibration failure or a poor calibration effect, upload the panoramic surround data used for calibration to the cloud server, perform data cleaning and calibration (calibrate the correct calibration data) on the panoramic surround data through the data middle platform in the AI platform, and then use the calibrated panoramic surround data on the physical server to train the deep learning model and optimize the deep learning model. After optimization, obtain the trained deep learning model and send it to the PC side.

[0231] Generate the corresponding panoramic surround calibration development tool (AVM SDK) on the PC side according to the trained deep learning model.

[0232] On the PC side, use the panoramic surround data fed back by the vehicle end as the verification panoramic surround data for calibration verification, perform calibration verification processing using this panoramic surround calibration development tool to obtain the calibrated and verified panoramic surround data, and display it for the corresponding personnel to view. Then the personnel make a judgment based on the displayed calibrated and verified panoramic surround data to determine whether the calibration verification is correct, and then obtain the calibration verification result according to the judgment result.

[0233] If the calibration verification result is wrong, the computer side will incrementally expand the verification panoramic view data used for calibration verification (at least one of changing color, changing color difference, changing brightness, deleting edge features and adjusting size) to obtain multiple different verification panoramic view data and feed them back to the data middle station, repeating the data middle station preprocessing and physical server training processing process until the calibration verification result is correct.

[0234] If the calibration verification result is correct, the PC will transmit the panoramic view calibration development tool (AVM SDK) to the vehicle through the OTA platform, allowing the vehicle to re-execute the panoramic view calibration process based on the panoramic view calibration development tool (AVM SDK).

[0235] If the surround view calibration performed again by the vehicle based on the surround view calibration development tool (AVM SDK) is correct, the calibration is completed.

[0236] If the panoramic view calibration performed again by the vehicle-side based on the panoramic view calibration development tool (AVM SDK) is wrong, the above-mentioned deep learning model training process is repeated until the final panoramic view calibration development tool (AVMSDK) can correctly perform the panoramic view calibration and obtain the correct panoramic view calibration result.

[0237] It should be noted that the method of the embodiment of the present application can be performed by a single device, such as a computer or server. The method of this embodiment can also be applied to a distributed scenario and completed by multiple devices cooperating with each other. In the case of such a distributed scenario, one of the multiple devices can only perform one or more steps in the method of the embodiment of the present application, and the multiple devices will interact with each other to complete the described method.

[0238] It should be noted that the above describes some embodiments of the present application. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the above embodiments and still achieve the desired results. In addition, the processes depicted in the accompanying drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0239] Based on the same inventive concept, corresponding to the panoramic surround view calibration method of the above-mentioned embodiment applied to the vehicle side, the present application also provides a panoramic surround view calibration device, which is arranged on the vehicle side.

[0240] refer to Figure 9 , the device comprises:

[0241] The corner point calibration module 301 is configured to receive a panoramic view calibration instruction, obtain panoramic view information, and perform panoramic view calibration on the panoramic view information by using a corner point calibration method to obtain a first calibration result;

[0242] The deep learning calibration module 302 is configured to call up a deep learning model in response to the first calibration result being a calibration error, and use the deep learning model to complete a correct panoramic view calibration of the panoramic view information, wherein the deep learning model is a pre-trained neural network model that can perform panoramic view calibration.

[0243] In some embodiments, the deep learning calibration module 302 is specifically configured to:

[0244] Performing panoramic surround view calibration on the panoramic surround view information using a deep learning model to obtain a second calibration result;

[0245] In response to the second calibration result being correct, determining that the panoramic view calibration process is completed; or,

[0246] In response to the second calibration result being a calibration error, sending the panoramic surround view information with the calibration error to the service platform, so that the server platform trains the deep learning model according to the panoramic surround view information with the calibration error to obtain a trained deep learning model;

[0247] Receive the trained deep learning model sent by the service platform, use the trained deep learning model to process the panoramic view data, and complete the panoramic view calibration process.

[0248] Based on the same inventive concept, corresponding to the panoramic surround view calibration method of the above-mentioned embodiment applied to the service platform, the present application also provides a panoramic surround view calibration device, and the panoramic surround view calibration device is arranged on the service platform.

[0249] like Figure 10 As shown, the device comprises:

[0250] The data receiving module 401 is configured to receive the panoramic view data with calibration errors fed back by the vehicle end, wherein when the vehicle end uses the deep learning model to perform panoramic view calibration on the panoramic view data, the panoramic view data will be fed back after the calibration error occurs;

[0251] The training module 402 is configured to pre-process the panoramic view data and train the deep learning model using the pre-processed panoramic view data to obtain a trained deep learning model;

[0252] The model sending module 403 is configured to send the trained deep learning model to the vehicle side for the vehicle side to correctly calibrate the panoramic surround view data using the trained deep learning model.

[0253] In some embodiments, the service platform includes: a data middle platform and a training device, wherein the deep learning model is stored in the training device;

[0254] The data middle platform is configured to filter and clean the panoramic surround view data, then calibrate the cleaned panoramic surround view data according to the correct calibration data to obtain the calibrated panoramic surround view data, and send the calibrated panoramic surround view data to the training device;

[0255] The training device is configured to train the deep learning model according to the calibrated panoramic surround view data, and iteratively train and adjust the parameters in the deep learning model according to the difference between the training output result and the correct calibration data until the training output result matches the correct calibration data, so as to obtain the trained deep learning model.

[0256] In some embodiments, the deep learning model is an object detection network model;

[0257] The training device is specifically configured to:

[0258] Adjust the size of the calibrated panoramic surround view data, input the panoramic surround view data with adjusted size into the object detection network model for panoramic surround view calibration processing to obtain a model processing result;

[0259] Use the non-maximum suppression algorithm to remove redundant detection boxes in the model processing result to obtain the trained angle calibration result as the training output result;

[0260] Compare the training output result with the correct calibration data to determine the degree of difference;

[0261] Determine the adjustment parameters of each network layer of the object detection network model according to the degree of difference, and adjust and correct each network layer of the object detection network model according to the adjustment parameters to obtain the trained deep learning model.

[0262] In some embodiments, the training device is configured to:

[0263] Generate a panoramic surround view calibration development tool according to the trained deep learning model;

[0264] Perform panoramic surround view calibration verification processing according to the panoramic surround view calibration development tool to obtain a calibration verification result;

[0265] In response to the calibration verification result being incorrect, the verification panoramic surround view data used for calibration verification is fed back to the data center, and the data center preprocessing and the training device training process are repeated until the obtained calibration verification result is correct;

[0266] In response to the calibration verification result being correct, the panoramic surround view calibration development tool is sent to the vehicle terminal for the vehicle terminal to execute the function of the panoramic surround view calibration development tool to perform correct panoramic surround view calibration using the trained deep learning model.

[0267] In some embodiments, the service platform further includes: a cockpit simulation platform;

[0268] The cockpit simulation platform is configured to simulate the vehicle and the installation positions of cameras on the vehicle, and cooperate with the training device to determine a virtual calibration scenario;

[0269] The training device is configured to, in the virtual calibration scenario, use the panoramic surround view calibration development tool to control the cockpit simulation platform to perform virtual calibration adjustment of the panoramic surround view, and display the result obtained from the virtual calibration adjustment for the user to make a determination based on the displayed panoramic surround view data after calibration verification;

[0270] The training device is configured to receive the determination result for the panoramic surround view data after calibration verification and use the determination result as the calibration verification result.

[0271] In some embodiments, the training device is specifically configured to:

[0272] Perform calibration verification on the panoramic surround view data according to the panoramic surround view calibration development tool, and display the panoramic surround view data after calibration verification obtained, for the user to make a determination based on the displayed panoramic surround view data after calibration verification;

[0273] Receive the determination result for the panoramic surround view data after calibration verification and use the determination result as the calibration verification result.

[0274] In some embodiments, the training device is specifically configured to:

[0275] In response to the calibration verification result being incorrect, the verification panoramic surround view data used for calibration verification is expanded into multiple different verification panoramic surround view data in an incremental expansion manner;

[0276] Feed back the multiple different verification panoramic surround view data to the data center for the data center to perform iterative training processing in combination with the training device based on the multiple different verification panoramic surround view data.

[0277] For the convenience of description, when describing the above device, it is divided into various modules according to functions and described separately. Of course, when implementing this application, the functions of each module can be implemented in the same or multiple software and / or hardware.

[0278] The device in the above embodiment is used to implement the corresponding method in any of the foregoing embodiments, and has the beneficial effects of the corresponding method embodiments, which will not be elaborated here.

[0279] Based on the same inventive concept, corresponding to the method in any of the above embodiments, the present application further provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the method described in any of the above embodiments.

[0280] Figure 11 FIG. shows a more specific schematic diagram of the hardware structure of the electronic device provided in this embodiment. The device may include: a processor 1010, a memory 1020, an input / output interface 1030, a communication interface 1040, and a bus 1050. Among them, the processor 1010, the memory 1020, the input / output interface 1030, and the communication interface 1040 are communicatively connected to each other inside the device through the bus 1050.

[0281] The processor 1010 may be implemented in a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, etc., and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this specification.

[0282] The memory 1020 may be implemented in the form of a ROM (Read Only Memory), a RAM (Random Access Memory), a static storage device, a dynamic storage device, etc. The memory 1020 may store an operating system and other application programs. When implementing the technical solutions provided in the embodiments of this specification through software or firmware, the relevant program codes are stored in the memory 1020 and called and executed by the processor 1010.

[0283] The input / output interface 1030 is used to connect to an input / output module to implement information input and output. The input / output module may be configured as a component in the device (not shown in the figure) or externally connected to the device to provide corresponding functions. Among them, the input device may include a keyboard, a mouse, a touch screen, a microphone, various sensors, etc., and the output device may include a display, a speaker, a vibrator, an indicator light, etc.

[0284] The communication interface 1040 is used to connect to a communication module (not shown in the figure) to enable communication and interaction between this device and other devices. The communication module can achieve communication through wired means (such as USB, network cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.).

[0285] The bus 1050 includes a path for transmitting information between various components of the device (such as the processor 1010, the memory 1020, the input / output interface 1030, and the communication interface 1040).

[0286] It should be noted that although the above device only shows the processor 1010, the memory 1020, the input / output interface 1030, the communication interface 1040, and the bus 1050, in the specific implementation process, the device may also include other components necessary for normal operation. In addition, those skilled in the art can understand that the above device may also only include the components necessary to implement the solution of the embodiments of this specification, and does not necessarily include all the components shown in the figure.

[0287] The electronic device of the above embodiment is used to implement the corresponding method in any of the foregoing embodiments, and has the beneficial effects of the corresponding method embodiments, which will not be elaborated here.

[0288] Based on the same inventive concept, corresponding to the method of any of the above embodiments, the present application also provides a non-transitory computer-readable storage medium. The non-transitory computer-readable storage medium stores computer instructions, and the computer instructions are used to cause the computer to execute the method described in any of the above embodiments.

[0289] The computer-readable medium of this embodiment includes permanent and non-permanent, removable and non-removable media, and information storage can be implemented by any method or technology. The information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassette tapes, magnetic disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information accessible by a computing device.

[0290] The computer instructions stored in the storage medium of the above embodiments are used to cause the computer to execute the method described in any of the above embodiments, and have the beneficial effects of the corresponding method embodiments, which will not be elaborated here.

[0291] Based on the same inventive concept, corresponding to the method of any of the above embodiments, the present application further provides a computer program product, including computer program instructions. When the computer program instructions run on a computer, the computer is caused to execute the method described in any of the above embodiments, and has the beneficial effects of the corresponding method embodiments, which will not be elaborated here.

[0292] Based on the same inventive concept, the present application further provides a vehicle, including the panoramic surround calibration device disposed at the vehicle end in the above embodiment or the electronic device described in the above embodiment. It has the beneficial effects of the corresponding embodiments of the panoramic surround calibration device or electronic device disposed at the vehicle end, which will not be elaborated here.

[0293] It can be understood that before using the technical solutions of the various embodiments of the present application, the types, usage scopes, usage scenarios, etc. of the personal information involved will be informed to the user in an appropriate manner, and the user's authorization will be obtained.

[0294] For example, when responding to receiving an active request from the user, a prompt message is sent to the user to clearly prompt the user that the operation requested by the user will require obtaining and using the user's personal information. Thus, the user can autonomously choose whether to provide personal information to software or hardware such as an electronic device, application program, server, or storage medium that executes the technical solution of the present application according to the prompt message.

[0295] As an optional but non-limiting implementation manner, the manner of sending a prompt message to the user in response to receiving an active request from the user may be, for example, in the form of a pop-up window, and the prompt message may be presented in text in the pop-up window. In addition, the pop-up window may also carry a selection control for the user to choose "agree" or "disagree" to provide personal information to the electronic device.

[0296] It can be understood that the above process of notifying and obtaining the user's authorization is only illustrative and does not limit the implementation manner of the present application. Other manners that comply with relevant laws and regulations can also be applied to the implementation manner of the present application.

[0297] Those of ordinary skill in the art should understand that the discussion of any of the above embodiments is merely exemplary and is not intended to imply that the scope of the present application (including the claims) is limited to these examples; under the concept of the present application, the technical features in the above embodiments or different embodiments can also be combined, the steps can be implemented in any order, and there are many other variations in different aspects of the embodiments of the present application as described above, and they are not provided in detail for the sake of brevity.

[0298] In addition, for the sake of simplicity of explanation and discussion, and in order not to make the embodiments of the present application difficult to understand, the well-known power / ground connections to integrated circuit (IC) chips and other components may or may not be shown in the provided drawings. Further, the devices may be shown in block diagram form in order to avoid making the embodiments of the present application difficult to understand, and this also takes into account the fact that the details of the implementation of these block diagram devices are highly dependent on the platform on which the embodiments of the present application are to be implemented (i.e., these details should be fully within the understanding of those skilled in the art). In cases where specific details (such as circuits) are set forth to describe exemplary embodiments of the present application, it will be apparent to those skilled in the art that the embodiments of the present application may be implemented without these specific details or with variations of these specific details. Therefore, these descriptions should be considered illustrative rather than restrictive.

[0299] Although the present application has been described in connection with specific embodiments of the present application, many alternatives, modifications, and variations of these embodiments will be apparent to those of ordinary skill in the art based on the foregoing description. For example, other memory architectures (such as dynamic RAM (DRAM)) may be used with the embodiments discussed.

[0300] The embodiments of the present application are intended to cover all such alternatives, modifications, and variations that fall within the broad scope of the appended claims. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the embodiments of the present application shall be included within the protection scope of the present application.

Claims

1. A calibration method for panoramic surround view, characterized in that, Applied to the vehicle end, the method includes: Receiving an instruction for panoramic surround view calibration, obtaining panoramic surround view information, and performing panoramic surround view calibration on the panoramic surround view information by using a corner point calibration method to obtain a first calibration result; In response to the first calibration result being a calibration error, invoking a deep learning model and using the deep learning model to complete correct panoramic surround view calibration on the panoramic surround view information, where the deep learning model is a neural network model that has been pre-trained to be capable of performing panoramic surround view calibration.

2. The method according to claim 1, wherein The using the deep learning model to complete correct panoramic surround view calibration on the panoramic surround view information includes: Performing panoramic surround view calibration on the panoramic surround view information by using the deep learning model to obtain a second calibration result; In response to the second calibration result being a correct calibration, determining that the panoramic surround view calibration process is completed; or, In response to the second calibration result being a calibration error, sending the panoramic surround view information with a calibration error to a service platform for the server platform to train the deep learning model according to the panoramic surround view information with a calibration error to obtain a trained deep learning model; Receiving the trained deep learning model sent by the service platform and using the trained deep learning model to process panoramic surround view data to complete the panoramic surround view calibration process.

3. A calibration method for panoramic surround view, characterized in that, Applied to the service platform, the method includes: Receiving panoramic surround view data with a calibration error fed back by the vehicle end, where when the vehicle end uses the deep learning model to perform panoramic surround view calibration on the panoramic surround view data, the panoramic surround view data will be fed back after a calibration error occurs; Preprocessing the panoramic surround view data and using the preprocessed panoramic surround view data to train the deep learning model to obtain a trained deep learning model; Sending the trained deep learning model to the vehicle end for the vehicle end to use the trained deep learning model to perform correct calibration on the panoramic surround view data.

4. The method according to claim 3, wherein The service platform includes: a data middle platform and a training device, where the deep learning model is stored in the training device; The preprocessing the panoramic surround view data and using the preprocessed panoramic surround view data to train the deep learning model to obtain a trained deep learning model includes: Using the data middle platform to perform filtering and cleaning processing on the panoramic surround view data, then calibrating the cleaned panoramic surround view data according to correct calibration data to obtain calibrated panoramic surround view data, and sending the calibrated panoramic surround view data to the training device; Using the training device to perform training processing on the deep learning model according to the calibrated panoramic surround view data, and iteratively training and adjusting the parameters in the deep learning model according to the difference between the training output result and the correct calibration data until the training output result matches the correct calibration data to obtain a trained deep learning model.

5. The method according to claim 4, characterized in that, The deep learning model is a target detection network model; Using the training device to train and process a deep learning model according to the calibrated panoramic surround view data, and iteratively training and adjusting the parameters in the deep learning model according to the difference between the training output result and the correct calibration data until the training output result matches the correct calibration data, to obtain a trained deep learning model, including: Using the training device to execute: Adjust the size of the calibrated panoramic surround view data, and input the size-adjusted panoramic surround view data into the target detection network model for panoramic surround view calibration processing to obtain a model processing result; Using the non-maximum suppression algorithm to remove redundant detection frames in the model processing result to obtain a trained angle calibration result as the training output result; Compare the training output result with the correct calibration data to determine the degree of difference; Determine the adjustment parameters of each network layer of the target detection network model according to the degree of difference, and adjust and correct each network layer of the target detection network model according to the adjustment parameters to obtain a trained deep learning model.

6. The method according to claim 4 or 5, characterized in that Sending the trained deep learning model to the vehicle terminal includes: Using the training device to execute: Generate a panoramic surround view calibration development tool according to the trained deep learning model; Perform panoramic surround view calibration verification processing according to the panoramic surround view calibration development tool to obtain a calibration verification result; In response to the calibration verification result being incorrect, feedback the verification panoramic surround view data used for calibration verification to the data middle platform, and repeat the data middle platform preprocessing and the training device training processing process until the obtained calibration verification result is correct; In response to the calibration verification result being correct, send the panoramic surround view calibration development tool to the vehicle terminal for the vehicle terminal to execute the functions of the panoramic surround view calibration development tool and perform correct panoramic surround view calibration using the trained deep learning model.

7. The method according to claim 6, wherein The service platform further includes: a cockpit simulation platform; Performing panoramic surround view calibration verification processing according to the panoramic surround view calibration development tool to obtain a calibration verification result, including: Using the cockpit simulation platform to simulate the vehicle and the installation positions of the cameras on the vehicle, and cooperate with the training device to determine a virtual calibration scenario; In the virtual calibration scenario, the training device uses the panoramic surround view calibration development tool to control the cockpit simulation platform to perform virtual calibration adjustment of the panoramic surround view, and display the result obtained from the virtual calibration adjustment for the user to determine according to the displayed calibrated panoramic surround view data; Using the training device to receive the determination result for the calibrated panoramic surround view data, and use the determination result as the calibration verification result.

8. The method according to claim 6, characterized in that, Performing panoramic surround view calibration verification processing according to the panoramic surround view calibration development tool to obtain a calibration verification result, including: Using the training device to perform calibration verification on the panoramic surround view data according to the panoramic surround view calibration development tool, and display the obtained calibrated panoramic surround view data for the user to determine according to the displayed calibrated panoramic surround view data; The training device is used to receive the determination result for the panoramic surround view data after calibration verification, and use the determination result as the calibration verification result.

9. The method according to claim 8, wherein The training device performs the following: in response to an incorrect calibration verification result, feedback the verification panoramic surround view data used for calibration verification to the data center, including: In response to an incorrect calibration verification result, the training device expands the verification panoramic surround view data used for calibration verification into multiple different verification panoramic surround view data in an incremental expansion manner; The training device feeds the multiple different verification panoramic surround view data back to the data center for the data center to perform iterative training processing in combination with the training device based on the multiple different verification panoramic surround view data.

10. An electronic device, comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the method according to any one of claims 1 to 9.

Images