Alignment method and device for model training end and model deployment end, and storage medium

Abstract

The invention discloses an alignment method and device for a model training end and a model deployment end, and a storage medium. The method comprises the following steps: aligning data involved in a data processing stage of the model training end with data involved in a data processing stage of the model deployment end; aligning the model training module of the model training end with the model reasoning module of the model deployment end; when the model is applied to a model deployment end from a model training end, aligning an initial deep learning framework model, an intermediate representation format model and an inference engine model involved in a model framework conversion process; and performing alignment evaluation on the final reasoning effects of the model training end and the model deployment end. According to the method, the alignment process is streamlined and modularized, data processing alignment, model training reasoning alignment, model framework conversion alignment and effect evaluation are carried out respectively, and the prediction effect which the model should reach is guaranteed at the model deployment end.

Description

Technical Field

[0001] The present invention belongs to the technical field of autonomous driving, and specifically relates to a method, device and storage medium for aligning a model training end with a model deployment end. Background Art

[0002] In the field of autonomous driving, urban traffic scenarios often require deep learning models to learn complex interactive game scenarios in order to predict the future trajectories or behaviors of traffic participants. The prediction model undergoes structure and hyperparameter adjustment and multiple rounds of fine-tuning and training to achieve the ideal prediction effect on the server training end. Unlike the model training stage, in the deployment stage, the model parameters are fixed and do not require multiple rounds of parameter debugging tests to achieve the optimal effect.

[0003] In terms of vehicle-side deployment environments, the industry includes a variety of platforms such as mobile devices, embedded devices, and servers. Autonomous driving vehicles are often equipped with graphics cards such as NVIDIA Drive series or Jetson series. Different graphics card configurations require different adjustments to the trained model framework format.

[0004] In terms of model deployment framework conversion, the process generally adopted by the industry is:

[0005] 1) The deep learning training framework Tensorflow / PyTorch directly exports parameters or model structures, or supports exporting in torchscript format and directly deploying on the engineering side in C++libtorch model format;

[0006] 2) PyTorch is first converted into the ONNX / mmdnn intermediate representation format (Intermediate Representation), thereby decoupling from the training framework and target platform, avoiding the adaptation problem of different platforms, and then converted into the TensorRT engine inference engine for deployment. Because each format in the framework conversion belongs to different developers, such as Facebook's PyTorch, LibTorch, Microsoft's ONNX, and NVIDIA's TensorRT, the difference in format during the conversion process may cause loss of model accuracy. Therefore, it is necessary to ensure the consistency of the accuracy effect before and after the operator conversion. Often, based on the model format and effect required by the deployment platform, a specific conversion method will also be considered.

[0007] In terms of data processing, the model training end and the model deployment end often use different processing methods depending on the deployment platform environment.

[0008] On the model deployment side, due to limited hardware resources, it is often necessary to consider the real-time nature of online data processing in order to quickly respond to changes in complex traffic scenarios. On the model training side, with sufficient hardware resources, the amount of training data and data details should be expanded as much as possible.

[0009] The complexity of the deployment environment on the engineering vehicle side, the possible loss of accuracy caused by the conversion of the model framework format, and the complexity of data processing in actual traffic scenarios often lead to loss of model effectiveness during the deployment phase. Therefore, the alignment of the inference effects between the model deployment side and the model training side often plays a decisive role in autonomous driving perception, prediction or other tasks.

[0010] At present, there are many open source development platforms in the industry that integrate multiple deep learning functions. For example, Paddle can provide end-to-end development tools, from model training under different training frameworks to model deployment in different application scenarios or different platforms.

[0011] The FastDeploy deployment tool developed by Feijiang can help deep learning models to be quickly deployed and implemented. It provides data processing tools for some common models, automatic conversion and verification tools from different training frameworks to intermediate representations, and deployment inference methods for different hardware platforms and different backend calls.

[0012] However, Feijiang lacks a unified and procedural overall model effect alignment solution, and it is more oriented towards the deployment of common models on servers or clouds. Therefore, it also lacks application-oriented model deployment and effect alignment solutions for autonomous driving scenarios. Summary of the invention

[0013] In order to solve the above technical problems, the present invention proposes a method, device and storage medium for aligning a model training end with a model deployment end.

[0014] In order to achieve the above object, the technical solution of the present invention is as follows:

[0015] In a first aspect, the present invention discloses a method for aligning a model training end with a model deployment end, comprising:

[0016] Step S1: aligning the data involved in the data processing phase of the model training end with the data involved in the data processing phase of the model deployment end according to the preset data rules;

[0017] Step S2: aligning the model training module at the model training end with the model reasoning module at the model deployment end, so that the logical structure of the model reasoning module is consistent with the model training module, and the prediction effect of the model reasoning module is consistent with the prediction effect of the model training module before the model framework conversion;

[0018] Step S3: When applying the model from the model training end to the model deployment end, align the initial deep learning framework model, intermediate representation format model, and inference engine model involved in the model framework conversion process so that the conversion accuracy meets the requirements;

[0019] Step S4: Align and evaluate the final inference results of the model training end and the model deployment end.

[0020] Based on the above technical solution, the following improvements can be made:

[0021] As a preferred solution, the model architecture is adjusted at the model training end. In the inference process, an end-to-end process is performed from data input to model prediction output, and all operators and structures executed in the middle are recorded to obtain the final static calculation graph.

[0022] The model deployment end has an online model reasoning framework.

[0023] As a preferred solution, step S3 includes:

[0024] Step S3.1: According to the model reasoning environment of the model deployment end, further align the model of the model training end so that it can work in the corresponding model reasoning environment;

[0025] Step S3.2: Convert the original deep learning framework model into an intermediate representation format model, and compare and evaluate the accuracy before and after the conversion. If the accuracy meets the requirements, continue to step S3.3, otherwise re-execute step S3.2 until the accuracy meets the requirements;

[0026] Step S3.3: Convert the intermediate representation format model into the inference engine model, and compare and evaluate the accuracy before and after the conversion. When the accuracy meets the requirements, the step is completed, otherwise re-execute step S3.3 until the accuracy meets the requirements.

[0027] As a preferred solution, step S4 includes:

[0028] Step S4.1: Data evaluation, visually displaying one or more of the results of the model training end and the model deployment end in the data processing stage, the model training results of the model training end, the intermediate representation format model inference results of the model deployment end, the inference engine model inference results of the model deployment end, and the final inference results of the model deployment end;

[0029] Step S4.2: Reasoning performance evaluation, visually display one or more of the multi-stage data processing delay evaluation results of the model deployment end, the model reasoning delay evaluation results of the model deployment end, the CPU performance evaluation results of the model deployment end, and the GPU performance evaluation results of the model deployment end.

[0030] In a second aspect, the present invention further discloses an alignment device for a model training end and a model deployment end, comprising:

[0031] A data alignment module, used to align the data involved in the data processing phase of the model training end with the data involved in the data processing phase of the model deployment end according to preset data rules;

[0032] The model alignment module is used to align the model training module at the model training end with the model reasoning module at the model deployment end, so that the logical structure of the model reasoning module is consistent with the model training module, and the prediction effect of the model reasoning module is consistent with the prediction effect of the model training module before the model framework conversion;

[0033] The conversion alignment module is used to align the initial deep learning framework model, intermediate representation format model, and inference engine model involved in the model framework conversion process when applying the model from the model training end to the model deployment end, so that the conversion accuracy meets the requirements;

[0034] The alignment evaluation module is used to align and evaluate the final inference effects of the model training end and the model deployment end.

[0035] As a preferred solution, the model architecture is adjusted at the model training end. In the inference process, an end-to-end process is performed from data input to model prediction output, and all operators and structures executed in the middle are recorded to obtain the final static calculation graph.

[0036] The model deployment end has an online model reasoning framework.

[0037] As a preferred solution, the conversion alignment module includes:

[0038] A first alignment unit is used to further align the model of the model training end according to the model reasoning environment of the model deployment end, so that the model can work in the corresponding model reasoning environment;

[0039] The second alignment unit is used to convert the original deep learning framework model into an intermediate representation format model, and compare and evaluate the accuracy before and after the conversion. When the accuracy meets the requirements, the method of the third alignment unit is continued to be executed, otherwise the method of the second alignment unit is re-executed until the accuracy meets the requirements;

[0040] The third alignment unit is used to convert the intermediate representation format model into an inference engine model, and compare and evaluate the accuracy before and after the conversion. When the accuracy meets the requirements, the step is completed, otherwise the method of the third alignment unit is re-executed until the accuracy meets the requirements.

[0041] As a preferred solution, the alignment evaluation module includes:

[0042] A data evaluation unit is used for data evaluation, and visualizes one or more of the results of the model training end and the model deployment end in the data processing stage, the model training results of the model training end, the intermediate representation format model reasoning results of the model deployment end, the reasoning results of the inference engine model of the model deployment end, and the final reasoning results of the model deployment end;

[0043] The inference performance evaluation unit is used for inference performance evaluation, and performs visual display of one or more of the multi-stage data processing delay evaluation results of the model deployment end, the model inference delay evaluation results of the model deployment end, the CPU performance evaluation results of the model deployment end, and the GPU performance evaluation results of the model deployment end.

[0044] In a third aspect, the present invention further discloses a computing device, comprising:

[0045] one or more processors;

[0046] Memory;

[0047] And one or more programs, wherein the one or more programs are stored in a memory and configured to be executed by one or more processors, and the one or more programs include instructions for any of the above-mentioned alignment methods between the model training end and the model deployment end.

[0048] In a fourth aspect, the present invention further discloses a storage medium storing one or more computer-readable programs, wherein the one or more programs include instructions suitable for being loaded by a memory and executing any of the above-mentioned alignment methods between a model training end and a model deployment end.

[0049] The present invention also discloses a method, device and storage medium for aligning a model training end with a model deployment end, which has the following beneficial effects:

[0050] The present invention fully considers the possible loss of model accuracy and reasoning efficiency caused by the operating environment of autonomous driving vehicles, model conversion, and data processing, and streamlines and modularizes the alignment process. It goes through data processing alignment, model training and reasoning alignment, model framework conversion alignment, and effect evaluation, and ensures the prediction effect that the model should achieve at the model deployment end. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments are briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without creative work.

[0052] Figure 1A flowchart of a method for aligning a model training end and a model deployment end provided in an embodiment of the present invention.

[0053] Figure 2 A schematic block diagram of the alignment process between the model training end and the model deployment end provided in an embodiment of the present invention.

[0054] Figure 3 A schematic block diagram of the model framework conversion process provided in an embodiment of the present invention.

[0055] Figure 4 A schematic block diagram of effect evaluation provided by an embodiment of the present invention.

[0056] Figure 5 A schematic diagram of a computing device provided for an embodiment of the present invention. DETAILED DESCRIPTION

[0057] The preferred embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0058] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0059] Using ordinal numbers “first,” “second,” “third,” etc. to describe common objects merely indicates that different instances of similar objects are involved and is not intended to imply that the objects so described must have a given order in time, space, order, or in any other manner.

[0060] In addition, the expression of “comprising” an element is an “open” expression, which merely means that corresponding components or steps exist, and should not be interpreted as excluding additional components or steps.

[0061] In order to achieve the purpose of the present invention, in some embodiments of a method for aligning a model training end with a model deployment end, the present invention is applied in the field of autonomous driving, and the model deployment end is arranged on the vehicle end.

[0062] Specifically, Figure 1 As shown, the alignment methods include:

[0063] Step S101: aligning the data involved in the data processing phase of the model training end with the data involved in the data processing phase of the model deployment end according to preset data rules (such as data categories and corresponding data formats);

[0064] Step S102: aligning the model training module of the model training end with the model reasoning module of the model deployment end, so that the logical structure of the model reasoning module is consistent with the model training module, and the prediction effect of the model reasoning module is consistent with the prediction effect of the model training module before the model framework conversion;

[0065] Step S103: when applying the model from the model training end to the model deployment end, align the initial deep learning framework model, intermediate representation format model, and inference engine model involved in the model framework conversion process so that the conversion accuracy meets the requirements;

[0066] Step S104: Align and evaluate the final inference effects of the model training end and the model deployment end.

[0067] Each alignment step is described in detail below.

[0068] In order to ensure that the prediction effects of the models at the model training end and the model deployment end are consistent, the first step S101 needs to align the data processing stage (such as Figure 2 As shown in the figure, the model data processing module at the model training end is aligned with the vehicle-side data processing module at the model deployment end). The model training end uses scripts to perform automated scene mining and data extraction from offline data packets, while the model deployment end reads and processes all the information contained in each frame of the scene (for example, one frame every 0.1s) in real time on the vehicle side and processes the useful data required by the model online.

[0069] For example, in the trajectory prediction task of an autonomous vehicle, the input data required by the deep learning model includes the historical trajectory, category size, speed, orientation angle and other information of traffic participants (motor vehicles, non-motor vehicles, pedestrians, etc.) in the traffic scene, as well as the map searched according to specific rules (for example, lanes and crosswalks searched within a certain range centered on the vehicle) Position coordinates, orientation and other information. The data categories and corresponding data formats should be aligned one by one.

[0070] Of course, in some cases, the real-time processing efficiency of the model deployment end is taken into consideration. For example, the lane search range of the model training end can be all lane information in the entire scene, while the model deployment end only processes the local range considering the real-time performance. The map information processing at both ends is balanced and aligned without affecting or controlling the accuracy loss within the allowable range.

[0071] In the data processing part, the map visualization tool can be used to display the map information searched by the model training end and the model deployment end respectively, and the map lane ID can be output in real time for comparison. In addition, in order to improve the deployment efficiency, multi-threading in C++ is often used to perform parallel data processing.

[0072] The effect alignment performed by the present invention starting from the data source can provide a clean data foundation and can also improve the efficiency of troubleshooting in complex environments.

[0073] The second step S102 requires aligning the model training and reasoning (such as Figure 2 As shown in the figure, the model training module on the model training side is aligned with the vehicle-side model inference module on the model deployment side), thereby ensuring the consistency of the logical structure of the inference model and the consistency of the model prediction effect before the subsequent model framework conversion. The model architecture needs to be adjusted before the model framework conversion. In the inference process, an end-to-end process is experienced from data input to model prediction output to record all the operators and structures executed in the middle, without any selection branches in the middle, so that the final static calculation graph can be obtained. For example: PyTorch, torchscript, ONNX, these intermediate expressions, their formats are independent of the training or inference framework, and can be directly deployed on different platforms. The inference framework for building online models on the vehicle side includes video memory allocation, dynamic dimensional input, asynchronous data transmission, multi-stream asynchronous inference, etc.

[0074] The third step S103 requires the model frame to be transformed and aligned (such as Figure 2 As shown in the figure, the PyTorch model is converted to the TensorRT engine through the model framework conversion module).

[0075] The existing deployment method is to convert the PyTorch model into the ONNX intermediate format, then into the TensorRT engine, and then deploy it on the vehicle side for real-time inference of the model. This conversion involves the process of operator conversion and operator fusion optimization. Therefore, some operators of the model may not be supported by the current TensorRT and need to be implemented in the underlying CUDA (Compute Unified Devices Architecture) language, which is likely to cause a loss in accuracy.

[0076] Therefore, the present invention makes improvements in the model framework conversion, and step S103 includes:

[0077] Step S103.1: According to the model reasoning environment of the model deployment end, further align the model of the model training end so that it can work in the corresponding model reasoning environment;

[0078] Step S103.2: Convert the original deep learning framework model into an intermediate representation format model, and compare and evaluate the accuracy before and after the conversion. If the accuracy meets the requirement, continue to step S103.3, otherwise re-execute step S103.2 until the accuracy meets the requirement;

[0079] Step S103.3: Convert the intermediate representation format model into an inference engine model, and compare and evaluate the accuracy before and after the conversion. When the accuracy meets the requirements, the step is completed, otherwise, re-execute step S103.3 until the accuracy meets the requirements.

[0080] Specifically, Figure 3 As shown, the present invention adopts the following three methods to align the accuracy of the original PyTorch model (the above-mentioned initial deep learning framework model), the intermediate format ONNX model (the above-mentioned intermediate representation format model), and the converted TensorRT engine model (the above-mentioned inference engine model).

[0081] First, the model reasoning environment is aligned. For example, the TensorRT engine has a strong environmental dependence, so different graphics card configurations need to re-convert the reasoning engine unique to the current hardware platform. In addition, for some edge devices such as NVIDIA's Jetson series chips, the model needs to be quantized, so as to reduce the accuracy from FP32 during training to FP16 or INT8. The loss of accuracy also needs to be controlled within a controllable range, otherwise it is necessary to return to the model training end to perform work such as model quantization-aware training QAT (Quantization Aware Training) or post-training quantization PTQ (PostTraining Quantization). Finally, the present invention also provides different back-end implementations and calling methods (Python, C++) for real-time model reasoning, which can synchronize and compare the reasoning effects and efficiency of different back-ends.

[0082] Secondly, align the accuracy of the PyTorch model and the intermediate format ONNX model, and call the Python or C++ backend to compare and evaluate the accuracy of the two, including: building the PyTorch model inference link, building the ONNX real-time runtime inference link, and batch evaluation and comparison. The accuracy difference needs to be controlled within the allowable range. If it does not meet the requirements, return to re-convert the ONNX model, including but not limited to changing the model structure link, avoiding or changing operators that may cause problems in the conversion process from PyTorch to ONNX, until the accuracy is within a certain range before proceeding to the next step of alignment.

[0083] Finally, align the accuracy of the intermediate format ONNX model and the TensorRT engine model, and call the Python or C++ backend to compare and evaluate the accuracy of the two, including: the previous ONNX real-time runtime inference link, the construction of the TensorRT engine inference link, and batch comparison and evaluation. If it does not meet the requirements, return to the TensorRT engine conversion again, including but not limited to changing the model structure link, avoiding or changing operators that may cause problems from ONNX to the TensorRT engine, until the accuracy is within a certain range.

[0084] The fourth step S104 performs an overall alignment evaluation of the final inference effect of the model at the model training end and the model deployment end, including data evaluation and inference performance evaluation.

[0085] Specifically, step S104 includes:

[0086] Step S104.1: Data evaluation, visually displaying one or more of the results of the model training end and the model deployment end in the data processing stage, the model training results of the model training end, the intermediate representation format model reasoning results of the model deployment end, the reasoning results of the inference engine model of the model deployment end, and the final reasoning results of the model deployment end;

[0087] Step S104.2: Reasoning performance evaluation, visually displaying one or more of the multi-stage data processing delay evaluation results of the model deployment end, the model reasoning delay evaluation results of the model deployment end, the CPU performance evaluation results of the model deployment end, and the GPU performance evaluation results of the model deployment end. Figure 4 As shown, in data evaluation, in order to improve the evaluation efficiency, the present invention develops visualization functions in multiple intermediate stages to display the intermediate results of each alignment module, making the alignment process clearer and separable, including visualization of the results of the data processing stages at the model training end and the model deployment end (obstacle trajectory visualization and search area map visualization), visualization of the model training results at the model training end, visualization of the ONNX reasoning results at the model deployment end, visualization of the TensorRT engine reasoning results at the model deployment end, and visualization of the final vehicle-side reasoning results, thereby making the comparison clearer.

[0088] At the same time, in some other embodiments, the present invention uses a large number of corner case extreme cases that may cause large precision losses for evaluation, extracts abnormal frames with large data visualization comparison differences for investigation, compares the intermediate data of the current abnormal frames output in real time by the model deployment end and the model training end, and adopts the control variable method to change only one variable that may cause mutation each time, so as to search for the biggest source of loss more quickly.

[0089] In terms of inference performance evaluation, the model deployment end includes multi-stage data processing latency evaluation, model inference latency evaluation, CPU performance evaluation (usage rate, memory usage), GPU performance evaluation (usage rate, video memory usage), etc. In addition, some sudden or occasional core dump program crash events will greatly affect the inference effect of the vehicle-side model. Therefore, it is necessary to use the core core file to query the possible crash location. The source may include all the alignment operation design modules mentioned above. Specific problems should be traced back and investigated.

[0090] In some other embodiments, the present invention further discloses a device for aligning a model training end and a model deployment end, comprising:

[0091] A data alignment module, used to align the data involved in the data processing phase of the model training end with the data involved in the data processing phase of the model deployment end according to preset data rules;

[0092] The model alignment module is used to align the model training module at the model training end with the model reasoning module at the model deployment end, so that the logical structure of the model reasoning module is consistent with the model training module, and the prediction effect of the model reasoning module is consistent with the prediction effect of the model training module before the model framework conversion;

[0093] The conversion alignment module is used to align the initial deep learning framework model, intermediate representation format model, and inference engine model involved in the model framework conversion process when applying the model from the model training end to the model deployment end, so that the conversion accuracy meets the requirements;

[0094] The alignment evaluation module is used to align and evaluate the final inference effects of the model training end and the model deployment end.

[0095] Furthermore, the model architecture is adjusted at the model training end. In the inference process, an end-to-end process is experienced from data input to model prediction output, and all operators and structures executed in the middle are recorded to obtain the final static calculation graph.

[0096] The model deployment end has an online model reasoning framework.

[0097] Furthermore, the transformation alignment module includes:

[0098] A first alignment unit is used to further align the model of the model training end according to the model reasoning environment of the model deployment end, so that the model can work in the corresponding model reasoning environment;

[0099] The second alignment unit is used to convert the original deep learning framework model into an intermediate representation format model, and compare and evaluate the accuracy before and after the conversion. When the accuracy meets the requirements, the method of the third alignment unit is continued to be executed, otherwise the method of the second alignment unit is re-executed until the accuracy meets the requirements;

[0100] The third alignment unit is used to convert the intermediate representation format model into an inference engine model, and compare and evaluate the accuracy before and after the conversion. When the accuracy meets the requirements, the step is completed, otherwise the method of the third alignment unit is re-executed until the accuracy meets the requirements.

[0101] Furthermore, the alignment assessment module includes:

[0102] A data evaluation unit is used for data evaluation, and visualizes one or more of the results of the model training end and the model deployment end in the data processing stage, the model training results of the model training end, the intermediate representation format model reasoning results of the model deployment end, the reasoning results of the inference engine model of the model deployment end, and the final reasoning results of the model deployment end;

[0103] The inference performance evaluation unit is used for inference performance evaluation, and performs visual display of one or more of the multi-stage data processing delay evaluation results of the model deployment end, the model inference delay evaluation results of the model deployment end, the CPU performance evaluation results of the model deployment end, and the GPU performance evaluation results of the model deployment end.

[0104] Furthermore, it should be noted that: the alignment device between the model training end and the model deployment end provided in the above embodiment only uses the division of the above-mentioned functional modules as an example when determining the dominant vertex. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the alignment device between the model training end and the model deployment end is divided into different functional modules to complete all or part of the functions described above.

[0105] In addition, the embodiments of the device for aligning the model training end and the model deployment end and the method for aligning the model training end and the model deployment end provided in the above embodiments belong to the same concept, and their specific implementation process is detailed in the method embodiments and will not be repeated here.

[0106] In addition, in some other embodiments, Figure 5 As shown, the present invention also discloses a computing device, including:

[0107] One or more processors 201;

[0108] Memory 202;

[0109] And one or more programs, wherein the one or more programs are stored in the memory 202 and are configured to be executed by one or more processors 201, and the one or more programs include instructions for the alignment method between the model training end and the model deployment end disclosed in the above embodiment.

[0110] The processor 201 may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 201 may be implemented in at least one hardware form of DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). The processor 201 may also include a main processor and a coprocessor. The main processor is a processor for processing data in an awake state, also known as a CPU (Central Processing Unit); the coprocessor is a low-power processor for processing data in a standby state. In some embodiments, the processor 201 may be integrated with a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 201 may also include an AI (Artificial Intelligence) processor, which is used to process computing operations related to machine learning.

[0111] The memory 202 may include one or more computer-readable storage media, which may be non-transitory. The memory 202 may also include a high-speed random access memory, and a non-volatile memory, such as one or more disk storage devices, flash memory storage devices. In some embodiments, the non-transitory computer-readable storage medium in the memory 202 is used to store at least one instruction, which is used to be executed by the processor 201 to implement the alignment method of the model training end and the model deployment end provided in the method embodiment of the present invention.

[0112] In addition, the computing device may optionally include: a peripheral device interface and at least one peripheral device. The processor 201, the memory 202 and the peripheral device interface may be connected via a bus or a signal line. Each peripheral device may be connected to the peripheral device interface via a bus, a signal line or a circuit board. Schematically, the peripheral devices include but are not limited to: a radio frequency circuit, a touch display screen, an audio circuit, and a power supply.

[0113] Of course, the computing device may also include fewer or more components, which is not limited in this embodiment.

[0114] In addition, in some other embodiments, the present invention also discloses a storage medium, which stores one or more computer-readable programs, and the one or more programs include instructions, and the instructions are suitable for being loaded by a memory and executing the alignment method between the model training end and the model deployment end disclosed in the above embodiments.

[0115] The present invention also discloses a method, device and storage medium for aligning a model training end with a model deployment end, which has the following beneficial effects:

[0116] The present invention fully considers the possible loss of model accuracy and reasoning efficiency caused by the operating environment of autonomous driving vehicles, model conversion, and data processing, and streamlines and modularizes the alignment process. It goes through data processing alignment, model training and reasoning alignment, model framework conversion alignment, and effect evaluation, and ensures the prediction effect that the model should achieve at the model deployment end.

[0117] The above shows and describes the basic principles and main features of the present invention and the advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The above embodiments and descriptions are only for illustrating the principles of the present invention. Without departing from the spirit and scope of the present invention, the present invention may have various changes and improvements, which shall fall within the scope of the present invention to be protected. The scope of protection of the present invention is defined by the attached claims and their equivalents.

Claims

1. A method for aligning a model training end with a model deployment end, characterized in that: include: Step S1: aligning the data involved in the data processing phase of the model training end with the data involved in the data processing phase of the model deployment end according to the preset data rules; Step S2: aligning the model training module at the model training end with the model reasoning module at the model deployment end, so that the logical structure of the model reasoning module is consistent with the model training module, and the prediction effect of the model reasoning module is consistent with the prediction effect of the model training module before the model framework conversion; Step S3: When applying the model from the model training end to the model deployment end, align the initial deep learning framework model, intermediate representation format model, and inference engine model involved in the model framework conversion process so that the conversion accuracy meets the requirements; Step S4: Align and evaluate the final inference results of the model training end and the model deployment end.

2. The alignment method according to claim 1, characterized in that: The model architecture is adjusted at the model training end. In the inference process, an end-to-end process is experienced from data input to model prediction output, and all operators and structures executed in the middle are recorded to obtain the final static calculation graph. The model deployment end has an online model reasoning framework.

3. The alignment method according to claim 1, characterized in that: The step S3 comprises: Step S3.1: According to the model reasoning environment of the model deployment end, further align the model of the model training end so that it can work in the corresponding model reasoning environment; Step S3.2: Convert the original deep learning framework model into an intermediate representation format model, and compare and evaluate the accuracy before and after the conversion. If the accuracy meets the requirements, continue to step S3.3, otherwise re-execute step S3.2 until the accuracy meets the requirements; Step S3.3: Convert the intermediate representation format model into the inference engine model, and compare and evaluate the accuracy before and after the conversion. When the accuracy meets the requirements, the step is completed, otherwise re-execute step S3.3 until the accuracy meets the requirements.

4. The alignment method according to claim 1, characterized in that: The step S4 comprises: Step S4.1: Data evaluation, visually displaying one or more of the results of the model training end and the model deployment end in the data processing stage, the model training results of the model training end, the intermediate representation format model inference results of the model deployment end, the inference engine model inference results of the model deployment end, and the final inference results of the model deployment end; Step S4.2: Reasoning performance evaluation, visually display one or more of the multi-stage data processing delay evaluation results of the model deployment end, the model reasoning delay evaluation results of the model deployment end, the CPU performance evaluation results of the model deployment end, and the GPU performance evaluation results of the model deployment end.

5. The device for aligning the model training end and the model deployment end is characterized by comprising: A data alignment module, used to align the data involved in the data processing phase of the model training end with the data involved in the data processing phase of the model deployment end according to preset data rules; The model alignment module is used to align the model training module at the model training end with the model reasoning module at the model deployment end, so that the logical structure of the model reasoning module is consistent with the model training module, and the prediction effect of the model reasoning module is consistent with the prediction effect of the model training module before the model framework conversion; The conversion alignment module is used to align the initial deep learning framework model, intermediate representation format model, and inference engine model involved in the model framework conversion process when applying the model from the model training end to the model deployment end, so that the conversion accuracy meets the requirements; The alignment evaluation module is used to align and evaluate the final inference effects of the model training end and the model deployment end.

6. The alignment device according to claim 5, characterized in that: The model architecture is adjusted at the model training end. In the inference process, an end-to-end process is experienced from data input to model prediction output, and all operators and structures executed in the middle are recorded to obtain the final static calculation graph. The model deployment end has an online model reasoning framework.

7. The alignment device according to claim 5, characterized in that: The conversion alignment module comprises: A first alignment unit is used to further align the model of the model training end according to the model reasoning environment of the model deployment end, so that the model can work in the corresponding model reasoning environment; The second alignment unit is used to convert the original deep learning framework model into an intermediate representation format model, and compare and evaluate the accuracy before and after the conversion. When the accuracy meets the requirements, the method of the third alignment unit is continued to be executed, otherwise the method of the second alignment unit is re-executed until the accuracy meets the requirements; The third alignment unit is used to convert the intermediate representation format model into an inference engine model, and compare and evaluate the accuracy before and after the conversion. When the accuracy meets the requirements, the step is completed, otherwise the method of the third alignment unit is re-executed until the accuracy meets the requirements.

8. The alignment device according to claim 5, characterized in that: The alignment evaluation module includes: A data evaluation unit is used for data evaluation, and visualizes one or more of the results of the model training end and the model deployment end in the data processing stage, the model training results of the model training end, the intermediate representation format model reasoning results of the model deployment end, the reasoning results of the inference engine model of the model deployment end, and the final reasoning results of the model deployment end; The inference performance evaluation unit is used for inference performance evaluation, and performs visual display of one or more of the multi-stage data processing delay evaluation results of the model deployment end, the model inference delay evaluation results of the model deployment end, the CPU performance evaluation results of the model deployment end, and the GPU performance evaluation results of the model deployment end.

9. A computing device, characterized in that include: one or more processors; Memory; And one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by one or more processors, and one or more of the programs include instructions for the alignment method of the model training end and the model deployment end as described in any of claims 1-4 above.

10. A storage medium, characterized in that The storage medium stores one or more computer-readable programs, and the one or more programs include instructions, which are suitable for being loaded by the memory and executing the alignment method between the model training end and the model deployment end described in any one of claims 1-4.

Images