Model application method and device, vehicle, medium and program product

Abstract

The embodiment of the invention provides a model application method and device, a vehicle, a medium and a program product. The method comprises the following steps: firstly, determining a current network layer corresponding to a current data input address according to a model structure array of a target model and the current data input address; then, according to a function corresponding to the function pointer of the current network layer and a weight corresponding to the weight pointer, calculating data in the current data input address, and generating a calculation result; then, storing a calculation result to a data output address corresponding to a data output address pointer of the current network layer; and finally, determining the data output address as a new current data input address, repeating the previous steps until the data output address is a model result output address, and determining the current calculation result as a model output result. Through the method, the flexibility of the model in the execution process and the later maintenance process is improved.

Description

Technical Field

[0001] The present application relates to the field of communication technology, and in particular to a model application method, device, vehicle, medium and program product. Background Art

[0002] With the rapid development of technologies such as intelligent driving, in-vehicle artificial intelligence, and autonomous driving, in-vehicle systems have gradually become a key application for deep learning models. Commonly used deep learning models are typically trained on platforms using graphics processing units (GPUs), while in-vehicle systems typically use microcontroller units (MCUs). Therefore, how to deploy deep learning models from the cloud to the vehicle is an urgent problem to be solved.

[0003] Currently, given the different computing resources available on the cloud and on the vehicle, the models use different inference frameworks. Therefore, deep learning models are first converted from existing frameworks (such as PyTorch and TensorFlow) to the ONNX framework through the Open Neural Network eXchange (ONNX) to meet the requirements for running the model on the vehicle. The converted deep learning model is then deployed on the vehicle for inference in real-world applications.

[0004] However, models deployed through existing means have poor flexibility during use. Summary of the Invention

[0005] The embodiments of the present application provide a model application method, device, vehicle, medium and program product to solve the problem of poor flexibility in the use of models in the prior art.

[0006] In a first aspect, an embodiment of the present application provides a model application method, comprising:

[0007] Determine, according to a model structure array of a target model and a current data input address, a current network layer corresponding to the current data input address, wherein the model structure array includes metadata of each network layer in the target model, the metadata including a function pointer, a weight pointer, a data input address pointer, and a data output address pointer, the function pointer points to a function corresponding to the network layer in a function array, and the weight pointer points to a weight corresponding to the network layer in a weight array;

[0008] Calculate the data in the current data input address according to the function corresponding to the function pointer of the current network layer and the weight corresponding to the weight pointer to generate a calculation result;

[0009] Storing the calculation result to the data output address corresponding to the data output address pointer of the current network layer;

[0010] The data output address is determined as a new current data input address, and the above steps are repeated until the data output address is the model result output address, and the current calculation result is determined as the model output result.

[0011] In a possible implementation, before determining the current network layer corresponding to the current data input address based on the model structure array of the target model and the current data input address, the method further includes:

[0012] Converting the model format of the to-be-deployed model into a model format suitable for a computing unit with limited computing resources to generate the target model;

[0013] Encapsulate the functions and weights of the same type of network layers in the target model respectively, and generate function arrays and weight arrays corresponding to each network layer type;

[0014] determining metadata for each network layer in the target model;

[0015] Generate the model structure array according to the metadata of each network layer.

[0016] In a possible implementation, the model structure array further includes input data dimensions, output data dimensions, and sequence length of each model layer.

[0017] In a possible implementation, generating the model structure array according to the metadata of each network layer includes:

[0018] Merging adjacent network layers of a preset type in the target model into a target network layer;

[0019] Determine the metadata of the target network layer based on the metadata of the adjacent network layer;

[0020] Generate the model structure array according to the metadata of each network layer.

[0021] In one possible implementation, calculating the data in the current data input address according to the function corresponding to the function pointer of the current network layer and the weight corresponding to the weight pointer to generate a calculation result includes:

[0022] If the current network layer is the target network layer, merging functions corresponding to multiple function pointers of the current network layer to generate a target function;

[0023] Based on the relationship between the functions corresponding to the multiple function pointers of the current network layer, the weights corresponding to the multiple weight pointers corresponding to the current network layer are merged to generate a target weight;

[0024] The data in the current data input address is calculated according to the objective function and the target weight to generate the calculation result.

[0025] In one possible implementation, converting the model format of the to-be-deployed model into a model format suitable for a computing unit with limited computing resources to generate the target model includes:

[0026] Converting the model format of the to-be-deployed model into a model format suitable for a computing unit with limited computing resources, thereby generating a converted model;

[0027] For each network layer in the converted model, the operations in the network layer that meet the merging strategy are merged to generate the target model.

[0028] In one possible implementation, the target model is a driving behavior prediction model, which is used to predict the driving behavior of at least one adjacent vehicle based on environmental timing information, the vehicle timing information of the vehicle itself, and the adjacent vehicle timing information during vehicle driving.

[0029] In a second aspect, an embodiment of the present application provides a model application device, comprising:

[0030] a determination module, configured to determine, based on a model structure array of a target model and a current data input address, a current network layer corresponding to the current data input address, wherein the model structure array includes metadata of each network layer in the target model, the metadata including a function pointer, a weight pointer, a data input address pointer, and a data output address pointer, the function pointer pointing to a function corresponding to the network layer in the function array, and the weight pointer pointing to a weight corresponding to the network layer in the weight array;

[0031] a calculation module, configured to calculate the data in the current data input address according to the function corresponding to the function pointer of the current network layer and the weight corresponding to the weight pointer, and generate a calculation result;

[0032] An output module, configured to store the calculation result to a data output address corresponding to the data output address pointer of the current network layer;

[0033] The processing module is used to determine the data output address as a new current data input address, repeat the above steps until the data output address is a model result output address, and determine the current calculation result as a model output result.

[0034] In a possible implementation, the model application device further includes a generation module. Before determining the current network layer corresponding to the current data input address based on the model structure array of the target model and the current data input address, the generation module is specifically configured to:

[0035] Converting the model format of the to-be-deployed model into a model format suitable for a computing unit with limited computing resources to generate the target model;

[0036] Encapsulate the functions and weights of the same type of network layers in the target model respectively, and generate function arrays and weight arrays corresponding to each network layer type;

[0037] determining metadata for each network layer in the target model;

[0038] Generate the model structure array according to the metadata of each network layer.

[0039] In a possible implementation, the model structure array further includes input data dimensions, output data dimensions, and sequence length of each model layer.

[0040] In a possible implementation, the generating module is specifically configured to:

[0041] Merging adjacent network layers of a preset type in the target model into a target network layer;

[0042] Determine the metadata of the target network layer based on the metadata of the adjacent network layer;

[0043] Generate the model structure array according to the metadata of each network layer.

[0044] In a possible implementation, the calculation module is specifically configured to:

[0045] If the current network layer is the target network layer, merging functions corresponding to multiple function pointers of the current network layer to generate a target function;

[0046] Based on the relationship between the functions corresponding to the multiple function pointers of the current network layer, the weights corresponding to the multiple weight pointers corresponding to the current network layer are merged to generate a target weight;

[0047] The data in the current data input address is calculated according to the objective function and the target weight to generate the calculation result.

[0048] In a possible implementation, the generating module is specifically configured to:

[0049] Converting the model format of the to-be-deployed model into a model format suitable for a computing unit with limited computing resources, thereby generating a converted model;

[0050] For each network layer in the converted model, the operations in the network layer that meet the merging strategy are merged to generate the target model.

[0051] In one possible implementation, the target model is a driving behavior prediction model, which is used to predict the driving behavior of at least one adjacent vehicle based on environmental timing information, the vehicle timing information of the vehicle itself, and the adjacent vehicle timing information during vehicle driving.

[0052] In a third aspect, an embodiment of the present application provides a vehicle, characterized by comprising: a vehicle body, a memory, and a processor;

[0053] The memory stores computer-executable instructions;

[0054] The processor executes the computer-executable instructions stored in the memory, so that the processor executes the above first aspect and / or various possible implementations of the first aspect.

[0055] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, in which computer-executable instructions are stored. When the computer-executable instructions are executed by a processor, they are used to implement the first aspect above and / or various possible implementation methods of the first aspect.

[0056] In a fifth aspect, an embodiment of the present application provides a computer program product, including a computer program, which, when executed by a processor, implements the above first aspect and / or various possible implementation methods of the first aspect.

[0057] The model application method, device, vehicle, medium and program product provided by the embodiment of the present application include: first, according to the model structure array of the target model and the current data input address, determining the current network layer corresponding to the current data input address; then, according to the function corresponding to the function pointer of the current network layer and the weight corresponding to the weight pointer, calculating the data in the current data input address to generate a calculation result; then, storing the calculation result to the data output address corresponding to the data output address pointer of the current network layer; finally, determining the data output address as the new current data input address, repeating the above steps until the data output address is the model result output address, and determining the current calculation result as the model output result. In this technical solution, the fixed format of the target model is converted into a more flexible storage format through the model structure array. When the structure of the target model needs to be improved, it is only necessary to change the metadata (function pointer, weight pointer, data input address pointer and data output address pointer) of the corresponding layer in the target model without modifying too many parameters, so that the calculation of each layer in the model can be independently developed, modified or expanded without affecting the overall model structure, thereby reducing the complexity of the modification process. Moreover, for the same data (weights, functions) in different network layers of the target model, only one copy of the model structure array needs to be stored, which reduces the use of memory and unnecessary data copying and access during the calculation process, and improves the execution efficiency of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0059] Figure 1 Schematic diagram of the process of the model application method provided in the embodiment of this application Figure 1 ;

[0060] Figure 2 Schematic diagram of the process of the model application method provided in the embodiment of this application Figure 2 ;

[0061] Figure 3 A schematic diagram of the structure of the model application device provided in an embodiment of the present application;

[0062] Figure 4 A schematic diagram of the structure of a vehicle provided in an embodiment of the present application.

[0063] The above drawings illustrate specific embodiments of the present application, which will be described in more detail below. These drawings and the textual description are not intended to limit the scope of the present application in any way, but rather to illustrate the concepts of the present application to those skilled in the art by reference to specific embodiments. DETAILED DESCRIPTION

[0064] Exemplary embodiments will be described in detail herein, with examples illustrated in the accompanying drawings. In the following description, when referring to the drawings, identical numerals in different figures represent identical or similar elements, unless otherwise indicated. The embodiments described in the following exemplary embodiments are not intended to represent all embodiments consistent with the present application. Rather, they are merely examples of apparatus and methods consistent with certain aspects of the present application, as detailed in the appended claims.

[0065] First, let’s explain the terms used in this application:

[0066] Matrix multiplication: Matrix multiplication optimization based on loop order has been widely used since the days before GPUs. In naive implementations, this significantly slows down processing of large matrices, especially in deep learning, because the multiplications in the deepest loop cannot continuously access data from memory. Matrix multiplication is performed by swapping the last two layers of the loop to accumulate the results. Each element in the resulting matrix is obtained by iteratively multiplying and accumulating the matrix row by row, making data access continuous. Matrix multiplication is the backbone of the Transformer model, as the self-attention mechanism and linear layer operations in the Transformer model are essentially matrix multiplications.

[0067] Next, the application background involved in this application is explained.

[0068] With the rapid development of technologies such as intelligent driving, in-vehicle artificial intelligence, and autonomous driving, in-vehicle systems have gradually become a key application scenario for deep learning models. Common deep learning models are typically deployed on platforms using GPUs, while in-vehicle systems typically use microcontrollers (MCUs). Because MCUs have lower computing power than GPUs and lack the parallelization capabilities of GPUs, establishing a method for deploying deep learning models from the cloud to the vehicle is crucial to ensure that models can be correctly applied in in-vehicle systems.

[0069] Currently, the main task is to convert the deep learning model from the current framework to the inference framework (also known as the inference format) corresponding to the vehicle system, such as ONNX. After that, the converted deep learning model can be deployed on the vehicle side for inference use in actual applications.

[0070] However, the layer structure, calculation graph and execution process within the model of the prior art are basically static and fixed. When it is necessary to modify the internal structure or execution order of the model, especially when it involves modifications to the network layer (such as adding, deleting or adjusting certain layers), it is usually necessary to modify a large amount of data. For example, if you need to modify the function used in the pooling layer, you need to modify all the pooling layers in the model. The entire process needs to be implemented manually, and the efficiency of the modification cannot be guaranteed. In addition, due to the complexity of the model, manual modification will miss some pooling layers. For example, if you want to delete a network layer in the model, after deleting the network layer, you need to manually re-change the connection relationship between the layers and redesign the execution order, which takes a lot of time.

[0071] In summary, if the above modifications are made to the model during its use, the model will have poor flexibility.

[0072] The model application method provided by the present application takes into account that the layer structure, calculation graph and execution process within the model in the prior art are fixed, and there is dependency between different layers of the model. Even a small change to the model will cause a lot of workload. Based on the above considerations, the inventors found during relevant research that the layer structure information of the model can be modularized and parameterized, and the model structure array can be used to explain the model. Among them, the model structure array includes metadata for each network layer in the target model, and the metadata includes a function pointer, a weight pointer, a data input address pointer and a data output address pointer. The function pointer points to the corresponding function of the network layer in the function array, and the weight pointer points to the corresponding weight of the network layer in the weight array. In this way, when the model needs to be modified, the model structure array can be modified without redesigning the execution order or modifying the relevant parameters of each layer. The model can be modified flexibly during use without affecting the use of the model.

[0073] The following specific embodiments describe in detail the technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below in conjunction with the accompanying drawings.

[0074] Figure 1 Schematic diagram of the process of the model application method provided in the embodiment of this application Figure 1 ,like Figure 1 As shown, the method includes:

[0075] S101. Determine the current network layer corresponding to the current data input address according to the model structure array of the target model and the current data input address.

[0076] Among them, the model structure array includes the metadata of each network layer in the target model; the metadata includes function pointers, weight pointers, data input address pointers and data output address pointers; the function pointer points to the corresponding function of the network layer in the function array; the weight pointer points to the corresponding weight of the network layer in the weight array, the data input address pointer points to the address where the network layer obtains data, and the data output address pointer points to the address where the calculation and processing results of the network layer are stored.

[0077] Alternatively, the weight array can be stored as a flat global array of 32-bit floating point numbers (float32).

[0078] It's important to note that the model structure array also includes the input data dimensions, output data dimensions, and sequence length for each model layer. The input data dimensions describe the shape or size of the data received by each network layer (e.g., the number of rows, columns, and channels of a tensor). For example, the input to a convolutional layer might be a 3D tensor with dimensions (224, 224, 3), indicating a width of 224, a height of 224, and 3 channels. The output data dimensions describe the shape or size of the output data processed by each network layer. For example, after convolution, the output dimensions might become (112, 112, 64), indicating that the feature map is halved and 64 channels are generated. The sequence length describes the length of the input sequence. For example, if the input sequence length of a network layer is 10, it indicates that it needs to process data of 10 time steps. In addition to the aforementioned key information, the model structure array can also include auxiliary information to more fine-tune the data characteristics, such as the data type (floating point, integer, etc.); the data distribution range (maximum and minimum values); and the batch size.

[0079] It should be understood that in actual applications, the model structure array may also include parameters related to data processing at the network layer, which can be determined based on actual conditions. The embodiments of the present application do not impose specific restrictions on this.

[0080] It should be understood that based on the current data input address, the current network layer corresponding to the current data input address can be quickly determined through the metadata in each network layer in the model structure array of the target model. In this way, the calculation of each layer of the target model can be independently managed and executed. Moreover, when the target model needs to be modified, the execution logic and parameter allocation of the current network layer can be quickly changed by simply modifying the function pointer, weight pointer, data input address pointer, and data output address pointer, thereby reducing the amount of modification to the target model, improving the efficiency of target model modification and the flexibility of the model usage process.

[0081] In practical applications, the target model can be a driving behavior prediction model, which is used to predict the driving behavior of at least one adjacent vehicle based on the environment time series information, the vehicle time series information and the adjacent vehicle time series information during vehicle driving.

[0082] In actual application, the driving behavior prediction model has 15,850 32-bit floating-point parameters, and the average running time in a task with a cycle of 60 milliseconds reaches 5 milliseconds, 409 microseconds, and 612 nanoseconds.

[0083] In one feasible method, the driving behavior prediction model can be pre-trained on a specific cloud platform. The specific training process can be: during the driving process of the sample vehicle, multiple training samples containing environmental time series information, the vehicle time series information of the vehicle itself, and the time series information of adjacent vehicles are collected; then, the annotation information corresponding to the time series information of the adjacent vehicles of each sample is obtained, such as the vehicle's lane entry, lane exit, left turn, right turn and other behaviors; then, the initial model is trained according to the multiple training samples and the annotation information corresponding to each training sample to generate a driving behavior prediction model.

[0084] The loss function of the initial model includes a cross entropy loss sub-function and a constrained false positive sub-function. For example, the cross entropy loss sub-function can be expressed by the following formula:

[0085]

[0086] in, is the value of the cross entropy loss sub-function, y is the label value, is the predicted value.

[0087] Exemplarily, the constrained false alarm sub-function can be expressed by the following formula:

[0088]

[0089] in, To constrain the value of the false positive sub-function, weights is the weight.

[0090] It is understood that in the above example, the weight corresponding to other driving scenarios is 1, and the weight corresponding to the sample vehicle changing lanes or driving on a curve is a positive number greater than 1. In actual applications, the weight corresponding to other driving scenarios can also be other values, and this example only uses 1 for illustration.

[0091] During extensive validation of the trained driving behavior prediction model, we found that false positives primarily occurred when the sample vehicle changed lanes or drove on curves. Therefore, to ensure the initial model focused on these scenarios, we assigned higher weights to these scenarios during training. This improved the accuracy of the driving behavior prediction model and reduced its false positive rate.

[0092] It should be understood that a false positive of a driving behavior prediction model mainly refers to the driving behavior prediction model predicting an incorrect driving behavior or failing to predict a driving behavior.

[0093] S102. Calculate the data in the current data input address according to the function corresponding to the function pointer of the current network layer and the weight corresponding to the weight pointer to generate a calculation result.

[0094] It should be noted that during the calculation process, the corresponding function must first be determined based on the function pointer of the current network layer. Then, the data stored at the current data input address is obtained. Finally, the function is called to process the data and generate the calculation result.

[0095] S103. Store the calculation result in the data output address corresponding to the data output address pointer of the current network layer.

[0096] It should be understood that compared with the traditional model in which the calculation results of each layer are first stored in a temporary variable and then copied or transferred to the next network layer, the embodiment of the present application provides a method of directly storing the calculation results to the data output address corresponding to the data output address pointer of the current network layer, which effectively reduces memory usage and saves the limited computing resources of the MCU.

[0097] It can be understood that by storing the calculation results at the data output address corresponding to the data output address pointer of the current network layer so that subsequent layers can use them, data reuse in memory is achieved, further optimizing storage efficiency.

[0098] S104: Determine the data output address as the new current data input address, repeat the above steps until the data output address is the model result output address, and determine the current calculation result as the model output result.

[0099] It's important to note that by directly setting the data output address to the new current data input address, the need to store intermediate results in temporary storage, as in traditional methods, is avoided, further reducing memory overhead. Furthermore, using the output address to control data flow facilitates rapid and dynamic adjustment of the execution order of network layers without affecting the overall logic.

[0100] For example, assume a simple neural network consisting of three network layers: L1, L2, and L3. The normal execution order is: L1, L2, and L3, that is, the data output address pointer of L1 points to the data input address of L2, the data output address pointer of L2 points to the data input address of L3, and the data output address pointer of L3 points to the model result output address. In a specific scenario, when the execution order of the three network layers of the neural network needs to be changed to: L1, L3, and L2, it is only necessary to change the output address pointer of the corresponding network layer, that is, the output address pointer of L1 points to the input address of L3, the output address pointer of L3 points to the input address of L2, and the output address pointer of L2 points to the model result output address. There is no need to modify a large number of parameters related to layers and layer connections, which improves the flexibility of model application.

[0101] It can be understood that by forming the metadata of each network layer of the driving behavior prediction model into a model structure array, in the subsequent execution process, it is only necessary to determine the function pointer and weight pointer corresponding to each network layer based on the data input address, dynamically perform network layer calculations and store the calculation results to the data output address corresponding to the data output address pointer of the network layer, and then update the input address and continue the calculation, that is, determine the data output address as the new current data input address for calculation. In this way, the calculation of the entire model will be carried out layer by layer, from the first layer to the last layer of the target model, effectively ensuring the flexibility of the target model operation. In addition, the model weights are stored in the weight array, and the model functions are stored in the function array, so that the same data (weights, functions) only need to be stored once, avoiding the repeated storage of the same data in the existing technology and reducing the use of memory.

[0102] The model application method provided by the embodiment of the present application, first, according to the model structure array of the target model and the current data input address, the current network layer corresponding to the current data input address is determined; then, according to the function corresponding to the function pointer of the current network layer and the weight corresponding to the weight pointer, the data in the current data input address is calculated to generate a calculation result; then, the calculation result is stored in the data output address corresponding to the data output address pointer of the current network layer; finally, the data output address is determined as the new current data input address, and the above steps are repeated until the data output address is the model result output address, and the current calculation result is determined as the model output result. In this technical solution, the target model is converted from the existing fixed format to the model structure array. When the structure of the target model needs to be improved, it is only necessary to change the metadata of the corresponding layer in the target model without modifying too many parameters, so that the calculation of each layer in the model can be independently developed, modified or expanded without affecting the overall model structure, reducing the complexity of the modification process, and each layer directly accesses data and weights through pointers. The same data (weights, functions) only need to be stored once, reducing the use of memory and unnecessary data copying and access during the calculation process, thereby improving the efficiency of model execution.

[0103] Figure 2 Schematic diagram of the process of the model application method provided in the embodiment of this application Figure 2 ;like Figure 2 As shown, this embodiment Figure 2 Based on the embodiment, the process of generating a model structure array before deploying the model is described in detail. The method includes:

[0104] S201: Convert the model format of the model to be deployed into a model format suitable for a computing unit with limited computing resources to generate a target model.

[0105] For example, in a vehicle driving scenario, the computing unit with limited computing resources may be an MCU. Furthermore, the model format suitable for the MCU may be the ONNX format.

[0106] It should be understood that the model format suitable for the MCU can be manually configured and determined during the vehicle development phase based on the vehicle's usage scenario and computing power.

[0107] In practical applications, PyTorch's model conversion tools can be used to convert the model format of the model to be deployed. The torch.onnx function in this tool converts PyTorch models to the ONNX format, enabling the model to run on other platforms or frameworks that support ONNX. torch.save is used to save the model or tensor, storing the model's parameters, structure, and other information to disk for subsequent loading and use. To ensure efficient model operation on the vehicle, these tools are used to optimize the format of the model to be deployed to a model format suitable for MCU operation.

[0108] In one feasible approach, the model format of the model to be deployed is first converted into a model format suitable for computing units with limited computing resources to generate a converted model; then, for each network layer in the converted model, the operations in the network layer that meet the merging strategy are merged to generate a target model.

[0109] Among them, merging the operations in the network layer that meet the merging strategy means packaging the operations in the model that can be efficiently combined (that is, two or more operations are closely related in the model topology and logic, such as having a fixed order of precedence) to reduce loop overhead, memory usage and data redundancy during model execution.

[0110] For example, the linear activation layer in the self-attention mechanism is always followed by a residual connection (Add) and layer normalization (Norm) operations. In the model topology generated by torch.onnx, these are treated as four independent operations: matrix multiplication, Rectified Linear Unit (ReLU) function, residual addition, and sigmoid normalization. In existing models, the order of these operations is: matrix multiplication, ReLU, residual addition, and sigmoid normalization. This sequential computation incurs four loop cycles and requires at least four heap memory allocations and accesses, increasing memory usage. Because matrix multiplication and ReLU functions are sequential and dataflow-connected, they can be combined into a single operation: first performing the matrix multiplication, then applying the ReLU activation to the result in the same operation, without requiring separate computation steps. This step reduces the storage requirements for intermediate results and reduces memory access operations. Consequently, the residual addition operation and the normalization operation are closely related. The result of residual addition is directly used as the input of normalization, so the two can be optimized by combining them into the same operation step. During the merging process, the calculation of residual addition and normalization can share the same memory area, reducing additional memory allocation and repeated calculations in the loop.

[0111] S202. Encapsulate the functions and weights of the same type of network layers in the target model respectively, and generate a function array and a weight array corresponding to each network layer type.

[0112] It should be understood that by encapsulating functions and weights separately in arrays, the corresponding functions and weights can be directly indexed according to the layer type, thereby improving the efficiency of search and execution. Furthermore, encapsulating functions and weights can avoid redundant data storage and memory waste. For network layers of the same type, functions and weights of the same class can be shared, eliminating the need to store functions and weights separately for each network layer. Furthermore, if the calculation method or parameters of a layer need to be modified, it is only necessary to modify the function array or weight array, without having to reimplement the logic of the entire model.

[0113] S203: Determine the metadata of each network layer in the target model.

[0114] In a possible implementation, the weight pointer of each network layer can be determined according to the weight of the network layer, and the function pointer of the network layer can be determined according to the function of each network layer, thereby generating metadata for each network layer.

[0115] Optionally, after determining the function pointer and weight pointer of each network layer, the input data dimension, output data dimension and sequence length of the network layer can also be determined, and then the metadata of the network layer can be generated based on the input data dimension, output data dimension, sequence length, function pointer and weight pointer.

[0116] It can be understood that by determining the metadata of each network layer in the target model, the metadata can be directly read during the subsequent execution process to obtain the structure and calculation method of the corresponding network layer, avoiding repeated calculations and unnecessary searches during execution, so that the calculation process of each network layer can be carried out more efficiently and the execution speed can be improved.

[0117] S204: Generate a model structure array based on the metadata of each network layer.

[0118] In a feasible process, adjacent network layers of preset types in the target model are first merged into a target network layer; then, metadata of the target network layer is determined based on metadata of the adjacent network layers; finally, a model structure array is generated based on the metadata of each network layer.

[0119] Specifically, for some types of adjacent network layers, multiple adjacent network layers can be merged to obtain a new network layer, namely the target network layer. The metadata corresponding to the multiple adjacent network layers are then merged to generate the metadata corresponding to the target network layer, which is then combined with the metadata of other network layers to construct the model structure data.

[0120] In the process of merging metadata corresponding to multiple adjacent network layers, the function pointers and weight pointers of the multiple adjacent network layers may be retained, and the data input address pointers and data output address pointers of the multiple adjacent network layers may be merged. Specifically, for any network layer among the multiple adjacent network layers, if the data input address pointer of the network layer is the same as the data output address pointer of another network layer among the multiple adjacent network layers, then both the data input address pointer and the data output address pointer are deleted; otherwise, they are retained.

[0121] exist Figure 2 Based on the embodiment shown, S102 can be implemented through the following process:

[0122] First, if the current network layer is the target network layer, the functions corresponding to the multiple function pointers of the current network layer are merged to generate the target function; then, based on the relationship between the functions corresponding to the multiple function pointers of the current network layer, the weights corresponding to the multiple weight pointers corresponding to the current network layer are merged to generate the target weight; finally, the data in the current data input address is calculated according to the target function and the target weight to generate the calculation result.

[0123] It should be noted that since the operations of each network layer may be completed by multiple functions, such as matrix operations, addition, activation functions, etc., by merging the functions corresponding to multiple function pointers, it is possible to reduce the duplication of intermediate calculations and generate a simplified objective function. For example, there are two functions, function 1: ax + bz = y and function 2: 2y + z = q. These two functions can be merged to obtain function 3: 2ax + 2bz + z = q. Correspondingly, the weights corresponding to the multiple weight pointers corresponding to the network layer also need to be merged. For example, for variable x, the weight corresponding to variable x in function 1 is a. After the function is merged, the weight corresponding to variable x in function 3 is 2a.

[0124] It can be understood that using the combined objective function and target weights to calculate the input data avoids frequent memory access and repeated calculations, reduces computational complexity, and enables network layer operations to be completed in fewer steps, thereby improving the flexibility and scalability of the overall calculation.

[0125] Optionally, in some embodiments, the model structure array may be generated in the following manner.

[0126] First, the model format of the model to be deployed is converted into a model format suitable for computing units with limited computing resources. The target model is generated, and adjacent network layers of a preset type in the target model are merged into the target network layer. Next, the functions and weights of the network layers of the same type in the target model are encapsulated separately to generate a function array and weight array corresponding to each network layer type. Then, the metadata of each network layer in the target model is determined, and based on the metadata of each network layer, a model structure array is generated.

[0127] It should be understood that this embodiment is Figure 2 The difference between the illustrated embodiments is that this embodiment first merges adjacent network layers of a preset type into a target network layer. This target network layer type is treated as a separate network type, and the function array and weight array corresponding to each network layer type are generated. This way, the target network layer has a unique function pointer and weight pointer, which can be directly called during the model inference process, saving model inference time.

[0128] For example, assuming that the target network layer includes three network layers (network layer 1, network layer 2, network layer 3), the type of network layer 1 is type 1, the type of network layer 2 is type 2, and the type of network layer 3 is type 3, network layer 1 can be merged with network layer 2 to generate network layer 4. Figure 2 The embodiment shown will generate a function array and weight array corresponding to type 1, a function array and weight array of type 2, and a function array and weight array of type 3; while this embodiment will generate a function array and weight array of type 3, and a function array and weight array of type 4.

[0129] Figure 3 A schematic diagram of the structure of the model application device provided in the embodiment of the present application is shown as follows: Figure 3 As shown, the model application device 30 provided in this embodiment includes:

[0130] Determination module 301, for determining the current network layer corresponding to the current data input address based on the model structure array of the target model and the current data input address, the model structure array including metadata of each network layer in the target model, the metadata including a function pointer, a weight pointer, a data input address pointer, and a data output address pointer, the function pointer points to the corresponding function of the network layer in the function array, and the weight pointer points to the corresponding weight of the network layer in the weight array;

[0131] The calculation module 302 is used to calculate the data in the current data input address according to the function corresponding to the function pointer of the current network layer and the weight corresponding to the weight pointer to generate a calculation result;

[0132] Output module 303, used for storing the calculation result to the data output address corresponding to the data output address pointer of the current network layer;

[0133] The processing module 304 is used to determine the data output address as the new current data input address, repeat the above steps until the data output address is the model result output address, and determine the current calculation result as the model output result.

[0134] In one possible implementation, the model application device further includes a generation module, which, before determining the current network layer corresponding to the current data input address based on the model structure array of the target model and the current data input address, is configured to:

[0135] Convert the model format of the model to be deployed into a model format suitable for computing units with limited computing resources to generate a target model;

[0136] Encapsulate the functions and weights of the same type of network layers in the target model respectively, and generate function arrays and weight arrays corresponding to each network layer type;

[0137] Determine the metadata of each network layer in the target model;

[0138] Generate a model structure array based on the metadata of each network layer.

[0139] In one possible implementation, the model structure array further includes the input data dimension, output data dimension, and sequence length of each model layer.

[0140] In a possible implementation, the generation module is specifically configured to:

[0141] Merge adjacent network layers of preset types in the target model into the target network layer;

[0142] Determine the metadata of the target network layer based on the metadata of the adjacent network layer;

[0143] Generate a model structure array based on the metadata of each network layer.

[0144] In a possible implementation, the calculation module 302 is specifically configured to:

[0145] If the current network layer is the target network layer, the functions corresponding to multiple function pointers of the current network layer are merged to generate the target function;

[0146] Based on the relationship between the functions corresponding to the multiple function pointers of the current network layer, the weights corresponding to the multiple weight pointers corresponding to the current network layer are merged to generate the target weight;

[0147] Calculate the data in the current data input address according to the objective function and target weight to generate the calculation result.

[0148] In a possible implementation, the generation module is specifically configured to:

[0149] Convert the model format of the to-be-deployed model into a model format suitable for computing units with limited computing resources, and generate a converted model;

[0150] For each network layer in the converted model, the operations in the network layer that meet the merging strategy are merged to generate the target model.

[0151] In one possible implementation, the target model is a driving behavior prediction model, which is used to predict the driving behavior of at least one adjacent vehicle based on environmental timing information, the vehicle timing information of the vehicle itself, and the adjacent vehicle timing information during vehicle driving.

[0152] The model application device provided in the embodiment of the present application can execute the method provided in the above method embodiment. Its implementation principle and technical effects are similar, and will not be described in detail in this embodiment.

[0153] Figure 4 This is a schematic diagram of the structure of the vehicle provided in the embodiment of the present application. Figure 4 As shown, the vehicle 40 provided in this embodiment includes: a vehicle body 401, at least one processor 402, and a memory 403. Optionally, the vehicle 40 also includes a communication component 404. The processor 402, the memory 403, and the communication component 404 are connected via a bus 405.

[0154] In a specific implementation process, at least one processor 402 executes the computer-executable instructions stored in the memory 403, so that the at least one processor 402 performs the above method.

[0155] The specific implementation process of the processor 402 can be found in the above method embodiment. Its implementation principle and technical effects are similar and will not be repeated here in this embodiment.

[0156] In the above embodiments, it should be understood that the processor may be a central processing unit (CPU), other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), etc. A general-purpose processor may be a microprocessor or any conventional processor. The steps of the method disclosed in the present invention may be directly implemented by a hardware processor or implemented by a combination of hardware and software modules in the processor.

[0157] The memory may include a high-speed memory (Random Access Memory, RAM), and may also include a non-volatile memory (NVM), such as at least one disk memory.

[0158] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus. Buses can be classified into address buses, data buses, and control buses. For ease of illustration, the buses in the drawings of this application are not limited to just one bus or just one type of bus.

[0159] The present application also provides a computer program product, including a computer program, which implements the above method when executed by a processor.

[0160] The present application also provides a computer-readable storage medium, in which computer-executable instructions are stored. When a processor executes the computer-executable instructions, the above method is implemented.

[0161] The above-mentioned readable storage medium can be implemented by any type of volatile or non-volatile memory device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk. The readable storage medium can be any available medium that can be accessed by a general-purpose or special-purpose computer.

[0162] An exemplary readable storage medium is coupled to a processor so that the processor can read information from the readable storage medium and write information to the readable storage medium. Of course, the readable storage medium can also be an integral part of the processor. The processor and the readable storage medium can be located in an application specific integrated circuit (ASIC). Of course, the processor and the readable storage medium can also exist in the device as discrete components.

[0163] The division of units is merely a logical functional division; actual implementations may employ alternative divisions, such as combining or integrating multiple units or components into another system, or omitting or disabling certain features. Furthermore, any direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection between devices or units, either through an interface, electrical, mechanical, or other means.

[0164] Units described as separate components may or may not be physically separate, and components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0165] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0166] If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the various embodiments of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, and other media that can store program code.

[0167] Those skilled in the art will appreciate that all or part of the steps in the above-described method embodiments can be implemented using hardware associated with program instructions. The aforementioned program can be stored in a computer-readable storage medium. When executed, the program performs the steps of the above-described method embodiments. The aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.

[0168] Finally, it should be noted that those skilled in the art will readily identify other embodiments of the present invention after considering the specification and practicing the invention disclosed herein. The present invention is intended to cover any variations, uses, or adaptations of the present invention that follow the general principles of the present invention and include common knowledge or customary techniques in the art not disclosed herein. The present invention is not limited to the precise structure described above and illustrated in the accompanying drawings, and various modifications and variations may be made without departing from the scope thereof. The scope of the present invention is limited solely by the appended claims.

Claims

1. A model application method, characterized in that: include: Determine, according to a model structure array of a target model and a current data input address, a current network layer corresponding to the current data input address, wherein the model structure array includes metadata of each network layer in the target model, the metadata including a function pointer, a weight pointer, a data input address pointer, and a data output address pointer, the function pointer points to a function corresponding to the network layer in a function array, and the weight pointer points to a weight corresponding to the network layer in a weight array; Calculate the data in the current data input address according to the function corresponding to the function pointer of the current network layer and the weight corresponding to the weight pointer to generate a calculation result; Storing the calculation result to the data output address corresponding to the data output address pointer of the current network layer; The data output address is determined as a new current data input address, and the above steps are repeated until the data output address is the model result output address, and the current calculation result is determined as the model output result.

2. The method according to claim 1, characterized in that Before determining the current network layer corresponding to the current data input address based on the model structure array of the target model and the current data input address, the method further includes: Converting the model format of the to-be-deployed model into a model format suitable for a computing unit with limited computing resources to generate the target model; Encapsulate the functions and weights of the same type of network layers in the target model respectively, and generate function arrays and weight arrays corresponding to each network layer type; determining metadata for each network layer in the target model; Generate the model structure array according to the metadata of each network layer.

3. The method according to claim 1 or 2, characterized in that The model structure array also includes the input data dimension, output data dimension, and sequence length of each model layer.

4. The method according to claim 2, characterized in that The model structure array is generated according to the metadata of each network layer, including: Merging adjacent network layers of a preset type in the target model into a target network layer; Determine the metadata of the target network layer based on the metadata of the adjacent network layer; Generate the model structure array according to the metadata of each network layer.

5. The method according to claim 4, characterized in that The step of calculating the data in the current data input address according to the function corresponding to the function pointer of the current network layer and the weight corresponding to the weight pointer to generate a calculation result includes: If the current network layer is the target network layer, merging functions corresponding to multiple function pointers of the current network layer to generate a target function; Based on the relationship between the functions corresponding to the multiple function pointers of the current network layer, the weights corresponding to the multiple weight pointers corresponding to the current network layer are merged to generate a target weight; The data in the current data input address is calculated according to the objective function and the target weight to generate the calculation result.

6. The method according to any one of claims 2, 4 or 5, characterized in that The converting the model format of the to-be-deployed model into the model format of a computing unit suitable for limited computing resources to generate the target model includes: Converting the model format of the to-be-deployed model into a model format suitable for a computing unit with limited computing resources, thereby generating a converted model; For each network layer in the converted model, the operations in the network layer that meet the merging strategy are merged to generate the target model.

7. The method according to any one of claims 1, 2, 4 or 5, characterized in that The target model is a driving behavior prediction model, which is used to predict the driving behavior of at least one adjacent vehicle based on environmental time series information, the vehicle time series information of the vehicle itself, and the adjacent vehicle time series information during vehicle driving.

8. A model application device, characterized in that: include: a determination module, configured to determine, based on a model structure array of a target model and a current data input address, a current network layer corresponding to the current data input address, wherein the model structure array includes metadata of each network layer in the target model, the metadata including a function pointer, a weight pointer, a data input address pointer, and a data output address pointer, the function pointer pointing to a function corresponding to the network layer in the function array, and the weight pointer pointing to a weight corresponding to the network layer in the weight array; a calculation module, configured to calculate the data in the current data input address according to the function corresponding to the function pointer of the current network layer and the weight corresponding to the weight pointer, and generate a calculation result; An output module, configured to store the calculation result to a data output address corresponding to the data output address pointer of the current network layer; The processing module is used to determine the data output address as a new current data input address, repeat the above steps until the data output address is a model result output address, and determine the current calculation result as a model output result.

9. A vehicle, characterized in that: include: Vehicle body, memory, processor; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory, so that the processor performs the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer-executable instructions, which are used to implement the method according to any one of claims 1 to 7 when executed by a processor.

11. A computer program product, characterized in that The invention comprises a computer program, which implements the method according to any one of claims 1 to 7 when executed by a processor.

Images