A vehicle terminal model deployment method, device, equipment and medium

Abstract

The present application relates to the technical field of automobile intelligent cockpit, and provides a vehicle terminal model deployment method, device, equipment and medium, the method comprises the following steps: obtaining a target model and a test data set; according to the data format required by the preselected model inference framework, the target model is format converted to obtain the format converted target model; the format converted target model is transmitted to the model inference framework in the vehicle terminal, and the format converted target model is run; the test data set is input into the model inference framework, the model inference is carried out on the target model according to the test data set, and the inference result is obtained; the inference result is transmitted to the target function module to realize the deployment of the target model in the vehicle terminal. The method can directly deploy the target algorithm to the vehicle terminal, without the need to increase a separate AI chip, so that the automobile enterprises can master the simple algorithm deployment capability with strong deployability.

Description

Technical Field

[0001] This application relates to the field of automotive intelligent cockpit technology, specifically to a method, device, equipment, and medium for deploying an in-vehicle terminal model. Background Technology

[0002] With the development of AI (Artificial Intelligence) vision technology, the functions enabled by vision technology are constantly expanding, and its application in the field of automotive intelligent cockpit technology is becoming increasingly widespread. Most AI calculations are performed in the cloud, but cloud computing has problems such as network latency, inability to use in offline environments, untimely response, and inability to guarantee data privacy.

[0003] To address the aforementioned challenges in cloud-based AI computing, automakers currently rely heavily on supplier-provided AI chips and algorithms for AI computation in their in-vehicle mobile terminals. However, when automakers want to add new features, they must depend on suppliers for development and integration, increasing costs. The key issues that need to be addressed are how to deploy target algorithms to in-vehicle terminals without relying on suppliers, and how to reduce algorithm deployment costs. Summary of the Invention

[0004] In view of the shortcomings of the prior art described above, the purpose of this application is to provide a method, apparatus, device and medium for deploying in-vehicle algorithms, so as to solve the problem of how to deploy target algorithms to in-vehicle terminals without relying on suppliers and reduce algorithm deployment costs in the prior art.

[0005] To achieve the above and other related objectives, this application provides a method for deploying a vehicle-mounted terminal model, the method comprising:

[0006] Obtain the target model and test dataset;

[0007] Based on the data format required by the pre-selected model inference framework, the target model is converted to a new format to obtain the converted target model.

[0008] The target model after format conversion is transmitted to the model inference framework in the vehicle terminal and the target model after format conversion is run.

[0009] The test dataset is input into the model inference framework, and the target model is inferred based on the test dataset to obtain the inference result;

[0010] The inference results are transmitted to the target functional module to enable the deployment of the target model on the vehicle terminal.

[0011] In one embodiment of this application, the feature is that:

[0012] The target model is in torch format;

[0013] The data format required for the pre-selected model inference framework is tflite format.

[0014] In one embodiment of this application, transmitting the format-converted target model to the model inference framework within the vehicle terminal and running the format-converted target model includes:

[0015] The target model after the format conversion is transmitted to the system of the vehicle terminal, and the system of the vehicle terminal is pre-configured with the tensorflow-lite framework package;

[0016] The target model in the system is transferred to the model inference framework;

[0017] According to the tensorflow initialization interface in the tensorflow-lite framework package, obtain and respond to the tensorflow initialization command to initialize the parameters of the target model and run the target model through the model inference framework.

[0018] In one embodiment of this application, the test dataset includes an image test dataset. Before inputting the test dataset into the model inference framework and performing model inference on the target model based on the test dataset to obtain the inference result, the method further includes a step of preprocessing the image test dataset on the in-vehicle terminal. The preprocessing step includes:

[0019] The images in the image test dataset are scaled according to the preset image size to obtain the first image test dataset;

[0020] Multiple images are obtained from the first image test dataset. The multiple images are randomly cropped and stitched together. The cropped and stitched images are then added to the first image test dataset to obtain the second image test dataset.

[0021] Obtain multiple images and their corresponding labels from the second image test dataset. Mix the multiple images and labels according to a preset mixing ratio. Add the mixed images and labels to the second image test dataset to obtain the third image test dataset.

[0022] In one embodiment of this application, after obtaining the third image test dataset, the method further includes:

[0023] The first data format and the second data format of the third image test dataset are obtained. The first data format is the data format of the third image test dataset obtained after preprocessing on the vehicle terminal. The second data format is the data format of the third image test dataset obtained after preprocessing on the computer. The first data format and the second data format are in matrix form.

[0024] According to the matrix conversion method pre-configured in the computer terminal, the matrix order of the second data format is converted to the matrix order of the first data format to obtain the second data format after matrix order conversion;

[0025] The first data format is compared with the second data format after the matrix sequence conversion to obtain a comparison result. If the error between the first data format and the second data format after the matrix sequence conversion is lower than a preset error threshold, then the third image test dataset after preprocessing on the vehicle terminal is determined to be usable.

[0026] In one embodiment of this application, the step of inputting the test dataset into the model inference framework, performing model inference on the target model based on the test dataset, and obtaining the inference result includes:

[0027] The test dataset is input into the model inference framework so that the target model can call the test dataset;

[0028] The first inference result is obtained by calculating the data in the test dataset based on the target model;

[0029] Based on the preset data element correspondence, the data elements in the first reasoning result are mapped to the target data element to obtain the second reasoning result.

[0030] In one embodiment of this application, the inference result includes multiple image element categories and multiple image element indicator boxes corresponding to the multiple image element categories. The step of transmitting the inference result to the target functional module to achieve deployment of the target model on the vehicle terminal further includes a post-processing step on the inference result. The post-processing step includes:

[0031] Scores are calculated for the plurality of image element indicator boxes to obtain the scores of the plurality of image element indicator boxes;

[0032] Based on the scores of the multiple image element indicator boxes, determine the optimal image element indicator box with the highest score;

[0033] Calculate the intersection-union ratio (IUGR) between the remaining image element indicator boxes and the optimal image element indicator box. If the IUGR is greater than a preset IUGR threshold, remove the redundant indicator boxes in the remaining image element indicator boxes to obtain multiple candidate image element indicator boxes corresponding to the multiple image element categories.

[0034] If there is a combination relationship between the multiple image element categories, the center distance and corner distance between the multiple candidate image element indicator boxes corresponding to the multiple image elements are calculated, and it is determined whether the multiple candidate image element indicator boxes are nested based on the center distance and corner distance.

[0035] If the multiple candidate image element indicator boxes are nested, the output time of the multiple image element categories and multiple candidate image element indicator boxes is reduced according to the timing adjustment value to obtain the post-processed inference result.

[0036] In one embodiment of this application, a vehicle-mounted terminal model deployment device is also provided, the device comprising:

[0037] The information acquisition module is used to acquire the target model and the test dataset;

[0038] The format conversion module is used to convert the target model according to the data format required by the pre-selected model inference framework to obtain the format-converted target model.

[0039] The model execution module is used to transmit the target model after format conversion to the model inference framework in the vehicle terminal and run the target model after format conversion.

[0040] The model inference module is used to input the test dataset into the model inference framework, perform model inference on the target model based on the test dataset, and obtain the inference result;

[0041] The inference result transmission module is used to transmit the inference result to the target functional module to realize the deployment of the target model on the vehicle terminal.

[0042] In one embodiment of this application, an electronic device is also provided, the electronic device comprising:

[0043] One or more processors;

[0044] A storage device for storing one or more programs, which, when executed by one or more processors, enable the electronic device to implement the vehicle terminal model deployment method as described above.

[0045] In one embodiment of this application, a computer-readable storage medium is also provided, on which a computer program is stored, which, when executed by a computer's processor, causes the computer to perform the vehicle terminal model deployment method as described above.

[0046] The beneficial effects of this invention are as follows: First, a target model and a test dataset are acquired; then, according to the data format required by the pre-selected model inference framework, the target model is format-converted to obtain a format-converted target model; next, the format-converted target model is transmitted to the model inference framework within the in-vehicle terminal, and the format-converted target model is run; then, the test dataset is input into the model inference framework, and model inference is performed on the target model based on the test dataset to obtain the inference result; finally, the inference result is transmitted to the target functional module to realize the deployment of the target model on the in-vehicle terminal. This invention, by format-converting the target model, performing inference on the format-converted target model through the model inference framework within the in-vehicle terminal, and transmitting the inference result to the target functional module to realize the target function, allows the target algorithm to be directly deployed to the in-vehicle terminal without the need for a separate AI chip, enabling automakers to master the deployment capabilities of algorithms that are simple and highly deployable.

[0047] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and do not limit this application. Attached Figure Description

[0048] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application. It is obvious that the drawings described below are merely some embodiments of this application, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort. In the drawings:

[0049] Figure 1 This is a schematic diagram illustrating the implementation environment of a vehicle terminal model deployment method, as shown in an exemplary embodiment of this application.

[0050] Figure 2 This is an exemplary embodiment of the present application illustrating the deployment structure of an AI algorithm in an Android vehicle infotainment system;

[0051] Figure 3 This is a flowchart illustrating an exemplary embodiment of the method for deploying a vehicle-mounted terminal model according to this application;

[0052] Figure 4 This is a flowchart illustrating a method for deploying a vehicle-mounted terminal model, as shown in another exemplary embodiment of this application;

[0053] Figure 5 This is a block diagram illustrating an in-vehicle terminal model deployment device according to an exemplary embodiment of this application;

[0054] Figure 6 A schematic diagram of the structure of a computer system used to implement the embodiments of the present application is shown. Detailed Implementation

[0055] The embodiments of the present invention will be described below with reference to the accompanying drawings and preferred embodiments. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be understood that the preferred embodiments are only for illustrating the present invention and not for limiting the scope of protection of the present invention.

[0056] It should be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of the present invention. Therefore, the drawings only show the components related to the present invention and are not drawn according to the actual number, shape and size of the components in the actual implementation. In the actual implementation, the form, quantity and proportion of each component can be arbitrarily changed, and the layout of the components may also be more complex.

[0057] In the following description, numerous details are explored to provide a more thorough explanation of embodiments of the invention. However, it will be apparent to those skilled in the art that embodiments of the invention may be practiced without these specific details. In other embodiments, well-known structures and devices are shown in block diagram form rather than in detail to avoid obscuring embodiments of the invention.

[0058] In this application, "multiple" refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. The character " / " generally indicates that the preceding and following related objects have an "or" relationship.

[0059] First, it should be noted that:

[0060] Preprocessing: The input to the model's inference is Tensor data (mostly matrix data), but the input for normal AI applications is images, videos, text, etc. Therefore, preprocessing is needed to transform the input images, videos, text, etc., into Tensor data that the model can accept for inference. Image data preprocessing may include, for example, image format conversion, color space transformation, image transformation (resize, warpaffline (affine transformation)), and image filtering.

[0061] Model inference: In real-world applications, devices with deployed algorithm models run the actual input data through the trained model, achieving the expected performance and accuracy. During model inference, the inference engine, capable of efficiently executing operators in the deep learning network, converts intermediate representations into specific file formats and runs the model efficiently on the corresponding hardware platform.

[0062] Post-processing: The Tensor data after model inference is converted into feature data that can be recognized by the business. Different businesses will present different final effect data. That is, the inference results obtained from model inference are transmitted to the target functional module to realize the target function.

[0063] Figure 1 This is a schematic diagram illustrating the implementation environment of a vehicle-mounted terminal model deployment method, as shown in an exemplary embodiment of this application. (Refer to...) Figure 1 As shown, the implementation environment may include vehicle-mounted terminal device 101, cloud device 102, and server group 103.

[0064] Schematic illustration: Terminal device 101 is used to acquire the target model and test dataset, and deploys the target model using the vehicle-mounted terminal model deployment method in this embodiment. Cloud device 102 and server group 103 are used to train the pre-built target model to obtain the trained target model, and then transmit the trained target model and the pre-stored test dataset to terminal device 101 via a transmission medium. Additionally, terminal device 101 can also be connected to a camera mounted on the vehicle to acquire image data collected during vehicle operation.

[0065] Alternatively, as an illustration, relevant technical personnel can input the trained target model and test dataset into the terminal device 101. The terminal device 101 then deploys the target model using the vehicle-mounted terminal model deployment method described in this embodiment. After the model is deployed, the target function module in the terminal device 101 processes the result data inferred by the model and transmits the result data to the cloud device 102 and server group 103 via a transmission medium for storage or subsequent calculation.

[0066] It should be noted that, in this embodiment of the application, the source of the target model and the test dataset, and the data interaction method between the terminal device 101 and the cloud device 102 and the server group 103 are not specifically limited, and can be set according to specific actual needs.

[0067] in, Figure 1 The terminal device 101 shown may be an in-vehicle computer, but is not limited to this. Figure 1The server group 103 shown can be, for example, a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms. No restrictions are placed on this. The terminal device 101 can communicate with the cloud device 102 and the server group 103 via wireless networks such as 3G (third-generation mobile information technology), 4G (fourth-generation mobile information technology), and 5G (fifth-generation mobile information technology). No restrictions are placed on this as well.

[0068] Figure 2 This is an exemplary embodiment of the present application illustrating the deployment structure of an AI algorithm in an Android vehicle infotainment system. The AI ​​algorithm is deployed within an Android system, which runs on an onboard chip within the vehicle's infotainment system. Camera data can be transmitted to the Android system via the onboard chip, enabling the AI ​​algorithm to recognize images and thus complete the deployment of the AI ​​algorithm.

[0069] It's important to note that automakers purchase controllers, AI chips, and other components from multiple suppliers before assembling vehicles. For example, if a car needs to monitor driver fatigue, it requires in-vehicle cameras to capture real-time images of the driver, which are then transmitted to an AI chip for recognition. However, when automakers need to add new functionality to the image recognition model within this AI chip, they must contact suppliers to redevelop and integrate the AI ​​chip and model. This not only increases costs but also extends the vehicle assembly cycle.

[0070] The problems mentioned above are common in the deployment of general AI algorithm models. To solve these problems, embodiments of this application propose a vehicle-mounted terminal model deployment method, a vehicle-mounted terminal model deployment device, an electronic device, a computer-readable storage medium, and a computer program product. These embodiments will be described in detail below.

[0071] Please see Figure 3 , Figure 3 This is a flowchart illustrating a method for deploying a vehicle-mounted terminal model, as shown in an exemplary embodiment of this application. This method can be applied to... Figure 1 The implementation environment shown can also be applied to Figure 2 The vehicle-mounted system deployment structure shown is composed of... Figure 1 The terminal device 101 in the implementation environment shown is specifically executed.

[0072] like Figure 3 As shown, in an exemplary embodiment, the vehicle-mounted terminal model deployment method includes at least steps S310-S350, which are described in detail below:

[0073] In step S310, the target model and test dataset are obtained.

[0074] For example, an AI chip equipped with an Android system obtains the target model and test dataset from a server, where the target model has already been trained. The target model can be pre-selected by the algorithm developers. For instance, in this embodiment of the application, object detection needs to be performed on the image data collected by the camera. The algorithm developers can pre-select YOLOv5 (a series of object detection architectures and models pre-trained on a dataset), a model with high performance and adaptability in the field of computer vision-object detection in 2021, as the target model.

[0075] In step S320, the target model is converted according to the data format required by the pre-selected model inference framework to obtain the converted target model.

[0076] For example, the target model is in torch format, and the data format required by the pre-selected model inference framework is tflite format. In this embodiment, tflite is chosen because Android can call the tflite model through relevant functions, and the relevant library functions are more mature than other frameworks; therefore, conversion to the tflite model is chosen.

[0077] In step S330, the format-converted target model is transmitted to the model inference framework in the vehicle terminal, and the format-converted target model is run.

[0078] For example, the target model in tflite format is transmitted to the TensorRT model inference framework in the vehicle terminal, and the target model in tflite format is run through the TensorRT model inference framework.

[0079] In step S340, the test dataset is input into the model inference framework, and the target model is inferred based on the test dataset to obtain the inference result.

[0080] For example, the test dataset consists of image data collected by a camera mounted on a vehicle. The image data is input into the TensorRT model inference framework for model inference, and the model inference result is obtained. The model inference result can be, for example, "Category: Red".

[0081] In step S350, the inference results are transmitted to the target functional module to enable the deployment of the target model on the vehicle terminal.

[0082] For example, the target functional module is one that determines driving based on traffic light colors. In this case, the reasoning result could be, for example, the traffic light colors. By associating the reasoning result with the target functional module, the corresponding function is triggered when preset conditions are met, thus completing the development of the entire function.

[0083] Therefore, through the above steps S310 to S350, by converting the target model into a format, inferring the converted target model through the model inference framework in the vehicle terminal, and transmitting the inference result to the target functional module to realize the target function, this method can directly deploy the target algorithm to the vehicle terminal without adding a separate AI chip, allowing car companies to master the algorithm deployment capability of simple algorithms with strong deployability.

[0084] In one embodiment of this application, in Figure 3 Step 330, as shown, involves transmitting the format-converted target model to the model inference framework within the vehicle terminal and running the format-converted target model, including the following steps:

[0085] The target model after the format conversion is transmitted to the system of the vehicle terminal, and the system of the vehicle terminal is pre-configured with the tensorflow-lite framework package;

[0086] In response to a model transfer instruction, the target model in the system is transferred to the model inference framework;

[0087] According to the tensorflow initialization interface in the tensorflow-lite framework package, obtain and respond to the tensorflow initialization command to initialize the parameters of the target model and run the target model through the model inference framework.

[0088] For example, firstly, the target model in tflite format is stored in the system directory, the Android project is configured with the tensorflow-lite framework dependency, the target model in tflite format is read and loaded as a readable object; then, in a sub-thread, the initialization interface of tensorflow is called to respond to the tensorflow initialization command, so as to initialize the parameters of the target model and run the target model through the model inference framework.

[0089] In one embodiment of this application, the test dataset includes an image test dataset, in Figure 3Before step 340 shown, a preprocessing step is also included on the in-vehicle terminal for the image test dataset. The preprocessing step includes:

[0090] The images in the image test dataset are scaled according to the preset image size to obtain the first image test dataset;

[0091] Multiple images are obtained from the first image test dataset. The multiple images are randomly cropped and stitched together. The cropped and stitched images are then added to the first image test dataset to obtain the second image test dataset.

[0092] Obtain multiple images and their corresponding labels from the second image test dataset. Mix the multiple images and labels according to a preset mixing ratio. Add the mixed images and labels to the second image test dataset to obtain the third image test dataset.

[0093] For example, the test dataset includes an image test dataset, which can be obtained through the following steps: Open the camera using the native Android camera interface, and pass in different IDs according to different needs to control the opening of different cameras; when the camera is opened, the camera data of each frame can be obtained by listening to the camera preview data. Since the camera has more than 25 frames per second, the image processing time of each frame cannot exceed 40 milliseconds, otherwise the camera will freeze. When the image processing time exceeds the calculation time (1000 milliseconds / frame), the image frames need to be filtered, and the time-consuming operation needs to be put into a sub-thread for processing, so as to obtain the image test dataset.

[0094] After obtaining the image test dataset, the image data in the image dataset is compressed or expanded using the resize algorithm (an image scaling algorithm). Then, multiple image data in the image test dataset are randomly cropped and stitched using Mosaic data augmentation. Finally, multiple image data in the image test dataset and their corresponding labels are mixed using MixUp data augmentation to obtain the preprocessed image data.

[0095] In one embodiment of this application, the preprocessing step of the image test dataset further includes:

[0096] The first data format and the second data format of the third image test dataset are obtained. The first data format is the data format of the third image test dataset obtained after preprocessing on the vehicle terminal. The second data format is the data format of the third image test dataset obtained after preprocessing on the computer. The first data format and the second data format are in matrix form.

[0097] According to the matrix conversion method pre-configured in the computer terminal, the matrix order of the second data format is converted to the matrix order of the first data format to obtain the second data format after matrix order conversion;

[0098] The first data format is compared with the second data format after the matrix sequence conversion to obtain a comparison result. If the error between the first data format and the second data format after the matrix sequence conversion is lower than a preset error threshold, then the third image test dataset after preprocessing on the vehicle terminal is determined to be usable.

[0099] It should be noted that the data format received by each model may be different. The data obtained from the camera cannot be directly used by the model. In this case, the camera data needs to be converted into data that the model can recognize. Therefore, the image test dataset in this application embodiment can also be an image test dataset that has been converted into a data format.

[0100] Inconsistencies may arise during data preprocessing. In such cases, a test image can be used first, followed by a detailed comparison of the processed data. Inconsistencies will lead to errors in subsequent inferences. For example, the data format of the test dataset obtained by algorithm engineers from image preprocessing on the computer can be compared with that obtained from image preprocessing on the vehicle's infotainment system. This comparison determines whether the two data formats are identical or whether the error between them is within 1%. It should be noted that both data formats are in matrix form. If the two data formats are identical, the preprocessed test dataset is considered usable. Furthermore, because some image preprocessing functions differ between the PARC environment on the computer and the Android environment on the vehicle (e.g., the order of image matrices in PARC is NCWH, while in Android it is NHWC), integration personnel need to convert the data format from the PARC environment to the Android environment before comparing the two data formats.

[0101] In one embodiment of this application, in Figure 3 In step 340, the test dataset is input into the model inference framework, and model inference is performed on the target model based on the test dataset to obtain the inference result, including:

[0102] The test dataset is input into the model inference framework so that the target model can call the test dataset;

[0103] The first inference result is obtained by calculating the data in the test dataset based on the target model;

[0104] Based on the preset data element correspondence, the data elements in the first reasoning result are mapped to the target data element to obtain the second reasoning result.

[0105] For example, the image test dataset is input into the model inference framework, and the model inference method is called to complete the model inference. In this embodiment, model inference refers to inputting any pre-processed image into a series of matrix parameters of the model for multiplication to obtain specific data. This data is then mapped logically to obtain the final result, such as "color category is red". The inference process is time-consuming and needs to be processed in a sub-thread. After obtaining the initial inference result, the inference result is format-converted according to the data format required by the target functional module.

[0106] It should be noted that the output of each model may differ, requiring coordination between integration personnel and algorithm developers. Algorithm developers should compare the results on the computer with those running on the Android in-vehicle system. If inconsistencies are found, first check for differences in the data preprocessing. Minor deviations are likely normal, as the different environments of a computer and an in-vehicle system can lead to slight discrepancies in the results.

[0107] In one embodiment of this application, the inference result includes multiple image element categories and multiple image element indicator boxes corresponding to the multiple image element categories. Figure 3 The step S350 shown above transmits the inference result to the target functional module to enable deployment of the target model on the vehicle terminal. It also includes a post-processing step for the inference result, which includes:

[0108] Scores are calculated for the plurality of image element indicator boxes to obtain the scores of the plurality of image element indicator boxes;

[0109] Based on the scores of the multiple image element indicator boxes, determine the optimal image element indicator box with the highest score;

[0110] Calculate the intersection-union ratio (IUGR) between the remaining image element indicator boxes and the optimal image element indicator box. If the IUGR is greater than a preset IUGR threshold, remove the redundant indicator boxes in the remaining image element indicator boxes to obtain multiple candidate image element indicator boxes corresponding to the multiple image element categories.

[0111] If there is a combination relationship between the multiple image element categories, the center distance and corner distance between the multiple candidate image element indicator boxes corresponding to the multiple image elements are calculated, and it is determined whether the multiple candidate image element indicator boxes are nested based on the center distance and corner distance.

[0112] If the multiple candidate image element indicator boxes are nested, the output time of the multiple image element categories and multiple candidate image element indicator boxes is reduced according to the timing adjustment value to obtain the post-processed inference result.

[0113] For example, the model output may not be directly usable data; a series of post-processing steps are required to obtain the desired data. The inference result includes multiple image element categories and multiple image element bounding boxes corresponding to those categories. However, in a real image, a single category A may predict multiple image element bounding boxes. Using NMS (Non-Maximum Suppression), redundant bounding boxes are removed from the multiple bounding boxes based on a calculated threshold between them, avoiding mutual interference and ensuring the accuracy and effectiveness of the inference result.

[0114] Furthermore, after some traffic lights are identified as individual categories, some categories may have combination relationships. For example, when making driving decisions based on the number of traffic light images, it is necessary to identify the collected traffic signal images in the model. There is a combination relationship between countdown timers and red / green traffic lights; that is, the driving decision requires summing the countdown image recognition result with the traffic light image timing result. The center distance and corner distance between the multiple candidate image element indicator boxes corresponding to the multiple image elements are calculated, and the nesting of the multiple candidate image element indicator boxes is determined based on the center distance and corner distance. When multiple candidate image element indicator boxes are nested, for example, indicator box A is inside indicator box B, then the multiple candidate image element indicator boxes corresponding to the multiple image elements with combination relationships can be output.

[0115] Because the computing power of the vehicle terminal is very limited, and the time required for preprocessing, model inference, and postprocessing is approximately 1 second, such a delay in outputting traffic light times is fatal. Therefore, it is necessary to reduce the output time of image element categories and image element indicator boxes through timing adjustment values. For example, if the model output time is 10 seconds, after timing adjustment, the output time can be reduced to 9 seconds.

[0116] The following describes an embodiment of the apparatus described in this application, which can be used to execute the vehicle-mounted terminal model deployment method described in the above embodiments of this application. For details not disclosed in the apparatus embodiments of this application, please refer to the embodiments of the vehicle-mounted terminal model deployment method described above.

[0117] Figure 5 This is a block diagram illustrating a vehicle-mounted terminal model deployment device according to an exemplary embodiment of this application. The device can be applied to… Figure 3The implementation environment shown is specifically configured in terminal device 101. This device can also be applied to other exemplary implementation environments and specifically configured in other devices. This embodiment does not limit the implementation environment to which the device is applicable.

[0118] like Figure 5 As shown, the exemplary vehicle-mounted terminal model deployment device includes: an information acquisition module 501, a format conversion module 502, a model running module 503, a model inference module 504, and an inference result transmission module 505.

[0119] Among them, the information acquisition module 501 is used to acquire the target model and the test dataset;

[0120] The format conversion module 502 is used to convert the target model according to the data format required by the pre-selected model inference framework to obtain the format-converted target model.

[0121] The model running module 503 is used to transmit the target model after format conversion to the model inference framework in the vehicle terminal and run the target model after format conversion.

[0122] The model inference module 504 is used to input the test dataset into the model inference framework, perform model inference on the target model based on the test dataset, and obtain the inference result;

[0123] The inference result transmission module 505 is used to transmit the inference result to the target functional module to realize the deployment of the target model on the vehicle terminal.

[0124] In one embodiment of the application, according to the foregoing, the format conversion module 502 is further configured as follows:

[0125] The target model after the format conversion is transmitted to the system of the vehicle terminal, and the system of the vehicle terminal is pre-configured with the tensorflow-lite framework package;

[0126] The target model in the system is transferred to the model inference framework;

[0127] According to the tensorflow initialization interface in the tensorflow-lite framework package, obtain and respond to the tensorflow initialization command to initialize the parameters of the target model and run the target model through the model inference framework.

[0128] In one embodiment of the application, according to the foregoing, the vehicle-mounted terminal model deployment device further includes a pre-processing module, which is configured to:

[0129] The images in the image test dataset are scaled according to the preset image size to obtain the first image test dataset;

[0130] Multiple images are obtained from the first image test dataset. The multiple images are randomly cropped and stitched together. The cropped and stitched images are then added to the first image test dataset to obtain the second image test dataset.

[0131] Obtain multiple images and their corresponding labels from the second image test dataset. Mix the multiple images and labels according to a preset mixing ratio. Add the mixed images and labels to the second image test dataset to obtain the third image test dataset.

[0132] In one embodiment of the application, based on the foregoing, the preprocessing module is further configured as follows:

[0133] The first data format and the second data format of the third image test dataset are obtained. The first data format is the data format of the third image test dataset obtained after preprocessing on the vehicle terminal. The second data format is the data format of the third image test dataset obtained after preprocessing on the computer. The first data format and the second data format are in matrix form.

[0134] According to the matrix conversion method pre-configured in the computer terminal, the matrix order of the second data format is converted to the matrix order of the first data format to obtain the second data format after matrix order conversion;

[0135] The first data format is compared with the second data format after the matrix sequence conversion to obtain a comparison result. If the error between the first data format and the second data format after the matrix sequence conversion is lower than a preset error threshold, then the third image test dataset after preprocessing on the vehicle terminal is determined to be usable.

[0136] In one embodiment of the application, based on the foregoing, the model inference module 504 is configured as follows:

[0137] The test dataset is input into the model inference framework so that the target model can call the test dataset;

[0138] The first inference result is obtained by calculating the data in the test dataset based on the target model;

[0139] Based on the preset data element correspondence, the data elements in the first reasoning result are mapped to the target data element to obtain the second reasoning result.

[0140] In one embodiment of the application, according to the foregoing, the vehicle-mounted terminal model deployment device further includes a post-processing module, which is configured to:

[0141] Scores are calculated for the plurality of image element indicator boxes to obtain the scores of the plurality of image element indicator boxes;

[0142] Based on the scores of the multiple image element indicator boxes, determine the optimal image element indicator box with the highest score;

[0143] Calculate the intersection-union ratio (IUGR) between the remaining image element indicator boxes and the optimal image element indicator box. If the IUGR is greater than a preset IUGR threshold, remove the redundant indicator boxes in the remaining image element indicator boxes to obtain multiple candidate image element indicator boxes corresponding to the multiple image element categories.

[0144] If there is a combination relationship between the multiple image element categories, the center distance and corner distance between the multiple candidate image element indicator boxes corresponding to the multiple image elements are calculated, and it is determined whether the multiple candidate image element indicator boxes are nested based on the center distance and corner distance.

[0145] If the multiple candidate image element indicator boxes are not nested, the output time of the multiple image element categories and multiple candidate image element indicator boxes is reduced according to the timing adjustment value to obtain the post-processed inference result.

[0146] It should be noted that the vehicle-mounted terminal model deployment device and the vehicle-mounted terminal model deployment method provided in the above embodiments belong to the same concept. The specific ways in which each module and unit performs operations have been described in detail in the method embodiments, and will not be repeated here. In practical applications, the vehicle-mounted terminal model deployment device provided in the above embodiments can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above, and this is not a limitation here.

[0147] Embodiments of this application also provide an electronic device, including: one or more processors; and a storage device for storing one or more programs, which, when executed by the one or more processors, cause the electronic device to implement the vehicle terminal model deployment method as described above.

[0148] Figure 6 A schematic diagram of the structure of a computer system used to implement the embodiments of this application is shown. It should be noted that... Figure 6 The computer system 600 of the electronic device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of this application.

[0149] like Figure 6 As shown, the computer system 600 includes a Central Processing Unit (CPU) 601, which can perform various appropriate actions and processes based on programs stored in Read-Only Memory (ROM) 602 or programs loaded from Storage Unit 608 into Random Access Memory (RAM) 603, such as performing the methods described in the above embodiments. The RAM 603 also stores various programs and data required for system operation. The CPU 601, ROM 602, and RAM 603 are interconnected via a bus 604. An Input / Output (I / O) interface 605 is also connected to the bus 604.

[0150] The following components are connected to I / O interface 605: an input section 606 including a keyboard, mouse, etc.; an output section 607 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and speakers, etc.; a storage section 608 including a hard disk, etc.; and a communication section 609 including a network interface card such as a LAN (Local Area Network) card, modem, etc. The communication section 609 performs communication processing via a network such as the Internet. A drive 610 is also connected to I / O interface 605 as needed. A removable medium 611, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on drive 610 as needed so that computer programs read from it can be installed into storage section 608 as needed.

[0151] Specifically, according to embodiments of this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program including a computer program for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication section 609, and / or installed from removable medium 611. When the computer program is executed by central processing unit (CPU) 601, it performs various functions defined in the system of this application.

[0152] It should be noted that the computer-readable medium shown in the embodiments of this application can be a computer-readable signal medium or a computer-readable storage medium, or any combination of the two. A computer-readable storage medium can be, for example, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), flash memory, optical fiber, portable compact disc read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this application, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying a computer-readable computer program. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media can also be any computer-readable medium other than computer-readable storage media, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The computer program contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to wireless, wired, etc., or any suitable combination thereof.

[0153] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. Each block in a flowchart or block diagram may represent a module, segment, or portion of code, which contains one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0154] The units described in the embodiments of this application can be implemented in software or hardware, and the described units can also be located in a processor. The names of these units do not necessarily limit the specific unit itself.

[0155] Another aspect of this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a computer's processor, causes the computer to perform the vehicle terminal model deployment method as described above.

[0156] It should be noted that although several modules or units for the device used to perform actions have been mentioned in the detailed description above, this division is not mandatory. In fact, according to the embodiments of this application, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided and embodied by multiple modules or units.

[0157] Another aspect of this application provides a computer program product or computer program including computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the vehicle terminal model deployment method provided in the various embodiments described above.

[0158] Through the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solutions according to the embodiments of this application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) or on a network, including several instructions to cause a computing device (such as a personal computer, server, touch terminal, or network device, etc.) to execute the method according to the embodiments of this application.

[0159] In the above embodiments, unless otherwise specified, the use of serial numbers such as "first" and "second" to describe common objects only indicates that they refer to different instances of the same object, rather than indicating that the objects being described must be in a given order, whether temporally, spatially, sequentially, or in any other way.

[0160] In the above embodiments, references to "this embodiment," "an embodiment," "another embodiment," or "other embodiments" in the specification indicate that a specific feature, structure, or characteristic described in connection with the embodiments is included in at least some embodiments, but not necessarily all embodiments. Multiple appearances of "this embodiment," "an embodiment," or "another embodiment" do not necessarily all refer to the same embodiment. If the specification describes a component, feature, structure, or characteristic as "may," "may," or "can" be included, then that specific component, feature, structure, or characteristic is not necessarily required to be included. If the specification or claims refer to an element "a," it does not mean that there is only one element. If the specification or claims refer to "an additional" element, it does not exclude the existence of more than one additional element.

[0161] In the above embodiments, although the invention has been described in conjunction with specific embodiments thereof, many substitutions, modifications, and variations of these embodiments will be apparent to those skilled in the art from the foregoing description. For example, other memory structures (e.g., dynamic RAM (DRAM)) may be used with the embodiments discussed. The embodiments of the invention are intended to cover all such substitutions, modifications, and variations falling within the broad scope of the appended claims.

[0162] The above embodiments are merely illustrative of the principles and effects of the present invention and are not intended to limit the invention. Any person skilled in the art can modify or alter the above embodiments without departing from the spirit and scope of the present invention. Therefore, all equivalent modifications or alterations made by those skilled in the art without departing from the spirit and technical concept disclosed in the present invention should still be covered by the claims of the present invention.

Claims

1. A method for deploying a vehicle terminal model, characterized in that, The method comprises: obtaining a target model and a test data set, the test data set comprising an image test data set; format-converting the target model according to a data format required by a preselected model inference framework, to obtain a format-converted target model; transmitting the format-converted target model to a model inference framework in a vehicle terminal and running the format-converted target model; inputting the test data set into the model inference framework, performing model inference on the target model according to the test data set, and obtaining an inference result; transmitting the inference result to a target function module to realize deployment of the target model on the vehicle terminal; Before the step of inputting the test data set into the model inference framework, performing model inference on the target model according to the test data set, and obtaining an inference result, the method further comprises a step of pre-processing the image test data set on the vehicle terminal to obtain a third image test data set, wherein the pre-processing comprises scaling, cropping, splicing, and label mixing; After obtaining the third image test data set, the method further comprises the following steps: obtaining a first data format and a second data format of the third image test data set, the first data format being a data format of the third image test data set obtained after pre-processing on the vehicle terminal, the second data format being a data format of the third image test data set obtained after pre-processing on a computer terminal, the first data format and the second data format being in matrix form; converting the matrix order of the second data format to the matrix order of the first data format according to a preconfigured matrix conversion mode in the computer terminal, to obtain a matrix order-converted second data format; comparing the first data format with the matrix order-converted second data format to obtain a comparison result, and determining that the third image test data set pre-processed on the vehicle terminal is available if the comparison result indicates that the error between the first data format and the matrix order-converted second data format is lower than a preset error threshold.

2. The vehicle terminal model deployment method according to claim 1, wherein: the format of the target model is a torch format; the data format required by the preselected model inference framework is a tflite format. 3.The in-vehicle terminal model deployment method of claim 1, wherein, The step of transmitting the format-converted target model to a model inference framework in a vehicle terminal and running the format-converted target model comprises: transmitting the format-converted target model to a system of the vehicle terminal, wherein a tensorflow-lite framework package is preconfigured in the system of the vehicle terminal; transmitting the target model in the system to the model inference framework; obtaining and responding to a tensorflow initialization instruction according to a tensorflow initialization interface in the tensorflow-lite framework package, to initialize parameters of the target model and run the target model through the model inference framework. 4.The method of claim 1, wherein, The pre-processing step comprises: scaling images in the image test data set according to a preset image size, to obtain a first image test data set; obtaining a plurality of images in the first image test data set, respectively performing random clipping and splicing on the plurality of images, and adding the images obtained after clipping and splicing to the first image test data set, to obtain a second image test data set; obtaining a plurality of images in the second image test data set and labels corresponding to the plurality of images, mixing the plurality of images and the labels according to a preset mixing ratio, and adding the mixed images and labels to the second image test data set, to obtain a third image test data set. 5.The in-vehicle terminal model deployment method of claim 1, wherein, The method further includes the following steps of: inputting the test data set into the model inference framework, so that the target model calls the test data set; calculating data in the test data set according to the target model, to obtain a first inference result; mapping data elements in the first inference result to target data elements according to a preset data element correspondence relationship, to obtain a second inference result. 6.The in-vehicle terminal model deployment method of claim 1, wherein, The inference result includes a plurality of image element categories and a plurality of image element bounding boxes corresponding to the plurality of image element categories, and the inference result is transmitted to a target function module, so that the target model is deployed on the vehicle terminal. The method further includes the following steps of post-processing the inference result: performing score calculation on the plurality of image element bounding boxes, to obtain scores of the plurality of image element bounding boxes; determining an optimal image element bounding box with the highest score according to the scores of the plurality of image element bounding boxes; calculating intersection over union (IoU) between the optimal image element bounding box and remaining image element bounding boxes, respectively, and removing redundant bounding boxes in the remaining image element bounding boxes if the IoU is greater than a preset IoU threshold, to obtain a plurality of candidate image element bounding boxes corresponding to the plurality of image element categories; if there is a combination relationship between the plurality of image element categories, calculating center distance and edge angle distance between the plurality of candidate image element bounding boxes corresponding to the plurality of image elements, and judging whether the plurality of candidate image element bounding boxes are nested according to the center distance and the edge angle distance; 7. A vehicle terminal model deployment apparatus characterized by comprising: if the plurality of candidate image element bounding boxes are nested, reducing output time of the plurality of image element categories and the plurality of candidate image element bounding boxes according to a timing adjustment value, to obtain a post-processed inference result. The device includes: an information acquisition module configured to acquire a target model and a test data set, the test data set including an image test data set; a format conversion module configured to convert a format of the target model according to a data format required by a preselected model inference framework, to obtain a format-converted target model; a model running module configured to transmit the format-converted target model to a model inference framework in a vehicle terminal, and run the format-converted target model. The model inference module is configured to input the test data set into the model inference framework, perform model inference on the target model according to the test data set, and obtain an inference result. The inference result transmission module is configured to transmit the inference result to a target function module to realize deployment of the target model on the vehicle terminal. The preprocessing module is configured to preprocess the image test data set on the vehicle terminal to obtain a third image test data set. The preprocessing includes scaling, cropping, splicing, and label mixing. The preprocessing module is further configured to obtain a first data format and a second data format of the third image test data set. The first data format is a data format of the third image test data set after preprocessing on the vehicle terminal, and the second data format is a data format of the third image test data set after preprocessing on a computer terminal. The first data format and the second data format are in a matrix form. The matrix order of the second data format is converted into the matrix order of the first data format according to a preconfigured matrix conversion manner in the computer terminal, to obtain a second data format after matrix order conversion. A comparison result is obtained by comparing the first data format and the second data format after matrix order conversion. If the comparison result is that the error between the first data format and the second data format after matrix order conversion is lower than a preset error threshold, it is determined that the third image test data set after preprocessing on the vehicle terminal is available.

8. An electronic device, comprising: The electronic device includes: one or more processors; a storage device configured to store one or more programs that, when executed by the one or more processors, cause the electronic device to implement the vehicle terminal model deployment method of any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, A computer program is stored thereon, and when the computer program is executed by a processor of a computer, the computer executes the vehicle terminal model deployment method of any one of claims 1 to 6.

Images