Network model training and search method, device and storage medium

Abstract

The embodiments of the present application provide a method, device, and storage medium for training and searching a network model. In the embodiments of the present application, a knowledge distillation framework with a block design is proposed. While ensuring the size of the model, it can be guided by a reference neural network model and combined with automatic search technology to fully tap the learning ability of the network model, so that the searched target neural network model can fully learn the prior information of the reference neural network model, guide the search for a high-performance, lightweight network structure, and improve the performance of the searched target neural network model.

Description

Technical Field

[0001] The present application relates to the field of deep learning technology, and in particular to a method, device, and storage medium for training and searching a network model. Background Art

[0002] In recent years, deep learning has achieved significant breakthroughs and progress in fields such as image processing, speech recognition, and natural language processing. This is due to the fact that neural networks generated using deep learning possess far more robust feature expression capabilities than traditional machine learning methods. Consequently, to exploit these neural network features, academia and industry have designed numerous complex network structures. These complex network structures require significant computing and memory resources and typically run on powerful servers.

[0003] Terminal devices connect remotely to servers via the internet, which provide deep learning-based services. However, these services can be slow and require a robust network environment. This necessitates migrating neural network models to terminal devices. However, terminal devices have limited computing and memory resources. To enable the practical application of complex network structures, lightweight model approaches have emerged. These aim to design, compress, and quantize small models to enable accelerated neural network models on terminal devices.

[0004] One approach to designing a small model is to manually design a small student model and then transfer the knowledge of the teacher model to it, so that it achieves performance close to that of the teacher model. However, different student models have different learning capabilities and can learn different prior information about the teacher model. Manually designed student models often cannot exhaust all structures and therefore cannot obtain optimal prior information. Summary of the Invention

[0005] Various aspects of the present application provide a network model training and search method, device, and storage medium to improve the performance of lightweight network models.

[0006] An embodiment of the present application provides a network model training method, comprising: receiving model requirement description information, wherein the model requirement description information includes a specified reference neural network model and a specified target neural network model model size; dividing the reference neural network model into N reference modules, and generating N target modules corresponding to the N reference modules to obtain an initial neural network model, where N is an integer greater than or equal to 2; inputting a sample image into the reference neural network model and the initial neural network model for processing respectively to obtain output results of the N reference modules and the N target modules; using the model size as a constraint condition, based on the output results of the N reference modules and the N target modules, jointly training the N target modules until a set stop condition is met to obtain a target neural network model.

[0007] An embodiment of the present application also provides a network model search method, including: dividing a given reference neural network model into N reference modules, generating N target modules corresponding to the N reference modules to obtain an initial neural network model, where N is an integer greater than or equal to 2; inputting a sample image into the reference neural network model and the initial neural network model for processing respectively to obtain output results of the N reference modules and the N target modules; with a specified model size as a constraint condition, based on the output results of the N reference modules and the N target modules, jointly training the N target modules until a set stop condition is met to obtain a target neural network model.

[0008] An embodiment of the present application also provides a server-side device, comprising: a memory and a processor; the memory is used to store a computer program; the processor is coupled to the memory and is used to execute the computer program, so as to: receive model requirement description information, wherein the model requirement description information includes a specified reference neural network model and a specified target neural network model model size; divide the reference neural network model into N reference modules, and generate N target modules corresponding to the N reference modules to obtain an initial neural network model, where N is an integer greater than or equal to 2; input the sample image into the reference neural network model and the initial neural network model respectively for processing to obtain the output results of the N reference modules and the N target modules; with the model size as a constraint condition, the N target modules are jointly trained based on the output results of the N reference modules and the N target modules until the set stop condition is met to obtain the target neural network model.

[0009] An embodiment of the present application also provides a server-side device, comprising: a memory and a processor; the memory is used to store a computer program; the processor is coupled to the memory, and is used to execute the computer program, so as to: divide a given reference neural network model into N reference modules, and generate N target modules corresponding to the N reference modules to obtain an initial neural network model, where N is an integer greater than or equal to 2; input the sample image into the reference neural network model and the initial neural network model respectively for processing to obtain the output results of the N reference modules and the N target modules; with a specified model size as a constraint, the N target modules are jointly trained based on the output results of the N reference modules and the N target modules until the set stop condition is met to obtain the target neural network model.

[0010] An embodiment of the present application further provides a computer storage medium storing a computer program, which, when executed by a processor, causes the processor to implement the steps of the method embodiment of the present application.

[0011] An embodiment of the present application further provides a computer program product, including a computer program / instruction. When the computer program / instruction is executed by a processor, the processor is caused to implement the steps in the method embodiment of the present application.

[0012] In an embodiment of the present application, a knowledge distillation framework with a block design is proposed. While ensuring the model size, it can be guided by a reference neural network model and combined with automatic search technology to fully tap the learning ability of the network model, so that the searched target neural network model can fully learn the prior information of the reference neural network model, and guide the search for a high-performance, lightweight network structure, thereby improving the performance of the searched target neural network model. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0014] Figure 1a A flowchart of a network model search method is provided for an exemplary embodiment of the present application;

[0015] Figure 1b A schematic diagram of the principle framework of the network model search method provided by the exemplary embodiment of the present application;

[0016] Figure 1c A schematic diagram of a state in which the weighted sum of the outputs of the previous target module and the previous reference module is used as the input of the next target module during the network model search process provided by an exemplary embodiment of the present application;

[0017] Figure 1d A schematic diagram of the weighting factor p attenuating over time provided by an exemplary embodiment of the present application;

[0018] Figure 2a A schematic diagram of the structure of a network model service system provided by an exemplary embodiment of the present application;

[0019] Figure 2b A flowchart of a network model training method provided by an exemplary embodiment of the present application;

[0020] Figure 3 A schematic diagram of the structure of a server device provided in an exemplary embodiment of the present application. DETAILED DESCRIPTION

[0021] To make the purpose, technical solutions, and advantages of this application more clear, the technical solutions of this application will be clearly and completely described below in conjunction with the specific embodiments of this application and the corresponding drawings. Obviously, the embodiments described are only part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0022] With the application of deep learning technology, it is necessary to deploy neural network models locally on terminal devices. On the one hand, it can improve the efficiency of terminal devices in obtaining model services, and on the other hand, it can also reduce the processing burden on the server side. However, due to the limited computing resources and memory resources of terminal devices, it is necessary to design a lightweight model for the terminal device that is adapted to its various resources. In order to design lightweight models, the embodiment of the present application proposes a knowledge distillation framework with a block design, which can be guided by the reference neural network model while ensuring the size of the model, and combined with automatic search technology to fully explore the learning ability of the network model, so that the searched target neural network model can fully learn the prior information of the reference neural network model, and guide the search for a high-performance, lightweight network structure, thereby improving the performance of the searched target neural network model.

[0023] The following describes in detail the technical solutions provided by various embodiments of the present application in conjunction with the accompanying drawings.

[0024] Figure 1a The following is a flow chart of a network model search method provided as an exemplary embodiment of the present application. Figure 1a As shown, the method includes:

[0025] 101. Divide the given reference neural network model into N reference modules, and generate N target modules corresponding to the N reference modules to obtain an initial neural network model, where N is an integer greater than or equal to 2;

[0026] 102. Input the sample image into the reference neural network model and the initial neural network model for processing, so as to obtain output results of N reference modules and N target modules;

[0027] 103. With the specified model size as the constraint condition, based on the output results of N reference modules and N target modules, the N target modules are jointly trained until the set stopping condition is met to obtain the target neural network model.

[0028] In this embodiment, a block-based knowledge distillation framework is proposed. This framework is combined with automatic search technology. Using a reference neural network model as a guide, it fully exploits the learning capabilities of different network models, automatically searching for a lightweight neural network model that meets the requirements while controlling model size. In this embodiment, the lightweight neural network model ultimately automatically found is referred to as the target neural network model. The reference neural network model is a teacher network with a more complex network structure and richer knowledge, while the target neural network is a student network with a simpler network structure that has learned the prior knowledge of the reference neural network model. Although the target neural network model has a simpler structure, it can achieve the same or similar performance as the reference neural network model because it has learned the prior knowledge of the reference neural network model. Furthermore, due to its lightweight nature, the target neural network model has lower requirements for computing and memory resources and can run on terminal devices such as mobile phones, laptops, desktops, tablets, robots, autonomous vehicles, and drones. Of course, the target neural network model can also run on servers with more abundant computing and memory resources.

[0029] Depending on the application scenario, the functions implemented by a given reference neural network model will also vary, and accordingly, the functions of the target neural network model will also vary. For example, in the field of speech recognition, the given reference neural network model may be a speech recognition model based on a relatively complex neural network structure and relatively rich knowledge; accordingly, the target neural network model may be a lightweight speech recognition model. For another example, in the field of image processing, the given reference neural network model may be a target detection model based on a relatively complex neural network structure and relatively rich knowledge; accordingly, the target neural network model may be a lightweight target detection model. For another example, in the field of natural language processing, the given reference neural network model may be a language translation model based on a relatively complex neural network structure and relatively rich knowledge; accordingly, the target neural network model may be a lightweight language translation model.

[0030] After a reference neural network model is given, the reference neural network model can be divided into N reference modules, where N is an integer greater than or equal to 2, such as Figure 1b As shown, N reference modules are represented as T0, T1, ..., T N-1 Each reference module is a part of the reference neural network model, which can be one or more layers in the reference neural network model; N reference modules are connected together to form a complete reference neural network model, that is, the reference neural network model T = T0 + T1 + ... + T N-1 . In this embodiment, the implementation method of dividing the reference neural network model into N reference modules is not limited. In an optional embodiment, the number N of reference modules can be predetermined, and then the reference neural network model can be divided into N reference modules by random division. Alternatively, in another optional embodiment, the resolution size supported by the reference neural network model can be obtained from the model parameters of the reference neural network model; according to the resolution size supported by the reference neural network model, the reference neural network model is divided into N reference modules, wherein the N reference modules have different resolutions. Preferably, the number N is the number of resolution sizes supported by the reference neural network model.

[0031] In the knowledge distillation framework of block design, N target modules are generated for N reference modules, such as Figure 1b As shown, N target modules are represented as S0, S1, ..., S N-1 Among them, the target module can also be called a super module, and N target modules (or super modules) together constitute the initial neural network model, that is, the initial neural network model S = S0 + S1 + ... + S N-1 . Among them, the initial neural network model is the initial structure of the target neural network model.

[0032] For any reference module, the process of generating a corresponding target module for it includes two aspects: one is generating the search space of the target module, and the other is generating the initial operator of the target module. In this embodiment, the search space of the target module includes the maximum number of network layers M that the target module can support, the layer coefficient of each layer, and the operators and operator coefficients that each layer can support; wherein, each layer can support multiple operators, and each operator corresponds to its own operator coefficient. Among them, for the same target module, the number of operators supported by each layer and the specific operators can be the same, or they can be different, or they can be completely different. It should be noted that for each of the N target modules, each has its own search space, and the search spaces of different target modules can be the same or different. For different target modules, the maximum number of network layers they support can be the same or different.

[0033] For each target module, after determining its search space, the initial operator of the target module can be generated based on the maximum number of network layers in the search space, the layer coefficient of each layer, and the operators and operator coefficients that each layer can support. Among them, the initial operator is the operator used by the target module at the beginning. The operator is a mixed operation of the operators of each layer contained in the target module and is related to the network structure of the target module. Among them, the number of network layers contained in the target module at the beginning and the operators used in each layer can be pre-selected from its search space and can be flexibly selected according to application requirements. In an optional embodiment, assuming that the target module initially includes all the network layers in its search space and each layer contains all the operators that can be supported by the layer, the initial operator of the target module can be expressed as the following formula (1).

[0034]

[0035] In formula (1), represents the initial operator of the kth target module, d i is the layer coefficient of the i-th layer, D is the maximum number of network layers of the target module, and w j is the operator coefficient of the jth operator in the i-th layer, P is the maximum number of operators in the i-th layer, F j (x) is the jth operator in the i-th layer. Optionally, F j (x) can be, but is not limited to, a filter operator. As the training process progresses, the network structure of the target module will continue to change, i.e., the number of network layers included in the target module and the operators used in each layer may change. Upon completion of the training, the final network structure of the target module and the target operator adapted to the final network structure are obtained.

[0036] After obtaining the initial neural network model S, the model training process can be started. The model training process includes: inputting the sample image into the reference neural network model T and the initial neural network model S for processing. Figure 1b As shown in the figure, in the process of using the reference neural network model T to process the sample image, the sample image is sent to the first reference module T0 as input data, and the output of the reference module T0 is used as the input of the reference module T1, and so on until the reference module T N-1 Output the result until the reference module T N-1 The output result of is also the output result of the reference neural network model T. In this embodiment, N reference modules T0, T1, ..., T N-1 The output result of . Further, Figure 1bAs shown in the figure, in the process of using the initial neural network model S to process the sample image, the sample image is sent to the first target module S0 as input data, and the output of the target module S0 is used as the input of the target module S1, and so on until the target module S N-1 Output the result until the target module S N-1 The output result of is also the output result of the initial neural network model S. In this embodiment, N target modules S0, S1, ..., S N-1 The output result is:

[0037] Furthermore, the model training process also includes: taking the specified model size as a constraint, based on the output results of N reference modules and N target modules, jointly training the N target modules until the set stop condition is met to obtain the target neural network model. The specified model size is used to limit the size of the trained target neural network model. The model size can be measured by the number of parameters of the model. Generally speaking, the fewer the number of parameters of the model, the simpler the model structure, and the more lightweight it is. Alternatively, the model size can also be measured by the delay of the deployment end. The deployment end refers to the end where the target neural network model needs to be deployed, such as a mobile phone, a robot, or an autonomous driving vehicle; the delay of the deployment end refers to the expected time from sending an image into the target neural network model to the output of the model, assuming that the target neural network model is deployed. The shorter the time, the better the model performance.

[0038] In this embodiment, model training is performed using a specified model size as a constraint, resulting in a target neural network model of the required size. Furthermore, joint training of the N target modules based on the outputs of the N reference modules and the N target modules is a "module"-based model training approach. By allowing each target module to learn as much as possible from the prior knowledge of its corresponding reference module, the prior knowledge of the entire reference neural network model can ultimately be transferred to the target neural network model, resulting in a lightweight network model with performance approaching that of the reference neural network model.

[0039] Among them, the process of jointly training N target modules includes: for each target module, in its corresponding search space, searching the number of network layers included in the target module and the operators used in each layer, until the loss function meets the set stopping condition, and obtaining the target operator of the target module; wherein, the target operators of the N target modules constitute the target neural network model. Optionally, in the process of searching for each target module, a gradient descent-based method can be used to calculate the layer coefficients and operator coefficients in each target module, and then determine the number of network layers included in each target module and the operators in each layer based on the layer coefficients and operator coefficients. Of course, in addition to using the gradient descent method to solve, it can also be solved by reinforcement learning and other methods, which are not limited to this.

[0040] In the embodiments of the present application, the stopping condition of model training is not limited. In an optional embodiment, the stopping condition may be the maximum number of training times. When the number of model training times reaches the maximum number of training times, the model training ends, and the target neural network model can be obtained at this time. In another optional embodiment, the stopping condition may be the longest training time. When the model training time reaches the longest training time, the model training ends, and the target neural network model can be obtained at this time. In another optional embodiment, a loss function for model training, or a cost function, can be defined. The set stopping condition may be a condition that the loss function should meet, such as the loss function is minimum, or the loss function is less than a set threshold, or the loss function is within a set interval, etc. Based on this, a process for jointly training N target modules includes: obtaining the loss function of the initial neural network model based on the output results of N reference modules and N target modules, combined with a specified model size; searching the N target modules in the search space corresponding to each of the N target modules until the loss function meets the set stopping condition, and obtaining the target neural network model. Among them, the search process is completed only after searching all N target modules. Each time the search process is completed, the loss function needs to be recalculated and judged whether the loss function meets the set stopping condition until the loss function meets the set stopping condition.

[0041] Furthermore, in order to fully learn the capabilities of the reference neural network model while constraining the model size, in an optional embodiment, the loss function for model training is defined to include two parts: a model performance part and a model size part. Among them, the model performance part further includes a model loss part and a supervision loss part of the reference neural network model. The model loss part represents the difference between the performance of the neural network model trained each time and the overall performance of the reference neural network model; the supervision loss part represents the loss in the process of the reference neural network model supervising the initial neural network model, which can be measured by the performance difference between N target modules and N reference modules. In this optional embodiment, the proportion of the model performance part and the model size part in the loss function is not limited. Weight coefficients can be set for the model performance part and the model size part according to application requirements, and the weight coefficients are used to represent the proportion of the model performance part and the model size part in the loss function. Based on this, a loss function can be expressed as the following formula (2):

[0042]

[0043] In formula (2), L loss Represents the loss function of model training; α is the weight coefficient of the model loss part, β is the weight coefficient of the model size part, L T_loss Represents the loss part of the model, Represents the supervision loss part; Model_Size represents the model size. In formula (2), the supervision loss part is represented by the weighted sum of the performance differences between N target modules and N reference modules, but is not limited to this. Among them, L F_loss represents the performance difference between the i-th target module and its corresponding reference model, u i is the supervision coefficient of the i-th reference module. Optionally, in order to learn the ability of the reference neural network model, in this optional embodiment, the supervision coefficient is required to be positively correlated with the output accuracy of the reference neural network model.

[0044] Based on the definition of the above loss function, such as Figure 1b As shown, when obtaining the loss function of the initial neural network model, the first performance loss function, that is, the loss part of the model dimension mentioned above, can be obtained based on the output results of the reference neural network model and the initial neural network model; the second performance loss function, that is, the supervision loss part mentioned above, can be obtained based on the output results of N reference modules and N target modules; the first performance loss function, the second performance loss function and the specified model size are weighted to obtain the loss function of the initial neural network model.

[0045] Furthermore, in some optional embodiments of the present application, in order to better learn the prior knowledge of the reference module, it is proposed to supervise the search process of the target module through a spatial attention mechanism. The spatial attention mechanism can effectively compress the prior knowledge of multiple reference modules into the search process of the target module, thereby improving the learning accuracy. Among them, the spatial attention mechanism is reflected in the calculation process of the second performance loss function, such as Figure 1b As shown in the figure, the output results of N reference modules and N target modules are respectively sent to the corresponding spatial attention mechanism modules; for each group of reference modules and target modules, the performance similarity of the two modules is calculated based on the output results of the two modules combined with the spatial attention mechanism; then, based on the performance similarity of the N target modules and their corresponding reference modules, the performance loss function of each of the N target modules can be obtained; finally, the performance loss functions of the N target modules are weighted to obtain the second performance loss function.

[0046] Among them, a process for calculating the performance similarity of N target modules and their corresponding reference modules based on the output results of N reference modules and N target modules, combined with a spatial attention mechanism, includes: for each target module, first, counting the feature quantities of the feature graphs output by the target module and its corresponding reference module, wherein the feature quantities include but are not limited to: maximum value distribution, mean distribution, etc. in the feature graph; then, extracting the feature quantities of the target module and its corresponding reference module to obtain an extraction result; optionally, a set of feature filters can be used to extract the feature quantities of the target module and its corresponding reference module, and the feature filters can be learnable convolution filters or manually designed operators; then, calculating the similarity of the above extraction results as the performance similarity between the target module and its corresponding reference module. Optionally, the similarity between the above extraction results can be calculated using a similarity operator, which includes but is not limited to: Euclidean distance operator, cosine similarity operator, etc. Among them, the Euclidean distance operator is used to calculate the Euclidean distance between the above extraction results; the cosine similarity operator is used to calculate the cosine similarity between the above extraction results.

[0047] Furthermore, in some optional embodiments of the present application, in order to accelerate the convergence speed of the target module, for any non-first target module among the N target modules, the weighted sum of the output of its previous target module and the output of the corresponding previous reference module can be used as the input of the non-first target module. Figure 1b In , for any non-first target module, the solid line input to the non-first target module represents the output result of its previous target module, and the dotted line input to the non-first target module represents the output result of the previous reference module corresponding to the previous target module. Figure 1cAs shown, for any non-first target module, the output results of its previous target module and the corresponding previous reference module are weighted and summed, and the weighted summation result is used as the input of the non-first target module to obtain the output result of the non-first target module. The output result of the non-first target module and the output result of its corresponding reference module are weighted and summed and then used as the input of the next target module, and so on. Figure 1c In , p represents the weight coefficient corresponding to the previous reference module, and 1-p represents the weight coefficient corresponding to the previous target module. The weighted sum of the output results of the previous target module and the corresponding previous reference module can be expressed as Among them, x t represents the input of the t-th target module; represents the output result of the t-1th target module, Represents the output result of the t-1th reference module; t is an integer greater than or equal to 1.

[0048] In an optional embodiment, the above-mentioned p is a fixed value, and the weight coefficients of the previous reference module and the previous target module remain unchanged. In another optional embodiment, p is a weighting factor that decays with time. This weighting factor can ensure that in the early stage of the search, the weight coefficient of the previous reference module is relatively large, which is conducive to accelerating convergence and fusion; however, as time goes by, the weight coefficient of the previous reference module gradually decreases, while the weight coefficient of the previous target module gradually increases, that is, in the later stage of the search, the weight coefficient of the previous target module is relatively large, which is conducive to accelerating the self-adjustment of the target module. Among them, one way in which p decays with time is as follows Figure 1d As shown, but not limited to.

[0049] In an embodiment of the present application, a knowledge distillation framework with block design is proposed, which can make full use of the structural characteristics of the reference neural network model, so that the target neural network model can "take its essence and discard its dross", and guide the search for a high-performance, lightweight target neural network model; in addition, the embodiment of the present application integrates automatic search technology, which can automatically search for the target neural network model. Compared with the manual design method, it can fully learn the structural information and knowledge information of the reference neural network model, and can further improve the performance of the target neural network model while ensuring the model size.

[0050] Furthermore, in some optional embodiments of the present application, a spatial attention mechanism is innovatively employed to supervise the search process of the target module, effectively compressing the knowledge of multiple reference modules into the search process of the target module. Furthermore, in some optional implementations of the present application, the output of the previous reference module is innovatively deeply fused with the output of the previous target module as the input of the next target module. This input method can effectively accelerate the convergence of the target module, which is conducive to improving the search efficiency and performance of the target module.

[0051] In addition to providing a network model search method, the embodiment of the present application can also provide a network model service for neural network model demanders, which can provide a lightweight neural network model for neural network model demanders. For neural network model demanders, there is no need to implement the lightweight neural network model by themselves, which can free the neural network model demanders from the implementation of the lightweight network model, allowing the neural network model demanders to focus more on the functions and applications of the neural network model. Figure 2a As shown in FIG, it is a structural diagram of a network model service system provided by an embodiment of the present application. Figure 2a As shown, the service system 20 includes: a server device 21 and a client 22. The server device 21 and the client 22 are connected by wire or wirelessly. Preferably, the server device 21 and the client 22 are connected via the Internet.

[0052] In terms of physical implementation, the server device 21 can be a conventional server, a cloud server, or a server array, etc.; accordingly, the client 22 can be a terminal device such as a laptop, a desktop computer, or a tablet computer, but is not limited thereto. The server device 21 is used to provide a network model lightweight service for the client 22, that is, it can provide the required lightweight neural network model for the client 22 according to its needs. The client 22 refers to the end that requires a lightweight neural network model, which can be simply referred to as the neural network model demander, and can use the network model lightweight service provided by the server device 21. Among them, the neural network model demander can be any party that needs to develop applications based on the neural network model.

[0053] For example, in an autonomous driving scenario, some researchers need to develop an in-vehicle navigation function based on a neural network model (used to plan navigation paths for autonomous driving vehicles), or need to develop an in-vehicle voice recognition function based on a neural network model (used for voice interaction with users), or need to develop a target detection function based on a neural network model (used to detect pedestrians or other vehicles on the path, etc., for avoidance), or need to develop a target trajectory prediction function based on a neural network model (used to predict the trajectory of pedestrians or other vehicles on the path, providing conditions for avoidance or path planning). Because the computing and memory resources of autonomous driving vehicles are limited, the neural network models required by these researchers need to be lightweight. In this case, researchers in the autonomous driving scenario can act as neural network model demanders and use the network model lightweight service provided by the server device 21 to directly obtain lightweight in-vehicle navigation models, in-vehicle voice recognition models, in-vehicle target detection models, or in-vehicle target trajectory prediction models.

[0054] It should be noted that these lightweight network models vary depending on the type of autonomous vehicle. For example, logistics vehicles, public service vehicles, medical service vehicles, and terminal service vehicles involve different functions, and different network models are required to implement different functions. The following examples illustrate the network models that implement different functions for these four types of autonomous vehicles:

[0055] Logistics vehicles refer to vehicles used in logistics scenarios, such as those with automatic sorting, refrigeration, and measurement capabilities. These vehicles have different functions, each requiring different network models.

[0056] For example, logistics vehicles can be equipped with automated sorting devices that automatically remove, transport, sort, and store goods upon arrival. This involves a network model for cargo sorting, which primarily implements logical control for image acquisition, cargo removal, transportation, sorting, and storage.

[0057] For another example, for cold chain logistics scenarios, logistics vehicles can also be equipped with refrigeration and insulation devices, which can refrigerate or insulate the transported fruits, vegetables, aquatic products, frozen foods, and other perishable foods, so that they are in a suitable temperature environment, solving the problem of long-distance transportation of perishable foods. This involves a network model for refrigeration and insulation control, which is mainly used to dynamically and adaptively calculate the appropriate temperature for cold meals or insulation based on the collected images, combined with information such as the nature of the food (or item), perishability, transportation time, current season, and climate, and automatically adjust the refrigeration and insulation device according to the appropriate temperature. In this way, when the vehicle transports different foods or items, the transportation personnel do not need to manually adjust the temperature, freeing the transportation personnel from the tedious temperature control and improving the efficiency of refrigerated and insulated transportation.

[0058] For example, in most logistics scenarios, charges are based on the volume and / or weight of the package. However, the number of logistics packages is very large. Simply relying on couriers to measure the volume and / or weight of the packages is very inefficient and has high labor costs. Therefore, in some logistics vehicles, measuring devices are added to automatically measure the volume and / or weight of logistics packages and calculate the fees for logistics packages. This involves a network model for logistics package measurement. The algorithm module is mainly used to combine images to identify the type of logistics package and determine the measurement method of the logistics package, such as volume measurement, weight measurement, or a combination of volume and weight measurement. It can also complete the volume and / or weight measurement according to the determined measurement method, and complete the fee calculation based on the measurement results.

[0059] Public service vehicles refer to vehicles that provide certain public services, such as fire trucks, de-icing trucks, water trucks, snowplows, garbage disposal vehicles, and traffic control vehicles. These public service vehicles perform different functions and therefore require different network models.

[0060] For example, the main task of an autonomous fire truck is to carry out reasonable firefighting tasks at the fire scene. This involves a network model for firefighting tasks. The network model must at least combine images to realize the logic of fire condition recognition, firefighting plan planning, and automatic control of firefighting equipment.

[0061] For example, the main task of a de-icing vehicle is to clear ice and snow from the road surface, which involves a de-icing network model. This network model must at least combine images to identify the ice and snow conditions on the road surface, formulate a de-icing plan based on the ice and snow conditions, such as which sections of the road require de-icing and which sections do not, whether to use salting and the amount of salt to spread, etc., as well as the logic of automatic control of the de-icing device when the de-icing plan is determined.

[0062] Among them, medical service vehicles refer to self-driving vehicles that can provide one or more medical services. Such vehicles can provide medical services such as disinfection, temperature measurement, medication, and isolation. This involves network models that provide various self-service medical services. These network models mainly combine images to realize the identification of disinfection needs and the control of disinfection devices, so that the disinfection devices can disinfect patients, or identify the patient's position and control the temperature measuring device to automatically approach the patient's forehead and other positions to measure the patient's temperature, or, it is used to realize the diagnosis of the disease, give a prescription based on the judgment result, and realize the identification of drugs / drug containers, as well as the control of the drug-taking robot to make it grab drugs for patients according to the prescription, etc.

[0063] Among them, terminal service vehicles refer to self-service autonomous driving vehicles that can replace some terminal devices to provide certain convenient services to users. For example, these vehicles can provide users with printing, attendance, scanning, unlocking, payment, retail and other services.

[0064] For example, in some application scenarios, users often need to go to a specific location to print or scan documents, which is time-consuming and labor-intensive. Therefore, a terminal service vehicle that can provide printing / scanning services to users has emerged. These service vehicles can be interconnected with user terminal devices. Users can issue printing instructions through the terminal devices, and the service vehicle responds to the printing instructions, automatically prints the documents required by the user, and can automatically deliver the printed documents to the user's location. Users do not need to queue up at the printer, which can greatly improve printing efficiency. Alternatively, the service vehicle can respond to the scanning instructions issued by the user through the terminal device and move to the user's location. The user places the document to be scanned on the scanning tool of the service vehicle to complete the scanning, without having to queue up at the printer / scanner, saving time and effort. This involves a network model for providing printing / scanning services, which at least needs to identify the interconnection with the user's terminal device, the response to the printing / scanning instructions, the positioning of the user's location, and the travel control.

[0065] For example, with the development of new retail applications, more and more e-commerce companies are using self-service vending machines to deliver goods to office buildings and public areas. However, these vending machines are fixed in place and cannot be moved, requiring users to visit the machine to purchase the items they need, which is still inconvenient. Therefore, self-driving vehicles that provide retail services have emerged. These service vehicles can carry goods and move automatically. They also provide corresponding self-service shopping apps or shopping portals. Users can use mobile devices or other terminals to place orders with the self-driving vehicles through the app or shopping portal. The order includes the name of the product to be purchased, the quantity, and the user's location. After receiving the order request, the vehicle can determine whether the product is available and whether the quantity is sufficient. If the product is available and the quantity is sufficient, it can automatically move to the user's location with the product and deliver it to the user, further improving shopping convenience for users, saving time and allowing them to spend more time on more important things. This involves the network model for providing retail services. These network models mainly implement logic such as responding to user order requests, order processing, maintaining product information, locating the user, and managing payments.

[0066] For another example, in scenarios such as banks, shopping malls, or office buildings, access control terminals are usually installed. The access control terminals need to perform facial recognition on users entering. Users can only enter after passing facial recognition to ensure the safety of these places. In these application scenarios, in order to not be restricted by the network environment and to improve the efficiency of facial recognition, the facial recognition function can be built into the access control terminal. In this case, the access control system provider needs to develop a facial recognition function based on a neural network model for the access control terminal. Considering that the computing resources and memory resources of the access control terminal are limited, the required facial recognition model needs to be lightweight. In this case, the access control system provider can act as the neural network model demander and use the network model lightweight service provided by the server device 21 to directly obtain a lightweight facial recognition model.

[0067] In this embodiment, if Figure 2a As shown, when a neural network model demander requires a lightweight neural network model, the client 22 can generate model requirement description information and submit the model requirement description information to the server device 21. The model requirement description information is information that can describe the neural network model demander's requirements for the neural network model, for example, including but not limited to: specifying the reference neural network model used in the model generation process and specifying the model size of the target neural network model to be generated. Furthermore, the model requirement description information can also include: performance requirements for the target neural network model.

[0068] Optionally, the server device 21 may provide a service entry interface to the client device 22, which may be a web page or an application page. The interface may include information items, which may be input items or selection items. The neural network model demander may use these information items to provide the server device 21 with a description of the model requirements.

[0069] The server device 21 receives the model requirement description information submitted by the client 22, and based on this, it can determine the reference neural network model specified by the neural network model demander, as well as the size requirements of the target neural network model by the neural network model demander; then, based on this information, a block-designed knowledge distillation framework is adopted. While ensuring the model size, the reference neural network model is used as a guide, and combined with automatic search technology to fully tap the learning ability of the network model, the target neural network model that has fully learned the prior information of the reference neural network model is automatically searched. The structure of the target neural network model is relatively simple, but the performance is the same or similar to that of the reference neural network model.

[0070] like Figure 2a As shown, after the server device 21 automatically searches for the target neural network model required by the client 22, it can provide the target neural network model to the client 22. The client 22 receives the target neural network model returned by the server device 21 and can deploy the target neural network model on the corresponding terminal device for operation. Figure 2a As shown, in the autonomous driving scenario, the client 22 can develop on-board navigation software, on-board voice recognition software, on-board target detection software or on-board target trajectory prediction software based on the target neural network model, and deploy these software on the autonomous driving vehicle for operation. During operation, the autonomous driving vehicle 23 can automatically plan the path based on the on-board navigation software, or can interact with the user by voice based on the on-board voice recognition software, or can detect pedestrians and other vehicles on the path based on the on-board target detection software and avoid the detected pedestrians and other vehicles, or can predict the movement trajectory of pedestrians or other vehicles on the path based on the on-board target trajectory prediction software and perform path navigation accordingly. Further, as Figure 2a As shown, in the access control system, client 22 can develop facial recognition software based on the target neural network model and deploy this facial recognition software on the access control terminal. During use, the access control terminal can perform facial recognition on the user based on this facial recognition software. This not only ensures the accuracy of the facial recognition results, but also outputs the facial recognition results promptly and quickly. Moreover, because facial recognition is performed locally, it is not restricted by the network environment.

[0071] Among them, the server device 21 adopts a block-designed knowledge distillation framework. While ensuring the model size, it uses the reference neural network model as a guide and combines automatic search technology to fully explore the learning ability of the network model. The process of automatically searching for a target neural network model that has fully learned the prior information of the reference neural network model includes: dividing the reference neural network model into N reference modules, generating N target modules corresponding to the N reference modules to obtain an initial neural network model, where N is an integer greater than or equal to 2; inputting the sample image into the reference neural network model and the initial neural network model for processing to obtain the output results of N reference modules and N target modules; using the model size included in the above-mentioned model requirement description information as a constraint condition, based on the output results of the N reference modules and the N target modules, jointly training the N target modules until the set stop condition is met to obtain the target neural network model. The detailed implementation process of these operations is the same or similar to the corresponding steps in the aforementioned network model search method embodiment. Please refer to the aforementioned embodiment and will not be repeated here.

[0072] In this embodiment, the server-side device adopts a knowledge distillation framework with a block design, which can fully utilize the structural characteristics of the reference neural network model, so that the target neural network model can "take its essence and discard its dross", and guide the search for a high-performance, lightweight target neural network model; in addition, the integration of automatic search technology can automatically search for the target neural network model. Compared with the manual design method, it can fully learn the structural information and knowledge information of the reference neural network model, and can further improve the performance of the target neural network model while ensuring the model size. Furthermore, the implementation of the lightweight network model is service-oriented, and the server-side device provides the lightweight neural network model to the client in the form of a service, which can free the neural network model demander from the implementation of the lightweight network model, allowing the neural network model demander to focus more on the function and application of the neural network model, which is conducive to improving the service quality of the application and increasing user stickiness.

[0073] Figure 2b The flowchart of a network model training method provided by the exemplary embodiment of the present application is as follows. Figure 2b As shown, the method includes:

[0074] 201. Receive model requirement description information, where the model requirement description information includes a specified reference neural network model and a specified model size of a target neural network model;

[0075] 202. Divide the reference neural network model into N reference modules, and generate N target modules corresponding to the N reference modules to obtain an initial neural network model, where N is an integer greater than or equal to 2;

[0076] 203. Input the sample image into the reference neural network model and the initial neural network model for processing, so as to obtain output results of N reference modules and N target modules;

[0077] 204. Taking the model size as a constraint, based on the output results of the N reference modules and the N target modules, the N target modules are jointly trained until the set stopping condition is met to obtain the target neural network model.

[0078] After obtaining the target neural network model, further optionally, as Figure 2b As shown, the method further includes:

[0079] 205. Provide the target neural network model to the client, so that the client can deploy the target neural network model on the terminal device for operation. The client is the model demander, and the above-mentioned model demand description information can be provided by the client, but is not limited thereto.

[0080] In an optional embodiment, the above-mentioned dividing the reference neural network model into N reference modules includes: dividing the reference neural network model into N reference modules according to the resolution size supported by the reference neural network model, and the N reference modules have different resolutions.

[0081] In an optional embodiment, the above-mentioned generation of N target modules corresponding to the N reference modules to obtain an initial neural network model includes: for each reference module, generating a search space for its corresponding target module, the search space including the maximum number of network layers that the target module can support, the layer coefficient of each layer, and the operators and operator coefficients that each layer can support; generating the initial operator of the target module according to the maximum number of network layers, the layer coefficient of each layer, and the operators and operator coefficients that each layer can support.

[0082] In an optional embodiment, the above-mentioned method uses the model size as a constraint condition and jointly trains the N target modules based on the output results of the N reference modules and the N target modules until the set stopping condition is met to obtain the target neural network model, including: obtaining the loss function of the initial neural network model based on the output results of the N reference modules and the N target modules in combination with the model size; searching the N target modules in the search space corresponding to each of the N target modules until the loss function meets the set stopping condition to obtain the target neural network model.

[0083] In an optional embodiment, the above-mentioned loss function of the initial neural network model is obtained based on the output results of N reference modules and N target modules, combined with the model size, including: obtaining a first performance loss function based on the output results of the reference neural network model and the initial neural network model; obtaining a second performance loss function based on the output results of the N reference modules and N target modules; and weighting the first performance loss function, the second performance loss function and the model size to obtain the loss function of the initial neural network model.

[0084] In an optional embodiment, the above-mentioned method of obtaining the second performance loss function of the initial neural network model based on the output results of N reference modules and N target modules includes: calculating the performance similarity of the N target modules and their corresponding reference modules based on the output results of the N reference modules and the N target modules in combination with the spatial attention mechanism; obtaining the performance loss function of the N target modules based on the performance similarity of the N target modules and their corresponding reference modules; and performing weighted processing on the performance loss function of the N target modules to obtain the second performance loss function of the initial neural network model.

[0085] In an optional embodiment, the above-mentioned performance similarity between the N target modules and their corresponding reference modules is calculated based on the output results of the N reference modules and the N target modules, combined with the spatial attention mechanism, including: for each target module, respectively counting the feature quantities of the feature graph output by the target module and its corresponding reference module; extracting the feature quantities of the target module and its corresponding reference module to obtain the extraction results; calculating the similarity of the extraction results as the performance similarity between the target module and its corresponding reference module.

[0086] In an optional embodiment, the above-mentioned search space corresponding to each of the N target modules is used to search the N target modules until the loss function satisfies the set stopping condition to obtain the target neural network model, including: for each target module, searching the number of network layers included in the target module and the operators used in each layer in its corresponding search space until the loss function satisfies the set stopping condition to obtain the target operator of the target module; wherein, the target operators of the N target modules constitute the target neural network model.

[0087] In an optional embodiment, the above-mentioned input of the sample image into the initial neural network model for processing to obtain the output results of N target modules includes: for non-first target modules among the N target modules, weighted summation of the output results of the previous target module and the corresponding previous reference module; and processing the weighted summation result as the input of the non-first target module to obtain the output result of the non-first target module.

[0088] In an optional embodiment, the above-mentioned weighted summation of the output results of the previous target module and the corresponding previous reference module includes: based on a weighting factor that decays over time, weighted summation of the output results of the previous target module and the corresponding previous reference module; wherein, as time goes by, the weight coefficient corresponding to the previous target module gradually increases, and the weight coefficient corresponding to the previous reference module gradually decreases.

[0089] In an optional embodiment, the target neural network model is a target detection model or a target trajectory prediction model, and the terminal device is an autonomous driving vehicle.

[0090] In this embodiment, the knowledge distillation framework with block design can make full use of the structural characteristics of the reference neural network model, so that the target neural network model can "take its essence and discard its dross", and guide the search for a high-performance, lightweight target neural network model; in addition, the integration of automatic search technology can automatically search for the target neural network model. Compared with the manual design method, it can fully learn the structural information and knowledge information of the reference neural network model, and further improve the performance of the target neural network model while ensuring the model size. Furthermore, the implementation of the lightweight network model is service-oriented, and the lightweight neural network model is provided to the client in the form of a service. This can free the neural network model demander from the implementation of the lightweight network model, allowing the neural network model demander to focus more on the function and application of the neural network model, which is conducive to improving the service quality of the application and increasing user stickiness.

[0091] It should be noted that the execution entity of each step of the method provided in the above embodiment can be the same device, or the method can be executed by different devices. For example, the execution entity of steps 101 to 103 can be device A; for another example, the execution entity of steps 101 and 102 can be device A, and the execution entity of step 103 can be device B; and so on.

[0092] In addition, some of the processes described in the above embodiments and the accompanying drawings include multiple operations that appear in a specific order, but it should be clearly understood that these operations may not be executed in the order in which they appear in this article or may be executed in parallel. The serial numbers of the operations, such as 101, 102, etc., are only used to distinguish between different operations, and the serial numbers themselves do not represent any order of execution. In addition, these processes may include more or fewer operations, and these operations may be executed in sequence or in parallel. It should be noted that the descriptions of "first", "second", etc. in this article are used to distinguish different messages, devices, modules, etc., and do not represent a sequential order, nor do they limit "first" and "second" to different types.

[0093] Figure 3This is a schematic diagram of the structure of a server device provided by an exemplary embodiment of the present application. Figure 3 As shown, the server device includes: a memory 31 , a processor 32 and a communication component 33 .

[0094] The memory 31 is used to store computer programs and can be configured to store various other data to support operations on the server device. Examples of such data include instructions, messages, images, videos, etc. for any application or method operating on the server device.

[0095] The processor 32 is coupled to the memory 31 and is configured to execute the computer program in the memory 31 to:

[0096] Receive model requirement description information through the communication component 33, where the model requirement description information includes a specified reference neural network model and a specified model size of a target neural network model;

[0097] The reference neural network model is divided into N reference modules, and N target modules are generated corresponding to the N reference modules to obtain an initial neural network model, where N is an integer greater than or equal to 2;

[0098] The sample images are input into the reference neural network model and the initial neural network model for processing to obtain the output results of N reference modules and N target modules;

[0099] Taking the model size as the constraint, based on the output results of N reference modules and N target modules, the N target modules are jointly trained until the set stopping condition is met to obtain the target neural network model.

[0100] Further optionally, the processor 32 is also used to: after obtaining the target neural network model, provide the target neural network model to the client through the communication component 33, so that the client can deploy the target neural network model on the terminal device for operation.

[0101] In an optional embodiment, when the processor 32 divides the reference neural network model into N reference modules, it is specifically used to: divide the reference neural network model into N reference modules according to the resolution size supported by the reference neural network model, and the N reference modules have different resolutions.

[0102] In an optional embodiment, when the processor 32 generates N target modules corresponding to N reference modules to obtain an initial neural network model, it is specifically used to: generate a search space for the corresponding target module for each reference module, the search space including the maximum number of network layers that the target module can support, the layer coefficient of each layer, and the operators and operator coefficients that each layer can support; generate the initial operator of the target module according to the maximum number of network layers, the layer coefficient of each layer, and the operators and operator coefficients that each layer can support.

[0103] In an optional embodiment, when the processor 32 jointly trains N target modules, it is specifically used to: obtain the loss function of the initial neural network model based on the output results of the N reference modules and the N target modules, combined with the model size; search the N target modules in the search space corresponding to each of the N target modules until the loss function meets the set stopping condition, thereby obtaining the target neural network model.

[0104] In an optional embodiment, when the processor 32 obtains the loss function of the initial neural network model based on the output results of N reference modules and N target modules in combination with the model size, it is specifically used to: obtain a first performance loss function based on the output results of the reference neural network model and the initial neural network model; obtain a second performance loss function based on the output results of the N reference modules and N target modules; and perform weighted processing on the first performance loss function, the second performance loss function and the model size to obtain the loss function of the initial neural network model.

[0105] In an optional embodiment, when the processor 32 obtains the second performance loss function based on the output results of N reference modules and N target modules, it is specifically used to: calculate the performance similarity of the N target modules and their corresponding reference modules based on the output results of the N reference modules and the N target modules in combination with the spatial attention mechanism; obtain the performance loss function of the N target modules based on the performance similarity of the N target modules and their corresponding reference modules; and perform weighted processing on the performance loss function of the N target modules to obtain the second performance loss function.

[0106] In an optional embodiment, when the processor 32 calculates the performance similarity between N target modules and their corresponding reference modules, it is specifically used to: for each target module, respectively count the feature quantities of the feature graph output by the target module and its corresponding reference module; extract the feature quantities of the target module and its corresponding reference module to obtain an extraction result; calculate the similarity of the extraction result as the performance similarity between the target module and its corresponding reference module.

[0107] In an optional embodiment, when obtaining the target neural network model, the processor 32 is specifically used to: for each target module, search the number of network layers included in the target module and the operators used in each layer in its corresponding search space until the loss function meets the set stopping condition, thereby obtaining the target operator of the target module; wherein, the target operators of N target modules constitute the target neural network model.

[0108] In an optional embodiment, when the processor 32 inputs the sample image into the initial neural network model for processing to obtain the output results of N target modules, it is specifically used to: perform weighted summation on the output results of the previous target module and the corresponding previous reference module for the non-first target module among the N target modules; and process the weighted summation result as the input of the non-first target module to obtain the output result of the non-first target module.

[0109] In an optional embodiment, when the processor 32 performs weighted summation on the output results of the previous target module and the corresponding previous reference module, it is specifically used to: perform weighted summation on the output results of the previous target module and the corresponding previous reference module based on a weighting factor that decays over time; wherein, as time goes by, the weight coefficient corresponding to the previous target module gradually increases, and the weight coefficient corresponding to the previous reference module gradually decreases.

[0110] In an optional embodiment, the target neural network model is a target detection model or a target trajectory prediction model, and the terminal device is an autonomous driving vehicle.

[0111] Further, if Figure 3 As shown, the server device also includes: a power supply component 34 and other components. Figure 3 Only some components are shown schematically, which does not mean that the server device only includes Figure 3 Components shown.

[0112] Accordingly, the embodiment of the present application further provides a computer-readable storage medium storing a computer program, which, when executed by a processor, enables the processor to implement the above-mentioned Figure 2b The steps in the method embodiment are shown.

[0113] Accordingly, the embodiment of the present application further provides a computer program product, including a computer program / instruction, which, when executed by a processor, enables the processor to implement the above-mentioned Figure 2b The steps in the method embodiment are shown.

[0114] The embodiment of the present application also provides a server device, the structure of which is similar to Figure 3 The structure of the server device shown is the same or similar, please refer to Figure 3, which is not shown in the figure. The server device of this embodiment is Figure 3 The difference between the server devices shown is that the functions implemented by the processor executing the computer program stored in the memory are different. The server device of this embodiment, whose processor executes the computer program stored in the memory, is used to: divide the given reference neural network model into N reference modules, generate N target modules corresponding to the N reference modules to obtain an initial neural network model, where N is an integer greater than or equal to 2; input the sample image into the reference neural network model and the initial neural network model for processing to obtain the output results of N reference modules and N target modules; with the specified model size as the constraint condition, the N target modules are jointly trained based on the output results of the N reference modules and the N target modules until the set stop condition is met to obtain the target neural network model. For the detailed implementation process of the above operations, please refer to the aforementioned embodiment and will not be repeated here.

[0115] Accordingly, the embodiment of the present application further provides a computer-readable storage medium storing a computer program, which, when executed by a processor, enables the processor to implement the above-mentioned Figure 1a The steps in the method embodiment are shown.

[0116] Accordingly, the embodiment of the present application further provides a computer program product, including a computer program / instruction, which, when executed by a processor, enables the processor to implement the above-mentioned Figure 1a The steps in the method embodiment are shown.

[0117] above Figure 3 The memory 31 can be implemented by any type of volatile or non-volatile memory device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0118] above Figure 3The communication component is configured to facilitate wired or wireless communication between the device where the communication component is located and other devices. The device where the communication component is located can access a wireless network based on a communication standard, such as WiFi, 2G, 3G, 4G / LTE, 5G and other mobile communication networks, or a combination thereof. In an exemplary embodiment, the communication component receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0119] above Figure 3 The power supply component in a device provides power to various components of the device in which the power supply component is located. The power supply component may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the device in which the power supply component is located.

[0120] Those skilled in the art will appreciate that the embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application can adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0121] The present application is described with reference to the flow chart and / or block diagram of the method, device (system), and computer program product according to the embodiment of the present application. It should be understood that each flow process and / or box in the flow chart and / or block diagram and the combination of the flow process and / or box in the flow chart and / or block diagram can be realized by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processing machine or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device produce a device for realizing the function specified in one flow chart flow or multiple flows and / or one box or multiple boxes of the block diagram.

[0122] These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a product including an instruction device that implements the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.

[0123] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, so that the instructions executed on the computer or other programmable device provide steps for implementing the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.

[0124] In a typical configuration, a computing device includes one or more processors (CPUs), input / output interfaces, network interfaces, and memory.

[0125] Memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. Memory is an example of a computer-readable medium.

[0126] Computer-readable media includes permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. The information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include transitory computer-readable media (transitory media), such as modulated data signals and carrier waves.

[0127] It should also be noted that the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, commodity, or apparatus that includes a series of elements includes not only those elements but also other elements not explicitly listed, or includes elements inherent to such process, method, commodity, or apparatus. In the absence of further limitations, an element defined by the phrase "comprises a ..." does not exclude the presence of other identical elements in the process, method, commodity, or apparatus that includes the element.

[0128] The foregoing is merely an embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present application should all be included within the scope of the claims of the present application.

Claims

1. A training method for a network model, characterized in that: include: Receiving model requirement description information, wherein the model requirement description information includes a specified reference neural network model and a specified model size of a target neural network model; Dividing the reference neural network model into N reference modules, and generating N target modules corresponding to the N reference modules to obtain an initial neural network model, where N is an integer greater than or equal to 2; Inputting the sample image into the reference neural network model and the initial neural network model for processing respectively to obtain output results of the N reference modules and the N target modules; wherein the step of obtaining the output results of the N target modules includes: for non-first target modules among the N target modules, performing weighted summation on the output results of the previous target module and the corresponding previous reference module; and processing the weighted summation result as the input of the non-first target module to obtain the output result of the non-first target module; Taking the model size as a constraint condition, based on the output results of the N reference modules and the N target modules, the N target modules are jointly trained until the set stopping condition is met to obtain the target neural network model.

2. The method according to claim 1, characterized in that The reference neural network model is divided into N reference modules, including: According to the resolution size supported by the reference neural network model, the reference neural network model is divided into N reference modules, and the N reference modules have different resolution sizes.

3. The method according to claim 1, characterized in that Generate N target modules corresponding to the N reference modules to obtain an initial neural network model, including: For each reference module, a search space is generated for its corresponding target module, where the search space includes the maximum number of network layers supported by the target module, the layer coefficients of each layer, and the operators and operator coefficients supported by each layer; The initial operator of the target module is generated according to the maximum number of network layers, the layer coefficient of each layer, and the operators and operator coefficients that can be supported by each layer.

4. The method according to claim 1, wherein Taking the model size as a constraint condition, based on the output results of the N reference modules and the N target modules, jointly training the N target modules until a set stopping condition is met to obtain a target neural network model, including: Based on the output results of the N reference modules and the N target modules and in combination with the model size, obtaining a loss function of the initial neural network model; In the search space corresponding to each of the N target modules, the N target modules are searched until the loss function meets the set stopping condition, thereby obtaining the target neural network model.

5. The method according to claim 4, characterized in that Based on the output results of the N reference modules and the N target modules, combined with the model size, obtaining a loss function of the initial neural network model, including: Obtaining a first performance loss function according to output results of the reference neural network model and the initial neural network model; Obtaining a second performance loss function according to output results of the N reference modules and the N target modules; The first performance loss function, the second performance loss function, and the model size are weighted to obtain a loss function of the initial neural network model.

6. The method according to claim 5, characterized in that Obtaining a second performance loss function according to output results of the N reference modules and the N target modules, including: Based on the output results of the N reference modules and the N target modules, combined with the spatial attention mechanism, the performance similarity between the N target modules and their corresponding reference modules is calculated; Obtaining a performance loss function of the N target modules according to the performance similarities between the N target modules and their corresponding reference modules; The performance loss functions of the N target modules are weighted to obtain the second performance loss function.

7. The method according to claim 6, characterized in that According to the output results of the N reference modules and the N target modules, combined with the spatial attention mechanism, the performance similarity between the N target modules and their corresponding reference modules is calculated, including: For each target module, the feature quantities of the feature graphs output by the target module and its corresponding reference module are counted respectively; Extracting feature quantities of the target module and its corresponding reference module to obtain an extraction result; The similarity of the extraction results is calculated as the performance similarity between the target module and its corresponding reference module.

8. The method according to claim 4, characterized in that Searching the N target modules in the search spaces corresponding to the N target modules until the loss function satisfies a set stopping condition, thereby obtaining the target neural network model, including: For each target module, in its corresponding search space, search the number of network layers included in the target module and the operators used in each layer until the loss function meets the set stopping condition, and obtain the target operator of the target module; wherein, the target operators of the N target modules constitute the target neural network model.

9. The method according to claim 1, characterized in that Perform a weighted summation on the output results of the previous target module and the corresponding previous reference module, including: Based on a weighting factor that decays over time, the output results of the previous target module and the corresponding previous reference module are weighted summed; Among them, as time goes by, the weight coefficient corresponding to the previous target module gradually increases, and the weight coefficient corresponding to the previous reference module gradually decreases.

10. The method according to any one of claims 1 to 8, characterized in that The target neural network model is a target detection model or a target trajectory prediction model.

11. A network model search method, characterized in that: include: Divide a given reference neural network model into N reference modules, and generate N target modules corresponding to the N reference modules to obtain an initial neural network model, where N is an integer greater than or equal to 2; Inputting the sample image into the reference neural network model and the initial neural network model for processing respectively to obtain output results of the N reference modules and the N target modules; wherein the step of obtaining the output results of the N target modules includes: for non-first target modules among the N target modules, performing weighted summation on the output results of the previous target module and the corresponding previous reference module; and processing the weighted summation result as the input of the non-first target module to obtain the output result of the non-first target module; With the specified model size as a constraint, based on the output results of the N reference modules and the N target modules, the N target modules are jointly trained until the set stopping condition is met to obtain the target neural network model.

12. A server device, characterized in that: include: memory and processor; The memory is configured to store a computer program; the processor is coupled to the memory and configured to execute the computer program to: Receiving model requirement description information, wherein the model requirement description information includes a specified reference neural network model and a specified model size of a target neural network model; Dividing the reference neural network model into N reference modules, and generating N target modules corresponding to the N reference modules to obtain an initial neural network model, where N is an integer greater than or equal to 2; Inputting the sample image into the reference neural network model and the initial neural network model for processing respectively to obtain output results of the N reference modules and the N target modules; wherein the step of obtaining the output results of the N target modules comprises: for non-first target modules among the N target modules, performing weighted summation on the output results of the previous target module and the corresponding previous reference module; and processing the weighted summation result as the input of the non-first target module to obtain the output result of the non-first target module; Taking the model size as a constraint condition, based on the output results of the N reference modules and the N target modules, the N target modules are jointly trained until the set stopping condition is met to obtain the target neural network model.

13. A server device, characterized in that: include: memory and processor; The memory is configured to store a computer program; the processor is coupled to the memory and configured to execute the computer program, so as to: Divide a given reference neural network model into N reference modules, and generate N target modules corresponding to the N reference modules to obtain an initial neural network model, where N is an integer greater than or equal to 2; Inputting the sample image into the reference neural network model and the initial neural network model for processing respectively to obtain output results of the N reference modules and the N target modules; wherein the step of obtaining the output results of the N target modules comprises: for non-first target modules among the N target modules, performing weighted summation on the output results of the previous target module and the corresponding previous reference module; and processing the weighted summation result as the input of the non-first target module to obtain the output result of the non-first target module; With the specified model size as a constraint, based on the output results of the N reference modules and the N target modules, the N target modules are jointly trained until the set stopping condition is met to obtain the target neural network model.

14. A computer storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the processor is caused to implement the steps of the method according to any one of claims 1 to 11.

Images