Network model pre-training method, image processing method and system

By introducing flexible student models and shared parameters, the problem of insufficient adaptability of model pre-training in the existing technology under the constraints of diversified resources is solved, efficient and flexible model deployment and resource utilization are achieved, and the deployment process is simplified and the deployment process is reduced. Calculate the cost.

CN119026637APending Publication Date: 2024-11-26ALIPAY (HANGZHOU) INFORMATION TECH CO LTD
View PDF 4 Cites 0 Cited by

Patent Information

Application Number
CN202411034590.X
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-30
Publication Date
2024-11-26

Smart Images

  • Figure CN119026637A_ABST
    Figure CN119026637A_ABST
Patent Text Reader

Abstract

The invention provides a pre-training method of a network model, and an image processing method and system. The pre-training method is applied to a preset network, and the preset network comprises a teacher model, a complete student model and an elastic student model. In different times of iterative training, the elastic student model becomes a sub-network of a complete student model based on different hyper-parameters. In the nth iteration training, inputting the first visual angle image into the teacher model to obtain a first prediction result, inputting the second visual angle image into the complete student model and the elastic student model to obtain a second prediction result output by the complete student model and a third prediction result output by the elastic student model, and the preset network parameters are updated by minimizing the difference between the first prediction result and the second prediction result, the difference between the first prediction result and the third prediction result and the difference between the second prediction result and the third prediction result as training targets. The teacher model obtained through the method is used for providing models corresponding to different hyper-parameters.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a network model pre-training method, an image processing method and a system. Background Art

[0002] Model pre-training technology is a key link in improving the performance of downstream tasks.

[0003] In related technologies, model pre-training techniques are usually pre-trained for a fixed scale, that is, only a single model of a specific size can be pre-trained at a time. This means that they have limitations in adapting to diverse resource constraints.

[0004] It is worth noting that the contents of the above-mentioned related technologies are merely information known to the inventor personally, which does not mean that the above-mentioned information has entered the public domain before the filing date of this specification, nor does it mean that it can become the prior art of this specification. Summary of the invention

[0005] This specification provides a network model pre-training method, an image processing method and a system to avoid at least one of the above-mentioned technical problems.

[0006] In a first aspect, the present specification provides a pre-training method for a network model, the method being applied to a preset network, the preset network comprising a teacher model, a complete student model, and an elastic student model; the complete student model and the elastic student model share model parameters; the method comprising multiple iterative trainings, in which the elastic student model is made a subnetwork of the complete student model based on different hyperparameters in different iterative trainings, and in the nth iterative training, comprising:

[0007] Obtain a first-perspective image and a second-perspective image of the sample object;

[0008] Inputting the first perspective image into the teacher model to obtain a first prediction result output by the teacher model;

[0009] Inputting the second perspective image into the complete student model and the elastic student model respectively, obtaining a second prediction result output by the complete student model and a third prediction result output by the elastic student model;

[0010] Updating preset network parameters with the training objective of minimizing the difference between the first prediction result and the second prediction result, the difference between the first prediction result and the third prediction result, and the difference between the second prediction result and the third prediction result;

[0011] The teacher model obtained by the method is used to provide models corresponding to the different hyperparameters.

[0012] In some embodiments, the complete student model and the elastic student model respectively include a backbone network, and the complete student model and the elastic student model share a head structure Head; the second perspective image is input to the complete student model and the elastic student model respectively, and a second prediction result output by the complete student model and a third prediction result output by the elastic student model are obtained, including:

[0013] Inputting the second perspective image into the backbone network of the complete student model, obtaining a second image feature output by the backbone network of the complete student model, and obtaining a second prediction result output by the common Head based on the second image feature;

[0014] The second perspective image is input into the backbone network of the elastic student model to obtain the third image feature output by the backbone network of the elastic student model, and the third prediction result output by the common Head based on the third image feature is obtained.

[0015] In some embodiments, the updating of preset network parameters with minimizing the difference between the first prediction result and the second prediction result, the difference between the first prediction result and the third prediction result, and the difference between the second prediction result and the third prediction result as the training objective includes:

[0016] updating the shared model parameters with the training objective of minimizing the difference between the first prediction result and the second prediction result, the difference between the first prediction result and the third prediction result, and the difference between the second prediction result and the third prediction result; and

[0017] The teacher model parameters are updated according to the updated shared model parameters.

[0018] In some embodiments, updating the teacher model parameters according to the updated shared model parameters includes:

[0019] Based on the exponential moving average method, the updated shared model parameters are updated to the teacher model to update the teacher model parameters.

[0020] In some embodiments, the teacher model and the complete student model have the same network structure; the hyperparameters include network width and network depth.

[0021] In some embodiments, inputting the first perspective image to the teacher model to obtain a first prediction result output by the teacher model includes:

[0022] Extracting features from the first viewing angle image to obtain first image features; and

[0023] The first image feature is preprocessed to obtain the first prediction result, wherein the preprocessing includes at least one of feature conversion, centering processing, and normalization processing.

[0024] In a second aspect, this specification provides an image processing method, comprising:

[0025] Obtaining an image to be processed and a target processing resource for processing the image to be processed;

[0026] Obtaining a target model corresponding to the target processing resource from a pre-trained teacher model, wherein the teacher model is pre-trained based on the method according to any one of the first aspects and is used to provide models corresponding to the different hyperparameters, and the models corresponding to the different hyperparameters support different processing resources; and

[0027] The image to be processed is processed based on the target model to obtain a processing result.

[0028] In some embodiments, obtaining a target model corresponding to the target processing resource from a pre-trained teacher model includes:

[0029] determining, from the different hyperparameters, a target hyperparameter corresponding to the target processing resource; and

[0030] The target model corresponding to the target hyperparameters is extracted from the teacher model.

[0031] In a third aspect, this specification provides a pre-training system, including:

[0032] At least one storage medium storing at least one set of instructions for performing pre-training of a network model;

[0033] At least one processor is communicatively connected to the at least one storage medium, wherein when the at least one processor executes the at least one set of instruction sets, the method as described in any one of the first aspects is implemented.

[0034] In a fourth aspect, this specification provides an image processing system, comprising:

[0035] at least one storage medium storing at least one set of instructions for performing image processing;

[0036] At least one processor is communicatively connected to the at least one storage medium, wherein when the at least one processor executes the at least one set of instruction sets, the method as described in any one of the second aspects is implemented.

[0037] In a fifth aspect, the present specification provides a processor-readable storage medium, wherein the processor-readable storage medium stores a computer program, wherein the computer program is used to enable the processor to execute any one of the methods in the first aspect or the second aspect.

[0038] In a sixth aspect, the present specification provides a computer program product, comprising a computer program, which, when executed by a processor, implements the method described in any one of the first aspect or the second aspect.

[0039] It can be seen from the above technical solutions that this specification provides a pre-training method, an image processing method and a system for a network model. Among them, the pre-training method of the network model is applied to a preset network, the preset network includes a teacher model, a complete student model, and an elastic student model, the complete student model and the elastic student model share model parameters, the method includes multiple iterative training, in different iterative trainings, the elastic student model is made into a sub-network of the complete student model based on different hyperparameters, and in the nth iterative training, it includes: obtaining a first-perspective image and a second-perspective image of a sample object, inputting the first-perspective image into the teacher model, obtaining a first prediction result output by the teacher model, inputting the second-perspective image into the complete student model and the elastic student model respectively, obtaining a second prediction result output by the complete student model and a third prediction result output by the elastic student model, and updating the preset network parameters with the training goal of minimizing the difference between the first prediction result and the second prediction result, the difference between the first prediction result and the third prediction result, and the difference between the second prediction result and the third prediction result. Among them, the teacher model obtained by this method is used to provide models corresponding to different hyperparameters. That is, the method provided in this manual can produce a series of models that can adapt to different resource constraints through one-time pre-training, thereby simplifying the deployment process, reducing additional training and computing costs, and improving the application efficiency and flexibility of the model in a variety of scenarios.

[0040] The pre-training method of the network model, the image processing method and other functions of the system provided in this specification will be partially listed in the following description. The creative aspects of the pre-training method of the network model, the image processing method and the system provided in this specification can be fully explained by practicing or using the methods, devices and combinations described in the following detailed examples. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] In order to more clearly illustrate the technical solutions in the embodiments of this specification, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this specification. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0042] Figure 1A schematic diagram of an application scenario of a network model pre-training method according to an embodiment of this specification;

[0043] Figure 2 A schematic diagram of the hardware structure of a pre-training system for a network model according to an embodiment of this specification;

[0044] Figure 3 A flowchart of a method for pre-training a network model according to an embodiment of the present specification;

[0045] Figure 4 A schematic diagram of the principle of a pre-training method for a network model according to an embodiment of this specification;

[0046] Figure 5 A schematic diagram of the flow of an image processing method according to an embodiment of this specification;

[0047] Figure 6 Schematic diagram of the principle of the image processing method of the embodiment of this specification. DETAILED DESCRIPTION

[0048] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The implementations described in the following exemplary embodiments do not represent all implementations consistent with this specification. Instead, they are merely examples of devices and methods consistent with some aspects of this specification as detailed in the appended claims.

[0049] It should be understood that the terms "including" and "having" and any variations thereof in the embodiments of this specification are intended to cover but not exclude inclusion. For example, a product or device comprising a series of components is not necessarily limited to those components explicitly listed, but may include other components not explicitly listed or inherent to these products or devices.

[0050] In the embodiments of this specification, the term "and / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B may represent: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the associated objects before and after are in an "or" relationship.

[0051] The term "plurality" in the embodiments of the present specification refers to two or more than two, and other quantifiers are similar.

[0052] The terms "first", "second", "third", etc. in this specification are used to distinguish similar or similar objects or entities, and do not necessarily mean to limit a specific order or sequence, unless otherwise indicated. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances, for example, they can be implemented in an order other than those given in the diagrams or descriptions of the embodiments of this specification.

[0053] The term "unit / module" as used in this specification refers to any known or later developed hardware, software, firmware, artificial intelligence, fuzzy logic or combination of hardware and / or software code that is capable of performing the functions associated with that element.

[0054] To facilitate the reader's understanding of this specification, at least some of the terms in this specification are explained as follows:

[0055] Deep Learning (DL) is a subfield of Machine Learning (ML). It is about learning the inherent laws and representation levels of sample data. The information obtained in the learning process is of great help in interpreting data such as text, images, and sounds.

[0056] Model pre-training is a common strategy in deep learning. It involves pre-training the model on a large amount of data to learn general feature representations, and then applying this pre-trained model to specific downstream tasks to learn specific task features through fine-tuning. Model pre-training significantly improves the performance of the model on specific tasks, especially in scenarios with less labeled data.

[0057] Self-supervised learning is a data model pre-training method that does not rely on manual annotation. It aims to enable the model to learn useful features through the intrinsic structure and information of the data itself. The core idea of ​​self-supervised learning is to let the model generate supervision signals by itself, and train the model through these self-generated signals to learn rich representations of data.

[0058] Knowledge distillation is a model compression technology. Its core idea is to transfer the knowledge of a large and complex model to a small and simple model. In this process, the small and simple model tries to imitate the behavior of the large and complex model to refine and transfer the knowledge of the large and complex model to the small and simple model. Among them, the large and complex model is usually called the teacher model (Teacher), and the small and simple model is usually called the student model (Student).

[0059] Weight pruning is a model compression technique. Its main purpose is to reduce the size of the neural network, thereby reducing the storage requirements and computational complexity of the model, and improving the inference speed, while trying to maintain or only slightly sacrifice the performance of the model.

[0060] Parameter quantization is a model optimization technique that aims to reduce the bit width of model parameters, thereby reducing the model's storage requirements and computational complexity.

[0061] To facilitate readers' understanding of this specification, the application scenarios of the pre-training method of the network model in this specification are now explained.

[0062] The network model pre-training method of this specification can be applied to the scene of training the network model based on the image. On this basis, the trained network model can be applied to the scene of processing the image, such as the scene of image recognition.

[0063] For example, image recognition can be performed for target detection of a target object in an image to obtain the location and category of the target object. For another example, image recognition can be performed for image classification to determine which predefined category the image belongs to. For another example, image recognition can be performed for semantic segmentation of an image to obtain the category label of each pixel in the image, and ultimately determine what object or category the different areas of the image belong to. For another example, image recognition can be performed for key point detection to determine the location of important key points (such as facial feature points or human key points) on a target object (such as a human body or a specific object) in an image. Of course, image recognition can also be performed for identifying the attributes of a target object in an image to obtain the attributes of the target object, such as color, texture, material, etc.

[0064] Taking target detection as an example, the image may be a remote sensing image; the target object may be a ground object in the remote sensing image; target detection may be to determine the category of the ground object, such as whether the ground object is a building or a forest. Remote sensing images refer to image data of the earth's surface or other targets obtained from a long distance through remote sensing technology.

[0065] Taking semantic segmentation as an example, the image may be an image to be identified obtained by automatic optical inspection (AOI). The target object is a plurality of objects to be detected in the image to be identified. Object detection may be to determine which object to be detected belongs to different regions in the image to be identified.

[0066] It is understandable that model pre-training technology is a key link in improving the performance of downstream tasks. It provides a powerful initialization model for subsequent specific tasks by learning common features on large-scale data sets. However, in the actual deployment process, different application scenarios face different resource constraints such as computing resources and storage resources, which has become a challenge that cannot be ignored. Model pre-training technology is usually pre-trained for a fixed scale, that is, only a single model of a specific size can be pre-trained at a time. This means that they have limitations in adapting to diverse resource constraints.

[0067] However, in real-world application scenarios, requirements are often variable and diverse, such as requiring a series of models of different sizes (such as different hyperparameters) to match various computing resources and storage resources. The current common solutions for deploying a pre-trained model to devices with different resource constraints include: model weight pruning, quantization compression, knowledge distillation, and even retraining a model. Although these methods have alleviated the resource consumption problem to a certain extent, they are often accompanied by additional cumbersome training and computing overhead, which increases the complexity and time consumption of model deployment.

[0068] Among them, model weight pruning removes unimportant weights in the model to reduce the size and computational complexity of the model; quantization converts model parameters from high precision to low precision to save storage space and accelerate the inference process; knowledge distillation transfers the knowledge of the teacher model to the student model, retaining most of the performance of the teacher model with a smaller student model size; and although retraining the model can be targeted to adapt to specific resource constraints, it means spending time and computing resources starting from scratch.

[0069] It is worth noting that the contents of the above-mentioned related technologies are merely information known to the inventor personally, which does not mean that the above-mentioned information has entered the public domain before the filing date of this specification, nor does it mean that it can become the prior art of this specification.

[0070] In order to avoid at least one of the above problems, this specification proposes a technical conception through creative labor: introducing an elastic student model (Elastic Student) in the student model, that is, the student model includes both a complete student model (Intact Student) and an elastic student model, and the elastic student model and the complete student model share parameters. In the process of knowledge distillation based on the teacher model and the student model, the elastic student model in different iterative trainings is a different sub-model extracted from the complete student model based on different hyperparameters, so that the student model learns the knowledge of the teacher model and the elastic student model learns the knowledge of the complete student model, and the knowledge learned by the student model is updated to the teacher model, so as to pre-train and output a teacher model including models supporting different sizes through self-supervised learning. Among them, the teacher model obtained by one-time pre-training can be called a large model, and the models of different sizes supported by the teacher model can be called small models. That is, through the technical solution of this specification, a large model including small models of different sizes can be obtained by one-time pre-training.

[0071] The complete student model can be understood as a model that is designed to have the same structure as the teacher model. This model may have a more complex architecture and more parameters in order to capture the behavior of the teacher model. For example, during the process of knowledge retention, the complete student model will try to imitate the output of the teacher model to learn as much knowledge as possible from the teacher model.

[0072] The elastic student model can be understood as a network model that is more flexible in architecture and number of parameters and can effectively operate under different constraints (such as computing resources, memory limitations, latency requirements, etc.). The elastic student model may contain variable architectural components, dynamic routing mechanisms, or adjustable parameter configurations, all of which enable the model to adapt to different hardware and application environments while maintaining high performance.

[0073] Exemplarily, the technical concept of this embodiment can be understood from the preparation stage, the knowledge distillation stage, and the model parameter update stage.

[0074] In the preparation stage, a plurality of sample image pairs are collected (a sample image pair includes images of two different perspectives, and the two different perspectives constitute a global perspective); a network (i.e., a preset network) is set. The preset network includes a teacher model and a student model. The student model further includes a complete student model and an elastic student model.

[0075] Knowledge distillation stage: The elastic student model is a sub-network of the complete student model, and in different iterations of training, the elastic student model will change its hyperparameters. This means that in each iteration of training, the structure or number of parameters of the elastic student model may be different to explore and learn the performance of models of different sizes and complexities throughout the training process. This process allows the elastic student model to adapt to different resource constraints such as computing resources or storage resources while maintaining high performance.

[0076] Model parameter update phase: After knowledge distillation, the model parameters of the student model will be updated to the teacher model. This update process essentially integrates the knowledge learned by the student model into the teacher model, thereby enriching the capabilities of the teacher model. Ultimately, the teacher model will support providing models corresponding to different hyperparameters, and the model corresponding to each hyperparameter represents the model configuration under the corresponding resource constraints. It can be understood that after pre-training, a large model is finally obtained, and this large model integrates sub-models corresponding to different hyperparameters. These sub-models strike a balance between performance and resource consumption, and can be flexibly selected according to the actual application environment to meet different resource requirements such as computing resources or storage resources.

[0077] Through the methods of the above stages, not only can the complex knowledge of the teacher model be transferred to the student model, but also the model performance under different resource constraints can be explored and optimized through the elastic student model, thereby achieving efficient and adaptable model deployment. That is, a series of models that can adapt to different resource constraints can be produced through one-time pre-training, thereby simplifying the deployment process, reducing additional training and computing costs, and improving the application efficiency and flexibility of the model in various scenarios.

[0078] Figure 1 FIG. 1 is a schematic diagram of an application scenario of the network model pre-training method of the embodiment of the present specification, wherein the network model pre-training method of the present specification can be applied to the following: Figure 1 The application scenario 100 shown. Figure 1 As shown, the application scenario 100 may include a target user 101 , a client 102 , a server 103 , and a network 104 .

[0079] The target user 101 is a user who triggers the pre-training of the network model. For example, the target user 101 can perform a target operation on the client 102 to trigger the pre-training of the network model.

[0080] The client 102 is a device for pre-training the network model in response to the target operation of the target user 101. That is, the pre-training method of the network model can be executed on the client 102. At this time, the client 102 can store data or instructions for executing the pre-training method of the network model described in this specification, and can execute or be used to execute the data or instructions. In some embodiments, the client 102 may include a hardware device with a data information processing function and the necessary program required to drive the hardware device to work.

[0081] In some embodiments, the client 102 may include a mobile device, a tablet computer, a laptop computer, a built-in device of a motor vehicle, or the like, or any combination thereof. In some embodiments, the mobile device may include a smart home device, a smart mobile device, a virtual reality device, an augmented reality device, or the like, or any combination thereof. In some embodiments, the smart home device may include a smart TV, a desktop computer, or the like, or any combination thereof. In some embodiments, the smart mobile device may include a smart phone, a personal digital assistant, a gaming device, a navigation device, or the like, or any combination thereof. In some embodiments, the built-in device in a motor vehicle may include an onboard computer, an onboard TV, or the like.

[0082] In some embodiments, the client 102 may be installed with one or more applications (Application, APP). APP can provide the target user 101 with the ability and interface to interact with the outside world through the network 104. APP includes but is not limited to: web browser APP programs, search APP programs, chat APP programs, shopping APP programs, video APP programs, financial management APP programs, instant messaging tools, email clients, social platform software, etc.

[0083] like Figure 1 As shown, the client 102 can be connected to the server 103 for communication. The server 103 can be connected to one client 102 for communication, or can be connected to multiple clients 102 for communication. In some embodiments, the client 102 can interact with the server 103 through the network 104 to receive or send messages, etc.

[0084] The server 103 may be a server that provides various services, such as a backend server that provides support for user data sets and account login information corresponding to multiple accounts collected on the client 102, and pre-training of network models for multiple accounts.

[0085] In some embodiments, the pre-training method of the network model may be executed on the server 103. At this time, the server 103 may store data or instructions for executing the pre-training method of the network model described in this specification, and may execute or be used to execute the data or instructions.

[0086] In some embodiments, the server 103 may include a hardware device with data information processing functions and a necessary program to drive the hardware device to work. Similarly, the server 103 may be connected to a client 102 for communication and receive data sent by the client 102, or may be connected to multiple clients 102 for communication and receive data sent by each client 102.

[0087] The network 104 is a medium for providing a communication connection between the client 102 and the server 103. The network 104 can facilitate the exchange of information or data. Figure 1 As shown, the client 102 and the server 103 can be connected to the network 104 respectively, and transmit information or data to each other through the network 104.

[0088] In some embodiments, the network 104 may be any type of wired or wireless network, or a combination thereof. For example, the network 104 may include a cable network, a wired network, a fiber optic network, a telecommunications network, an intranet, the Internet, a local area network (LAN), a wide area network (WAN), a wireless local area network (WLAN), a metropolitan area network (MAN), a public switched telephone network (PSTN), a Bluetooth network TM, a short-range wireless network (ZigBee TM), a near field communication (NFC) network, or a similar network.

[0089] In some embodiments, the network 104 may include one or more network access points. For example, the network 104 may include a wired or wireless network access point, such as a base station or an Internet exchange point, through which one or more components of the client 102 and the server 103 can connect to the network 104 to exchange data or information.

[0090] It is understandable that the above examples are only used to illustrate the application scenarios to which the network model pre-training method of this specification may be applicable, and cannot be understood as limiting the application scenarios. Figure 1The number of clients 102, servers 103, and networks 104 is merely illustrative. Depending on the implementation requirements, there may be any number of clients 102, servers 103, and networks 104. The network model pre-training method provided in this specification may be executed entirely on the client 102, entirely on the server 103, or partially on the client 102 and partially on the server 103.

[0091] Figure 2 The hardware structure diagram of a pre-training system for a network model (hereinafter referred to as the pre-training system) 200 provided according to an embodiment of the present specification is shown. The pre-training system 200 can execute the pre-training method for the network model described in the present specification. The pre-training method for the network model is introduced in other parts of the present specification. When the pre-training method for the network model is executed on the client 102, the pre-training system 200 can be the client 102. When the pre-training method for the network model is executed on the server 103, the pre-training system 200 can be the server 103. When the pre-training method for the network model can be partially executed on the client 102 and partially executed on the server 103, the pre-training system 200 can be a system including the client 102 and the server 103.

[0092] like Figure 2 As shown, the pre-training system 200 may include at least one storage medium 203 and at least one processor 202. In some embodiments, the pre-training system 200 may also include a communication port 204 and an internal communication bus 201. At the same time, the pre-training system 200 may also include an I / O component 205.

[0093] The internal communication bus 201 can connect different system components. For example, the internal communication bus 201 can connect the storage medium 203, the processor 202, the communication port 204 and the I / O component 205.

[0094] I / O component 205 supports input / output between pre-training system 200 and other components.

[0095] The communication port 204 is used for data communication between the pre-training system 200 and the outside world. For example, the communication port 204 can be used for data communication between the pre-training system 200 and the network 104. The communication port 204 can be a wired communication port or a wireless communication port.

[0096] The storage medium 203 may include a data storage device. The data storage device may be a non-temporary storage medium or a temporary storage medium. For example, the data storage device may include one or more of a disk 2031, a read-only storage medium (ROM) 2032, or a random access storage medium (RAM) 2033. The storage medium 203 also includes at least one instruction set stored in the data storage device. The instruction set includes a computer program code, which may include a program, a routine, an object, a component, a data structure, a process, a module, etc. for executing the pre-training method of the network model provided in this specification.

[0097] At least one processor 202 may be communicatively connected to at least one storage medium 203. At least one processor 202 is used to execute the at least one instruction set mentioned above. When the pre-training system 200 is running, at least one processor 202 reads the at least one instruction set and executes the pre-training method of the network model provided in this specification according to the instructions of the at least one instruction set. The processor 202 may execute all steps included in the pre-training method of the network model. The processor 202 may be in the form of one or more processors. In some embodiments, the processor 202 may include one or more hardware processors, such as a microcontroller, a microprocessor, a reduced instruction set computer (RISC), an application-specific integrated circuit (ASIC), an application-specific instruction set processor (ASIP), a central processing unit (CPU), a graphics processing unit (GPU), a physical processing unit (PPU), a microcontroller unit, a digital signal processor (DSP), a field programmable gate array (FPGA), an advanced RISC machine (ARM), a programmable logic device (PLD), any circuit or processor capable of performing one or more functions, etc., or any combination thereof.

[0098] For illustrative purposes only, only one processor 202 is shown in the pre-training system 200 in the accompanying drawings. However, it should be noted that the pre-training system 200 in this specification may also include multiple processors, and therefore, the operations and / or method steps disclosed in this specification may be performed by one processor or jointly by multiple processors. For example, if the processor 202 of the pre-training system 200 is described in this specification as performing steps A and B, it should be understood that steps A and B may also be performed jointly or separately by two different processors 202 (e.g., the first processor performs step A, the second processor performs step B, or the first and second processors perform steps A and B together).

[0099] Based on the above technical concept, this specification provides a pre-training method for a network model. Figure 3 This method is applied to Figure 4The preset network shown includes a teacher model, a complete student model, and an elastic student model; the complete student model and the elastic student model share model parameters.

[0100] This embodiment does not limit the types of teacher models and student models, which can be determined by the pre-training system based on demand, historical records, experiments, etc. For example, the teacher model can be ViT (Vision Transformer), ResNet, SwinTransformer, etc.

[0101] Combination Figure 4 It can be seen that the student model includes not only the complete student model but also the elastic student model. The model parameters of the student network are the shared model parameters of the elastic student model and the complete student model.

[0102] In terms of model structure, the complete student model is identical to the teacher model.

[0103] The method includes multiple iterative trainings, in which the elastic student model is made into a sub-network of the complete student model based on different hyperparameters in different iterative trainings.

[0104] Combination Figure 4 It can be seen that the elastic student model can be a sub-network of the complete student model. For example, in different iterations of training, the pre-training system can extract the elastic student model from the complete student model based on different hyperparameters. For example, the pre-training system can reduce the network width and network depth by a preset amount to determine a specific elastic student model.

[0105] In the process of different iterations of training, the preset number reduced by the pre-training system may be the same or different. For example, the pre-training system may reduce a certain amount of width and depth of the complete student model to determine a specific elastic student model.

[0106] That is to say, during the entire pre-training process, the hyperparameters related to the network structure of the elastic student model are constantly changing. Among them, the hyperparameters include network width and network depth. Network width can be understood as the number of neurons in each layer of the elastic student model. Network depth can be understood as the number of neural network layers in the elastic student model.

[0107] like Figure 3 As shown, in the nth iteration training, the following steps S301 to S303 are included:

[0108] S301: Obtain a first-perspective image and a second-perspective image of a sample object.

[0109] Exemplarily, the first-perspective image and the second-perspective image can be understood as two views of different perspectives generated from the same original image, and the two views of different perspectives can be combined to obtain a view of a global perspective.

[0110] For example, in combination with the above examples, it can be known that the original image may be a remote sensing image, the sample object may be a ground object, and the first-view image and the second-view image may be images showing the ground object from two different viewpoints.

[0111] In some embodiments, the first-perspective image and the second-perspective image may be perspective-enhanced images, such as the first-perspective image and the second-perspective image obtained through data enhancement processing.

[0112] This embodiment does not limit the way in which the pre-training system obtains the first-view image and the second-view image. Taking the pre-training system obtaining the first-view image as an example:

[0113] In one example, the pre-trained system can be connected to other systems and receive first-person perspective images sent by other systems.

[0114] For example, Figure 1 As an example of the application scenario shown in the figure, the pre-training system can be Figure 1 The server 103 shown, other devices may be as follows Figure 1 In the client 102 shown, the target user 101 can input the first-perspective image on the client 102 through an APP or a non-APP method to trigger the client 102 to send the first-perspective image to the server 103.

[0115] In another example, the pre-training system may provide a tool for loading the first-perspective image, so that the first-perspective image is transmitted to the pre-training system through the tool for loading the first-perspective image.

[0116] Among them, the tool for loading the first-perspective image can be an interface for connecting to an external device, such as an interface for connecting to other storage devices, through which the first-perspective image transmitted by the external device is obtained; the tool for loading the first-perspective image can also be a display device, such as the pre-training system can output an interface for loading the first-perspective image function on the display device, so as to import the first-perspective image into the pre-training system through the interface.

[0117] S302: Input the first-perspective image into the teacher model to obtain a first prediction result output by the teacher model.

[0118] Combined with the above analysis, it can be seen that the network model pre-training method of this specification can be applied to different scenarios. For different application scenarios, the prediction requirements may be different, and the sample objects may also be different.

[0119] For example, take the scenario of target detection of ground objects in remote sensing images as an example:

[0120] The first-perspective image is a remote sensing image of the object at a first perspective.

[0121] S302 can be understood as the teacher model identifying the category of the ground object in the first-person perspective image to obtain the category of the ground object (ie, the first prediction result).

[0122] Exemplarily, the pre-training system uses the first-perspective image as the input of the teacher model and allows the teacher model to process it. The teacher model predicts the possible category of the object in the first visual image based on its learned knowledge, which constitutes a first prediction result. The first prediction result can be a first probability, such as the probability that the object is a building.

[0123] For example, take the scenario of classifying animals in images as an example:

[0124] The first-perspective image is an image of the animal at a first perspective.

[0125] like Figure 4 As shown, the first-perspective image is the input of the teacher model. The teacher model predicts the category of the animal in the first-perspective image and obtains a first prediction result. For example, the probability that the animal in the first-perspective image is a dog is obtained.

[0126] In some embodiments, S302 may include the following steps 1 and 2:

[0127] Step 1: Extract features from the first-view image to obtain first image features.

[0128] For example, the network model can be generally divided into two parts: feature extraction part and classification / prediction part. The feature extraction part can be called the backbone network (Backbone, also called the skeleton structure). The classification / prediction part can be called the head structure (Head).

[0129] Correspondingly, such as Figure 4 As shown in FIG. 1 , the teacher model may include a backbone network and a Head. The first-view image is the input of the backbone network of the teacher model. The backbone network of the teacher model extracts features from the first-view image to obtain the first image feature Z a .

[0130] By extracting features from the first-person view image through the backbone network of the teacher model, the hierarchical features in the first-person view image can be captured. Low-level features may include edges, textures, etc., while high-level features may correspond to more abstract concepts, such as the partial or overall shape of the sample object. This feature representation method can greatly reduce the difficulty of subsequent classification or recognition tasks.

[0131] Step 2: Preprocess the first image feature to obtain a first prediction result, wherein the preprocessing includes at least one of feature conversion, centering, and normalization.

[0132] Exemplarily, feature transformation may include dimensionality reduction, projection or transformation to different feature spaces, such as using principal component analysis (PCA), etc. This transformation can help remove redundancy between features, reduce the complexity of the model, while retaining the integrity of the information and improving the generalization ability of pre-training.

[0133] Centralization can adjust the mean of features to zero. This can eliminate the impact of data offset and make pre-training easier to converge. In some optimization algorithms, such as gradient descent, centralized features can speed up convergence and improve stability.

[0134] Normalization can scale the range of feature values ​​to a fixed interval, such as [0, 1] or [-1, 1]. This helps prevent numerical differences between features from affecting model learning, especially when the feature scales vary greatly. Normalization can also avoid problems such as gradient disappearance.

[0135] Continuing with the examples above and Figure 4 , the first image feature Z a The input of the teacher model Head. The teacher model Head is responsible for the first image feature Z a The features after Head processing of the teacher model are centered by Sinkhorn-Knopp (SK) algorithm. The features after centering are normalized by the temperature-scaled Softmax function to obtain the first prediction result p a .

[0136] In some embodiments, the feature l after the centralization process a It can be obtained by formula 1:

[0137]

[0138] Among them, SK is centralized processing, H T is the feature processing (such as feature conversion) of the teacher model’s Head, P is the number of prototypes (such as categories), A collection of categories.

[0139] In some embodiments, the prediction result of category i can be obtained by express, It can be obtained by formula 2:

[0140]

[0141] Among them, exp() is an exponential function, τ is a preset temperature parameter, For arbitrary.

[0142] Based on the analysis of steps 1 and 2 above, it can be seen that in this embodiment, on the one hand, by effectively extracting and preprocessing the first-person perspective images, it is possible to provide guidance for the learning of the student network (as a soft label for learning), and on the other hand, centralized processing through the SK algorithm can make the prototypes in the training process more balanced.

[0143] S303: Input the second perspective image to the complete student model and the elastic student model respectively to obtain a second prediction result output by the complete student model and a third prediction result output by the elastic student model.

[0144] Similarly, in the above scenario of detecting a ground object in a remote sensing image, S303 can be understood as:

[0145] The pre-training system uses the second-view image as the input of the complete student model. The complete student model attempts to identify the category of the object in the second-view image and obtains a second prediction result. The second prediction result can characterize the possibility that the object is a building from a probability dimension.

[0146] The pre-training system uses the second-view image as the input of the elastic student model. The elastic student model attempts to identify the category of the object in the second-view image and obtains a third prediction result. The third prediction result can characterize the possibility that the object is a building from a probability dimension.

[0147] In the above scenario of classifying animals in an image, S303 can be understood as:

[0148] The pre-training system uses the second-view image as input to the full student model. The full student model attempts to identify the category of the animal in the second-view image and obtains a second prediction result. The second prediction result can characterize the possibility that the animal is a dog from a probability dimension.

[0149] The pre-training system uses the second-view image as input to the elastic student model. The elastic student model attempts to identify the category of the animal in the second-view image and obtains a third prediction result. The third prediction result can characterize the possibility that the animal is a dog from a probability dimension.

[0150] In some embodiments, Figure 4 As shown, the complete student model and the elastic student model include a backbone network respectively, and the complete student model and the elastic student model share a head structure Head.

[0151] Accordingly, S303 may include the following steps 1 and 2:

[0152] Step 1: Input the second-perspective image into the backbone network of the complete student model, obtain the second image features output by the backbone network of the complete student model, and obtain the second prediction result output by the shared Head based on the second image features.

[0153] Step 2: Input the second-perspective image into the backbone network of the elastic student model, obtain the third image features output by the backbone network of the elastic student model, and obtain the third prediction result output by the shared Head based on the third image features.

[0154] For example, see Figure 4 It can be seen that the input of the backbone network of the complete student model is the second-view image. The backbone network of the complete student model extracts features from the second-view image and obtains the second image feature Z b1 The second image feature v b1 The Head of the complete student model (that is, the Head of the elastic student model) is the input of the Head of the complete student model to the second image feature Z b1 The features after Head processing of the complete student model are normalized by the temperature-scaled Softmax function to obtain the second prediction result p b1 .

[0155] The backbone network of the elastic student model is a sub-network of the backbone network of the complete student model. For example, the pre-training system can extract the backbone network of the elastic student model from the backbone network of the complete student model based on preset parameters. The input of the backbone network of the elastic student model is the second-view image. The backbone network of the elastic student model extracts features from the second-view image to obtain the third image feature Z b2 The third image feature Z b2 is the input of the Head of the elastic student model (that is, the Head of the complete student model). The Head of the elastic student model is used for the third image feature Z b2 The features after Head processing of the elastic student model are normalized by the temperature-scaled Softmax function to obtain the third prediction result p b2 .

[0156] S304: updating the preset network parameters with the training objective of minimizing the difference between the first prediction result and the second prediction result, the difference between the first prediction result and the third prediction result, and the difference between the second prediction result and the third prediction result. The teacher model obtained by the method described in S301-S304 is used to provide models corresponding to different hyperparameters.

[0157] For example, combining the above examples and Figure 4 , the first prediction result is p a , the second prediction result is p b1 , the third prediction result is p b2 The prediction system calculates the first prediction result p a With the second prediction result p b1 The difference between (such as loss information L IS );Calculate the first prediction result p a And the third prediction result p b2 The difference between (such as loss information L ES1 );Calculate the second prediction result p b1 And the third prediction result p b2 The difference between (such as loss information L ES2 ).

[0158] This embodiment does not limit the method for calculating the loss information, which can be determined by the pre-training system based on demand, historical records, experiments, etc.

[0159] For example, the pre-training system can be based on the loss function shown in Equation 3 Calculate the loss information L ES1 , Formula 3:

[0160]

[0161] Here, log is the logarithm.

[0162] The pre-training system can be based on the loss function shown in Equation 4 Calculate the loss information L Es2 , Formula 4:

[0163]

[0164] In this embodiment, by minimizing the loss information L IS , the gap between the prediction performance of the complete student model and the teacher model can be reduced so that the performance of the complete student model on the second-view image is as close as possible to the performance of the teacher model on the first-view image.

[0165] By minimizing the loss information L ES1 , the gap between the prediction performance of the elastic student model and the teacher model can be reduced so that the performance of the full student model on the second-view image is as close as possible to the performance of the teacher model on the first-view image.

[0166] By minimizing the loss information L ES2, the gap between the prediction performance of the elastic student model and the complete student model can be reduced, so that the performance of the complete student model on the second-view image is as close as possible to the performance of the complete student model on the second-view image. That is, the elastic student model can learn the knowledge of the complete student model through the same-view images.

[0167] And through the above analysis, we can know that the teacher model finally pre-trained by the pre-training system is a large model, which integrates sub-models of different sizes. These sub-models have their own corresponding hyperparameters, so as to achieve a balance between performance and resource consumption through sub-models with different hyperparameters, and flexibly select from each sub-model according to the actual application scenario to meet the needs of specific computing resources or storage resources.

[0168] In some embodiments, S304 may include the following steps 1 and 2:

[0169] Step 1: Update the shared model parameters with the training objective of minimizing the difference between the first prediction result and the second prediction result, the difference between the first prediction result and the third prediction result, and the difference between the second prediction result and the third prediction result.

[0170] Combining the above examples with Figure 4 , the pre-training system can minimize the loss information L IS , loss information L ES1 , loss information L ES2 The sum of is calculated and the shared model parameters of the full student model and the elastic student model are updated based on the sum.

[0171] Step 2: Update the teacher model parameters according to the updated shared model parameters.

[0172] In some embodiments, step 2 may include: updating the updated shared model parameters to the teacher model based on an exponential moving average (EMA) method to update the teacher model parameters.

[0173] Continuing with the examples above and Figure 4 , the shared model parameters of the backbone network of the complete student model and the backbone network of the elastic student model are updated to the backbone network of the teacher model through EMA. The shared model parameters of the head shared by the complete student model and the elastic student model are updated to the head of the teacher model through EMA.

[0174] In this embodiment, since EMA is relatively smooth, updating the shared model parameters to the teacher model by EMA can avoid feature collapse during the training process. In addition, since EMA can better capture the average behavior of the student model throughout the pre-training process, it has a stronger generalization ability.

[0175] Based on the above technical concept, the present disclosure also provides an image processing method. Figure 5 As shown, the method includes the following S501 to S503:

[0176] S501: Obtain an image to be processed and a target processing resource corresponding to the image to be processed.

[0177] The execution subject of this embodiment may be an image processing system. The image processing system may be the same system as the pre-training system, or may be a system different from the pre-training system. For the structural principle of the image processing system, please refer to the above description of the pre-training system, which will not be repeated here.

[0178] Combined with the above analysis, it can be seen that the image processing method of this specification can be applied to different scenarios. For different application scenarios, the acquisition method and / or content of the image to be processed may be different. For example, the image to be processed may be an image including an animal to be identified, so as to determine the category of the animal to be identified, such as a dog or a cat, etc., through the image processing method of this embodiment.

[0179] Processing resources can be understood as resources for processing images. Corresponding to this embodiment, target processing resources can be understood as resources used by the image processing system to process the image to be processed. Among them, the target processing resources may include computing resources and / or storage resources; may also include user's device information, such as CPU, GPU, memory size, network bandwidth, etc.; may also include user-selected processing speed and quality preferences, etc., which are not listed here one by one.

[0180] S502: Obtain a target model corresponding to the target processing resources from a pre-trained teacher model, wherein the teacher model is obtained based on a pre-training method of a network model as described in any of the above embodiments, and is used to provide models corresponding to different hyperparameters, and the models corresponding to different hyperparameters support different processing resources.

[0181] Combining the above analysis and Figure 6 It can be seen that the teacher model can provide models corresponding to N (N is an integer greater than 1) different hyperparameters (such as Figure 6Each sub-model is optimized for different processing resources. For example, some sub-models are suitable for high-end devices and can provide the highest quality image processing; while other sub-models are suitable for low-end devices, although the processing quality is relatively low, but the speed can be achieved.

[0182] The image processing system determines the target model corresponding to the target processing resource from the N sub-models. Exemplarily, the image processing system can obtain a target model that can process the image to be processed. That is, the image processing system can select a relatively more suitable sub-model from the large model through resource analysis. For example, if the target processing resource represents that the user device has powerful performance and good network conditions, the image processing system can select a high-precision sub-model; otherwise, the image processing system can select a sub-model with less computational effort and faster speed. Figure 6 As shown, the target model determined by the image processing system from the N sub-models is sub-model 2.

[0183] In some embodiments, S502 may include the following sub-steps 1 and 2:

[0184] Step 1: From different hyperparameters, determine the target hyperparameters corresponding to the target processing resources.

[0185] Among them, the target hyperparameters include the target network width and the target network depth

[0186] This step can be understood as the image processing system can determine the target network width and target network depth that match the target processing resources. In the case of the target network width and target network depth, it can avoid the situation where the target processing resources cannot process the image to be processed, and it can also avoid the situation where there is excess resources when the target processing resources process the image to be processed.

[0187] Exemplarily, when determining the target network width and depth, the image processing system can make decisions based on target processing resources (such as CPU performance, GPU performance, memory size, battery status, network bandwidth, etc.). For example, for scenes with strong computing power, the image processing system can select a deeper and wider network to pursue higher accuracy; on the contrary, for scenes with limited resources, the image processing system can select a shallower and narrower network to ensure processing speed and energy efficiency.

[0188] Among them, a wider network means more neurons in each layer, which usually provides more powerful expression and higher prediction accuracy, but also requires more computing resources and larger memory. Deeper networks can learn more complex feature representations, but also increase computing latency and training difficulty, especially on devices with limited resources.

[0189] Step 2: Extract the target model corresponding to the target hyperparameters from the teacher model.

[0190] Exemplarily, when the target network width and the target network depth are determined, the image processing system can crop or extract from the large model to obtain a target model with the target network width and the target network depth as hyperparameters.

[0191] Based on the above steps 1 and 2, it can be seen that in this embodiment, the image processing system can maximize the use of existing resources, avoid resource waste, and ensure that the processing task can be successfully completed. In addition, in a resource-constrained environment, it is possible to quickly and accurately determine a model that matches the resources to obtain a better user experience.

[0192] S503: Process the image to be processed based on the target model to obtain a processing result.

[0193] Combining the above examples with Figure 6 , the image to be processed is used as the input of sub-model 2. Sub-model 2 processes the image to be processed to obtain a processing result. Among them, the processing process of the image to be processed by sub-model 2 may include feature extraction, head processing, centering, normalization, etc., which can be specifically referred to in the above example and will not be repeated here.

[0194] In combination with the above examples, the processing results may be different for different application scenarios. For example, the image to be processed may be a remote sensing image, which includes the objects to be identified. Sub-model 2 processes the remote sensing image to obtain the category of the objects to be identified, such as buildings or woodlands.

[0195] Combined with the above analysis of S501-S503, it can be seen that in this embodiment, by directly selecting a sub-model that matches the target processing resources from the large model, such as a sub-model that meets various computing resource requirements, without the need for other additional training processes and computing overheads, training resources are saved. Moreover, the image processing system can automatically adapt to the computing power and memory limitations of different devices. This means that even low-performance devices can efficiently complete image processing tasks, improving the popularity and user experience of the image processing system. Since the corresponding sub-model can be adapted, this ensures that when processing images, the algorithm can make full use of existing computing resources and avoid waste of resources.

[0196] It is worth noting that the above examples are only used to illustrate the possible implementation methods of the pre-training method and image processing method of the network model of this specification, and cannot be understood as limiting the implementation methods of the pre-training method and image processing method of the network model of this specification. Exemplarily, based on the above technical concept, some of the above technical features can be combined to obtain a new embodiment; new technical features can be added on the basis of the above examples to obtain a new embodiment; some technical features can be reduced on the basis of the above examples to obtain a new embodiment; some technical features in the above examples can be replaced with other technical features; some technical features and the order of the above examples can be adjusted to obtain a new embodiment, etc., which will not be listed here one by one.

[0197] Based on the above technical concept, this specification also provides a computer-readable non-temporary storage medium, which stores at least one instruction set. When the at least one instruction set is executed by a processor, the steps of the network model pre-training method and image processing method described in this specification are implemented.

[0198] In some possible implementations, various aspects of this specification may also be implemented in the form of a program product, which includes a program code. Taking the program product running on the pre-training system 200 as an example, the program code is used to enable the pre-training system 200 to perform the steps of the pre-training method of the network model described in this specification. The program product for implementing the above method can use a portable compact disk read-only memory (CD-ROM) to include program code and can be run on the pre-training system 200. However, the program product of this specification is not limited to this. In this specification, the readable storage medium can be any tangible medium containing or storing a program, which can be used by the instruction execution system or used in combination with it. The program product can use any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of readable storage media include: an electrical connection with one or more conductors, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. The computer-readable storage medium may include a data signal propagated in a baseband or as part of a carrier wave, wherein a readable program code is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The readable storage medium may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, device, or device. The program code contained on the readable storage medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination of the above. Program code for performing the operations of the present specification may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the pre-training system 200, partially on the pre-training system 200, as a stand-alone software package, partially on the pre-training system 200 and partially on a remote pre-training system, or entirely on the remote pre-training system 200.

[0199] Regarding the operating principle of the program product on the image processing system, please refer to the operating principle of the program product on the pre-training system 200, which will not be repeated here.

[0200] The above is a description of a specific embodiment of the present specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require a specific order or a continuous order to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0201] In summary, after reading this detailed disclosure, those skilled in the art will appreciate that the foregoing detailed disclosure may be presented only by way of example and may not be limiting. Although not explicitly stated herein, those skilled in the art will appreciate that this specification requires various reasonable changes, improvements and modifications to the embodiments. These changes, improvements and modifications are intended to be proposed by this specification and are within the spirit and scope of the exemplary embodiments of this specification.

[0202] In addition, certain terms in this specification have been used to describe embodiments of this specification. For example, "one embodiment", "an embodiment" and / or "some embodiments" mean that a particular feature, structure or characteristic described in conjunction with the embodiment may be included in at least one embodiment of this specification. Therefore, it can be emphasized and should be understood that two or more references to "an embodiment" or "one embodiment" or "an alternative embodiment" in various parts of this specification do not necessarily refer to the same embodiment. In addition, specific features, structures or characteristics may be appropriately combined in one or more embodiments of this specification.

[0203] It should be understood that in the foregoing description of the embodiments of this specification, in order to help understand a feature and for the purpose of simplifying this specification, this specification combines various features in a single embodiment, figure or its description. However, this does not mean that the combination of these features is necessary. When reading this specification, it is entirely possible for a person skilled in the art to mark out some of the devices as separate embodiments. In other words, the embodiments in this specification can also be understood as the integration of multiple secondary embodiments. And the content of each secondary embodiment is also valid when it is less than all the features of a single aforementioned disclosed embodiment.

[0204] Each patent, patent application, publication of patent applications, and other materials, such as articles, books, specifications, publications, documents, articles, etc., cited herein, except to the extent that it is inconsistent or conflicting with this document or that has a limiting effect on the broadest scope of the claims, may be incorporated herein by reference and used for all purposes now or hereafter associated with this document. In addition, in the event of any inconsistency or conflict between the description, definition, and / or use of a term in any material and the description, definition, and / or use of a term in this document, the term in this document shall prevail.

[0205] Finally, it should be understood that the embodiments of the application disclosed herein are explanations of the principles of the embodiments of this specification. Other modified embodiments are also within the scope of this specification. Therefore, the embodiments disclosed in this specification are only used as examples and not as limitations. Those skilled in the art can adopt alternative configurations according to the embodiments in this specification to implement the applications in this specification. Therefore, the embodiments of this specification are not limited to the embodiments accurately described in the application.

Claims

1. A pre-training method for a network model, the method being applied to a preset network, the preset network comprising a teacher model, a complete student model, and a flexible student model; The complete student model and the elastic student model share model parameters; the method includes multiple iterations of training, in which the elastic student model is made into a sub-network of the complete student model based on different hyperparameters in different iterations of training, and in the nth iteration of training, includes: Obtain a first-perspective image and a second-perspective image of the sample object; Inputting the first perspective image into the teacher model to obtain a first prediction result output by the teacher model; Inputting the second perspective image into the complete student model and the elastic student model respectively, obtaining a second prediction result output by the complete student model and a third prediction result output by the elastic student model; Updating preset network parameters with the training objective of minimizing the difference between the first prediction result and the second prediction result, the difference between the first prediction result and the third prediction result, and the difference between the second prediction result and the third prediction result; The teacher model obtained by the method is used to provide models corresponding to the different hyperparameters.

2. The method according to claim 1, wherein: The complete student model and the elastic student model respectively include a backbone network, and the complete student model and the elastic student model share a head structure Head; the second perspective image is input to the complete student model and the elastic student model respectively, and a second prediction result output by the complete student model and a third prediction result output by the elastic student model are obtained, including: Inputting the second perspective image into the backbone network of the complete student model, obtaining a second image feature output by the backbone network of the complete student model, and obtaining a second prediction result output by the common Head based on the second image feature; The second perspective image is input into the backbone network of the elastic student model to obtain the third image feature output by the backbone network of the elastic student model, and the third prediction result output by the common Head based on the third image feature is obtained.

3. The method according to claim 1, wherein: The updating of preset network parameters with minimizing the difference between the first prediction result and the second prediction result, the difference between the first prediction result and the third prediction result, and the difference between the second prediction result and the third prediction result as the training objective includes: updating the shared model parameters with the training objective of minimizing the difference between the first prediction result and the second prediction result, the difference between the first prediction result and the third prediction result, and the difference between the second prediction result and the third prediction result; and The teacher model parameters are updated according to the updated shared model parameters.

4. The method according to claim 3, wherein: The updating of the teacher model parameters according to the updated shared model parameters comprises: Based on the exponential moving average method, the updated shared model parameters are updated to the teacher model to update the teacher model parameters.

5. The method according to claim 1, wherein: The teacher model and the complete student model have the same network structure; the hyperparameters include network width and network depth.

6. The method according to claim 1, wherein: The step of inputting the first perspective image into the teacher model to obtain a first prediction result output by the teacher model includes: Extracting features from the first viewing angle image to obtain first image features; and The first image feature is preprocessed to obtain the first prediction result, wherein the preprocessing includes at least one of feature conversion, centering processing, and normalization processing.

7. An image processing method, comprising: Obtaining an image to be processed and a target processing resource for processing the image to be processed; Obtaining a target model corresponding to the target processing resource from a pre-trained teacher model, wherein the teacher model is pre-trained based on the method according to any one of claims 1 to 6, and is used to provide models corresponding to different hyperparameters, wherein the models corresponding to the different hyperparameters support different processing resources; as well as The image to be processed is processed based on the target model to obtain a processing result.

8. The method according to claim 7, wherein: The step of obtaining a target model corresponding to the target processing resource from the pre-trained teacher model includes: determining, from the different hyperparameters, a target hyperparameter corresponding to the target processing resource; and The target model corresponding to the target hyperparameters is extracted from the teacher model.

9. A pre-training system for a network model, comprising: At least one storage medium storing at least one set of instructions for performing pre-training of a network model; At least one processor is communicatively connected to the at least one storage medium, wherein when the at least one processor executes the at least one set of instruction sets, the method according to any one of claims 1 to 6 is implemented.

10. An image processing system, comprising: at least one storage medium storing at least one set of instructions for performing image processing; At least one processor is communicatively connected to the at least one storage medium, wherein when the at least one processor executes the at least one set of instruction sets, the method according to claim 7 or 8 is implemented.

Citation Information

Patent Citations

  • Image recognition method and system based on improved PATE

    CN114078203A

  • Model processing method, device and equipment

    CN116306868A

  • System and method for training machine learning models with teacher and student frameworks

    CN118279638A

  • Private deep learning

    US20200364542A1