A multi-device cooperative efficient online adaptation method

Abstract

The application discloses a kind of multi-device cooperation's efficient online adaptation method, comprising: for the device of rich computing resource, using the online cooperative adaptation mode based on forward and backward propagation to process first model in device: when detecting domain offset, dynamically save the weight learned by model to shared knowledge base;By introducing new parameter matrix in the first model parameter, to capture new knowledge in the adaptation process;For low-power device, using only online cooperative adaptation mode of forward propagation to process second model in device: model directly uses existing shared knowledge base for online adaptation.The application realizes the accumulation, sharing and dynamic utilization of knowledge among multiple devices by using dynamically updated shared knowledge base;For different power devices, different online cooperative adaptation modes are used for processing, which greatly improves the energy efficiency and practicality of online adaptation in real-world applications.The application can be widely applied in the field of transfer learning technology.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of transfer learning, and in particular to a multi-device cooperative efficient online adaptation method. BACKGROUND

[0002] Existing deep learning relies on the assumption that training and test data are independent and identically distributed, which is difficult to cope with unknown data distribution shift in dynamic scenarios, and is prone to cause a significant decline in the performance of deep models. Therefore, how to adapt to the test data online and achieve stable generalization in dynamic scenarios is a key prerequisite for the deployment and application of deep models in smart city, smart transportation and other scenarios.

[0003] Online adaptation technology aims to adapt to the scene only by using unlabeled online data. Related methods mainly include: 1) online adaptation based on self-supervision; and 2) online adaptation based on unsupervised learning. The online adaptation method based on self-supervision mainly focuses on learning information reconstruction from scene data, that is, by reducing the reconstruction loss of the scene online, the deep model understands and generalizes to the new application scenario; however, this method introduces an additional self-supervised branch, which significantly increases the training and deployment cost of the model, and is inefficient. The unsupervised online adaptation method is based on Shannon entropy analysis and optimization of model decision uncertainty, which helps the model to form a clearer decision boundary in the new scene, thereby improving the cross-scene decision-making ability of the model. Some methods further analyze the influence of different samples on online adaptation, construct a reliable and non-redundant active sample screening mechanism, and solve the online adaptation problem only on the reserved sample subset, thereby improving the performance and efficiency of adaptation.

[0004] However, in actual applications, there are often multiple collaborative devices, such as smartphones, surveillance cameras, unmanned vehicles, etc. However, existing online adaptation methods are limited to independent adaptation of each single device, and the efficiency and performance are limited in actual applications, which have the following bottlenecks: 1) there is a high degree of redundancy in the adaptation of similar scenes by different devices, and the computational resources are wasted, resulting in poor efficiency and performance of online adaptation; 2) existing online adaptation methods are prone to catastrophic forgetting, and it is difficult to accumulate the knowledge learned from adaptation, which cannot support knowledge sharing and utilization across devices, making it difficult for multiple devices to cooperate; 3) online adaptation learning relies on the backward propagation mechanism, which is computationally expensive, but edge devices have limited computational power, and the contradiction between the two is prominent, making it difficult to apply online adaptation technology to edge devices. SUMMARY

[0005] To at least partially solve one of the technical problems in the prior art, the purpose of the present application is to provide a multi-device cooperative efficient online adaptation method, an electronic device and a medium.

[0006] The first technical solution adopted by the present application is:

[0007] A multi-device cooperative efficient online adaptation method is used to realize accumulation, sharing and dynamic utilization of knowledge among multiple devices, different devices share the same shared knowledge base, including the following steps:

[0008] For devices with rich computing resources, the first model in the device is processed using an online cooperative adaptation mode based on forward and backward propagation: when domain drift is detected, the learned weights of the model are dynamically saved to the shared knowledge base, realizing knowledge accumulation without forgetting in cross-domain learning; by introducing a new parameter matrix △θ new in the first model parameters, new knowledge is captured during adaptation, thereby decoupling the knowledge reuse and new knowledge learning processes;

[0009] For low-power edge devices, the second model in the device is processed using an online cooperative adaptation mode that only uses forward propagation: the model directly uses the existing shared knowledge base for online adaptation, thereby avoiding the problem of high cost of knowledge learning computing power.

[0010] It should be noted that the first model in the above refers to the model deployed in the device with rich computing resources, and the second model refers to the model deployed in the low-power device, i.e., the models deployed in the device with rich computing resources and the low-power edge device are distinguished by "first" and "second".

[0011] Further, the expression of the parameters of the first model is:

[0012]

[0013] In the formula, θ 0 is the original parameter of the model; is the M pieces of knowledge accumulated in the shared knowledge base shared by multiple devices; α i is the normalized learnable weighting coefficient of different knowledge, i.e. △θ new is a newly introduced learnable parameter matrix.

[0014] Further, the expression of the parameters of the second model is:

[0015]

[0016] In the formula, θ 0 is the original parameter of the model, is the M pieces of knowledge accumulated in the shared knowledge base shared by multiple devices; γ i is the normalized learnable weighting coefficient of different knowledge, i.e. is used to support shared knowledge reuse between devices.

[0017] Further, the first model in the device is processed by adopting the online collaborative adaptation mode based on forward and backward propagation, comprising:

[0018] a batch of target images to be adapted inputting the first model in the device;

[0019] detecting whether domain shift occurs at present, and saving learned knowledge to a shared knowledge base if domain shift occurs

[0020] generating a prediction result according to the input image, updating the model by using an online optimization target and a backward propagation mechanism, until all image data are processed, and the model stops online adaptation.

[0021] Further, the detection of whether domain shift occurs at present, and saving learned knowledge to a shared knowledge base if domain shift occurs comprises:

[0022] using output features of the model backbone network to measure the distribution of the current input image data

[0023] calculating the distribution of the current input image data and a historical data distribution φ d

[0024] if a difference between the distribution of the current input image data and the historical data distribution φ is greater than a preset threshold, it is determined that domain shift exists in the current input image data, learned knowledge △θ M+1 and a corresponding domain distribution φ d are saved to the shared knowledge base

[0025] Further, the calculation formula of the distribution of the current input image data is as follows:

[0026]

[0027] In the formula, B(·) is an output feature of the backbone network, x i is input data, and N is a data quantity; and are a mean and a variance of the feature, which are composed of and

[0028] Further, the historical data distribution φ d is a moving average of the distribution of the input image data, and the expression is as follows:

[0029] ​​​

[0030] where λ is the equilibrium factor;

[0031] KL divergence is used to measure the difference between the image data distribution and the historical data distribution φ d .

[0032] The expression of learned knowledge △θ M+1 is:

[0033]

[0034] where, is the cumulative M pieces of knowledge in the shared knowledge base shared by multiple devices; α i is the normalized learnable weighting coefficient of different knowledge, that is, △θ new is the newly introduced learnable parameter matrix.

[0035] Further, the expression of model updating using online optimization target and back propagation mechanism is:

[0036]

[0037] where, represents any online self-supervised or unsupervised optimization target; θ is the parameter of the first model; x is the target data set to be adapted; α is the normalized learnable weighting coefficient vector;

[0038] where different devices can adjust α, △θ new according to their own input data to achieve efficient adaptation to application scenarios.

[0039] Further, for low-power edge devices, an online collaborative adaptation mode using only forward propagation is used to process the second model in the device, including:

[0040] a batch of target images to be adapted are input into the second model in the device;

[0041] calculate the similarity between the distribution of the current input image data and the distribution in the shared knowledge base;

[0042] According to the calculated similarity, the learned knowledge in the similar domain is obtained from the shared knowledge base to update the model until all image data are processed, and the model stops online adaptation.

[0043] Further, the calculation formula of the similarity is as follows:

[0044]

[0045] where D(·,·) is the difference of data distribution; φ i is the △θ i data distribution information φ d at learning time; ∈ is a small quantity to prevent numerical overflow.

[0046] The second technical solution adopted by the present application is:

[0047] An electronic device, comprising a processor and a memory, the memory storing at least one instruction, at least one program, a code set or an instruction set, the at least one instruction, the at least one program, the code set or the instruction set being loaded and executed by the processor to implement the efficient online adaptation method of multi-device cooperation as described above.

[0048] The third technical solution adopted by the present application is:

[0049] A computer readable storage medium, the storage medium storing at least one instruction, at least one program, a code set or an instruction set, the at least one instruction, the at least one program, the code set or the instruction set being loaded and executed by a processor to implement the efficient online adaptation method of multi-device cooperation as described above.

[0050] The fourth technical solution adopted by the present application is:

[0051] A computer program product or computer program, the computer program product or computer program comprising computer instructions stored in a computer readable storage medium. The processor of the computer device can read the computer instructions from the computer readable storage medium, and the processor executes the computer instructions, so that the computer device executes the above method.

[0052] The beneficial effects of the present application are: the present application proposes an efficient online adaptation framework of multi-device cooperation, adopts a dynamically updated shared knowledge base, realizes the accumulation, sharing and dynamic utilization of knowledge among multiple devices; in addition, different online cooperative adaptation modes are adopted to process different computing power devices, greatly improving the energy efficiency and practicality of online adaptation in real application. BRIEF DESCRIPTION OF DRAWINGS

[0053] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following introduces the drawings of the related technical solutions in the embodiments of the present application or the prior art. It should be understood that the drawings in the following introduction are only for the convenience of clearly describing part of the embodiments in the technical solutions of the present application, and for those skilled in the art, other drawings can also be obtained from these drawings without creative labor.

[0054] Figure 1 is a flow diagram of a high-efficiency online adaptation method of multi-device cooperation in an embodiment of the present application;

[0055] Figure 2 is a flow diagram of an online cooperative adaptation mode based on forward and backward propagation in an embodiment of the present application;

[0056] Figure 3 is a flow diagram of an online cooperative adaptation mode using only forward propagation in an embodiment of the present application. DETAILED DESCRIPTION

[0057] Embodiments of the present application are described in detail below with reference to the accompanying drawings, in which the same or similar elements or elements having the same or similar functions are denoted by the same reference numerals throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are intended only for the purpose of explaining the present application, and should not be understood as limiting the present application. For the step numbers in the following embodiments, they are only set for the convenience of explanation, and the order between the steps is not limited in any way, and the execution order of each step in the embodiments can be adaptively adjusted according to the understanding of those skilled in the art.

[0058] In the description of the present application, it should be understood that the orientation description, such as the orientation or position relationship indicated by up, down, front, back, left, right, etc. is based on the orientation or position relationship shown in the drawings, and is only for the convenience of describing the present application and simplifying the description, and does not indicate or imply that the device or element referred to must have a particular orientation, be constructed and operated in a particular orientation, and therefore should not be understood as limiting the present application.

[0059] In the description of the present application, several meanings are one or more, and the meaning of multiple is two or more, greater than, less than, more than, etc. are understood as not including the number, above, below, etc. are understood as including the number. If it is described as first, second, it is only for the purpose of distinguishing technical features, and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated or the order of technical features indicated.

[0060] In the description of the present application, unless otherwise explicitly limited, the words such as setting, installing, connecting, etc. should be broadly understood, and those skilled in the art can reasonably determine the specific meaning of the above words in the present application in combination with the specific content of the technical solution.

[0061] To solve the existing technical problems, the application provides a multi-device cooperative efficient online adaptation framework, constructs a parameter dynamic saving mechanism based on a shared knowledge base, realizes accumulation, sharing and dynamic utilization of knowledge among multiple devices, and further provides an online cooperative adaptation method based on forward and backward propagation and only based on forward propagation, to improve the energy efficiency and practicality of online adaptation in real applications.

[0062] Embodiment 1

[0063] As Figure 1 shown, the embodiment provides a multi-device cooperative efficient online adaptation method for realizing accumulation, sharing and dynamic utilization of knowledge among multiple devices, different devices sharing the same shared knowledge base, including the following steps:

[0064] For devices with rich computing resources, an online cooperative adaptation mode based on forward and backward propagation is used to process the first model in the device: when domain shift is detected, the learned weights of the model are dynamically saved to the shared knowledge base, realizing knowledge accumulation without forgetting in cross-domain learning; by introducing a new parameter matrix △θ new in the first model parameters, new knowledge is captured in the adaptation process, decoupling the knowledge reuse and new knowledge learning processes;

[0065] For edge devices with low computing power, an online cooperative adaptation mode using only forward propagation is used to process the second model in the device: the model directly uses the existing shared knowledge base for online adaptation, thereby avoiding the problem of high cost of knowledge learning computing power.

[0066] In this embodiment, the online cooperative adaptation mode based on forward and backward propagation is aimed at devices with rich computing resources, involving the learning, accumulation, sharing and utilization process of cross-domain knowledge. In this mode, a dynamic knowledge saving mechanism based on the knowledge base is constructed, the difference between the historical data distribution and the current test data distribution is measured, and the learned weights are dynamically saved to the knowledge base when the domain shifts, realizing knowledge accumulation without forgetting in cross-domain learning. Among them, different devices share the same knowledge base, effectively supporting cross-device knowledge sharing and subsequent knowledge utilization, solving the problem of catastrophic forgetting in the existing technical solutions. Secondly, a weighting coefficient is introduced for each shared knowledge, converting the reuse problem of shared knowledge into the adaptive learning problem of the weighting coefficient. Further, a new parameter matrix △θ new is introduced to capture new knowledge in the adaptation process, decoupling the knowledge reuse and new knowledge learning processes, and alleviating the optimization conflict between the two processes. The above parameters can be optimized using online optimization objectives and a backward propagation mechanism, realizing dynamic utilization of knowledge among multiple devices, improving the efficiency and performance of adaptation, and solving the problems of high redundancy in adaptation of different devices to similar scenarios, serious waste of computing resources, and poor efficiency and performance of online adaptation in the existing technical solutions.

[0067] In the embodiment, the online collaborative adaptation mode using only forward propagation is used for low-power edge devices, involving the process of utilizing shared knowledge. The main idea is to directly utilize the existing shared knowledge for online adaptation, thereby avoiding the problem of high cost of knowledge learning power. In this mode, the embodiment approximates the weighted coefficient optimization problem of knowledge utilization as a domain matching process, based on the similarity analysis of data distribution, and gives higher utilization weight to the knowledge learned when adapting in similar domains, realizing online dynamic adaptation without optimization. Based on this mode, the power cost of online adaptation is reduced, solving the problem that the application of online adaptation technology on edge devices is difficult due to the limited power of edge devices in the prior art; thus, the online adaptation technology is effectively expanded to edge devices such as smart phones with limited computing resources.

[0068] The embodiment of the application innovatively expands single-device independent online adaptation to efficient online adaptation of multiple-device collaboration, breaks through the catastrophic forgetting problem of online adaptation, realizes the accumulation, sharing and dynamic utilization of knowledge among multiple devices, improves the efficiency and performance of online adaptation, and makes the costly online adaptation technology applicable to resource-constrained edge devices. The embodiment of the application has the characteristics of low communication cost and high scalability, which will have an important impact on model deployment and online adaptation technology widely used in edge computing, Internet of Things and distributed systems, and improve the energy efficiency and practicality of online adaptation of models in applications.

[0069] The above method content will be explained and described in detail in combination with the drawings and specific embodiments.

[0070] (1) Online collaborative adaptation mode based on forward and backward propagation

[0071] This mode is aimed at devices with rich computing power resources, involving the learning, accumulation, sharing and utilization process of cross-domain knowledge. As an embodiment, see Figure 2 On a device using this mode, the algorithm includes the following steps:

[0072] Step one: input a batch of target images to be adapted The formal expression of model parameters θ is as follows:

[0073]

[0074] Where θ 0 is the original model parameter, is the accumulated M knowledge in the knowledge base shared by multiple devices, and α i is the normalized learnable weighting coefficient of different knowledge, i.e. △θ new is a newly introduced learnable parameter matrix. The above scheme decouples the utilization and update of knowledge into α,△θnew The learning process effectively supports the cross-device knowledge utilization and learning of new knowledge.

[0075] Step two: detect whether the domain shift occurs, and if so, save the learned knowledge to the shared knowledge base

[0076] Specifically, the domain detection shift includes the following processes:

[0077] 1) Use the model backbone network output feature to measure the current input data distribution The calculation formula is as follows:

[0078]

[0079] Where B(·) is the backbone network output feature, and is the mean and variance of the feature, which is composed of and .

[0080] 2) Measure the difference between the current input data distribution and the historical data distribution and. The historical data distribution d is the moving average of the input data distribution, which is formally expressed as:

[0081]

[0082] Where λ is the balance factor, which can be set to 0.8 in the implementation. The difference measure D(·,·) of the data distribution is the KL divergence between the data distributions under the assumption that the data distribution follows the Gaussian distribution, and the calculation formula is as follows:

[0083]

[0084] Where H is the dimension of the statistical quantity. When the data distribution difference is greater than a certain threshold, i.e. , it is considered that the current input data has domain shift. The above domain shift detection scheme does not need to save the original data, and the calculation overhead and storage cost are low, which can effectively maintain the efficiency of online adaptation.

[0085] Specifically, the learned knowledge M+1 and the corresponding domain distribution d are saved to the shared knowledge base The formal representation of the learned knowledge is as follows:

[0086]

[0087] The knowledge is immediately shared with other devices after saving, and an additional learnable weighting coefficient α is introduced into the model parameter θM+1 , the formalized representation of parameter θ is updated as: Since △θ M+1 , the catastrophic forgetting problem under continuous online adaptation is effectively alleviated, the gradual accumulation of knowledge is realized, and the stability of online adaptation is improved. Further, the utilization and updating of knowledge are decoupled as α, △θ new , the learning process has the following benefits: a) frequent updating of can be avoided, effectively reducing the expensive communication cost caused by cross-device synchronization, realizing efficient communication of multi-device collaborative online adaptation; b) multi-device collaboration is based only on the knowledge base , any device can join or exit the collaborative group at any time (using the latest or non-latest shared knowledge base), realizing completely asynchronous multi-device collaborative adaptation, and improving the practicality and expandability of the invented patent in real-world scenarios.

[0088] Step three: generate prediction results, use online optimization objectives and backpropagation mechanism for model updating, which is formally represented as:

[0089]

[0090] where represents any online self-supervised or unsupervised optimization objective. This patent focuses on breaking through the collaborative bottleneck of multi-device online adaptation, and can be flexibly combined with existing advanced online optimization objectives. Preferably, the entropy minimization loss can be used for parameter optimization, which is mathematically represented as where f θ (x i ) is the model output result. Based on the above loss for model optimization, the same knowledge base is shared between devices, but different devices can adaptively adjust α, △θ new according to their own input data, realizing efficient adaptation to application scenarios.

[0091] Step four: repeat steps one to three above until all test data is input and the model stops online adaptation.

[0092] (2) Online collaborative adaptation mode using only forward propagation

[0093] This mode is aimed at low-power edge devices and involves the utilization of shared knowledge. As an optional implementation, see Figure 3 , the algorithm on a device using this mode includes the following steps:

[0094] Step one: input a batch of target images to be adapted into the model. The formalized representation of model parameters θ is as follows:

[0095]

[0096] where θ 0 is the original parameter of the model, is the accumulated M pieces of knowledge in the shared knowledge base of multiple devices, γ i is the normalized learnable weighting coefficient of different knowledge, that is, Supports the reuse of shared knowledge across devices.

[0097] Step two: measure the similarity of the current input data distribution with the distribution in the shared knowledge base. Wherein the current input data distribution The calculation formula is the same as above, φ i is the data distribution information φ i of the shared knowledge base when learning △θ d . The data distribution similarity ρ i The calculation formula is as follows:

[0098]

[0099] Where D(·,·) is the data distribution difference measure defined above, ∈ is a small quantity to prevent numerical overflow.

[0100] Step three: based on the distribution similarity measure, preferentially select the knowledge learned from the similar domain in the shared knowledge base. Specifically, the formal definition of γ is:

[0101] γ = softmax(ρ)

[0102] By innovatively converting the model online adaptation problem into a dynamic utilization process of shared knowledge, and approximating the weighting coefficient optimization problem to a domain matching process, the embodiment effectively avoids the high cost problem of algorithm brought by knowledge learning and coefficient optimization, reduces the algorithm cost of online adaptation, realizes the dynamic adaptation without optimization, and thus expands the online adaptation technology with high cost to edge devices such as smart phones with limited computing resources.

[0103] Step four: repeat steps one to three above until all test data is input and the model stops online adaptation.

[0104] In summary, the present application aims to break through the bottleneck that existing online adaptation technology is limited to single-device independent adaptation, the catastrophic forgetting problem is prominent, it is difficult to utilize useful knowledge of other devices, the online adaptation performance and efficiency are poor and it cannot be applied to edge devices. To this end, the present application proposes a multi-device collaborative efficient online adaptation method, which realizes the accumulation, sharing and dynamic utilization of knowledge among multiple devices, and there are two cooperation modes as follows: 1) online collaborative adaptation mode based on forward and backward propagation: the model parameters are reconstructed as , thereby decoupling the reuse and update of knowledge as new , alleviating the optimization conflict of the two processes, supporting the utilization of knowledge across devices and the learning of new knowledge. Further, a dynamic knowledge preservation mechanism based on a knowledge base is constructed, and the learned new knowledge M+1 is saved to the shared knowledge base when the domain shifts, realizing the gradual accumulation of knowledge, alleviating the catastrophic forgetting problem under continuous online adaptation, supporting efficient knowledge sharing across devices, and improving the stability and efficiency of online adaptation. 2) online collaborative adaptation mode using only forward propagation: the model parameters are reconstructed as , thereby dynamically utilizing the shared knowledge of different devices, and transforming the optimization process of i into domain similarity analysis to solve it, effectively avoiding the high computational cost problem caused by knowledge learning and coefficient optimization, thereby expanding the costly online adaptation technology to edge devices such as smartphones with limited computing resources. In summary, the present application innovatively expands single-device independent online adaptation to multi-device collaborative efficient online adaptation, breaks through the catastrophic forgetting problem of online adaptation, realizes the accumulation, sharing and dynamic utilization of knowledge among multiple devices, improves the efficiency and performance of online adaptation, and makes the costly online adaptation technology applicable to resource-limited edge devices. The present application has the characteristics of low communication cost and high scalability, which will have an important impact on model deployment and online adaptation technology widely used in edge computing, Internet of Things and distributed systems, and improve the energy efficiency and practicality of online adaptation of models in applications.

[0105] Embodiment 2

[0106] The embodiments of the present application also provide an electronic device, which comprises a processor and a memory, and the memory stores at least one instruction, at least one program, a code set or an instruction set. The at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by the processor to implement a multi-device collaborative efficient online adaptation method as shown in Figure 1 .

[0107] It can be understood that the memory can include a random access memory (RAM) and can also include a read-only memory (ROM). Optionally, the memory includes a non-transitory computer-readable storage medium. The memory can be used to store instructions, programs, codes, code sets or instruction sets. The memory can include a program storage area and a data storage area, wherein the program storage area can store instructions for implementing an operating system, instructions for at least one function, instructions for implementing various method embodiments described above, etc.; and the data storage area can store data created according to the use of the server, etc.

[0108] The processor can include one or more processing cores. The processor connects various parts within the entire server through various interfaces and lines, executes various functions of the server and processes data by running or executing instructions, programs, code sets or instruction sets stored in the memory, and calling data stored in the memory. Optionally, the processor can be implemented in at least one of a hardware form of a digital signal processing (DSP), a field-programmable gate array (FPGA), and a programmable logic array (PLA). The processor can be integrated with a combination of one or more of a central processing unit (CPU) and a modem. Among them, the CPU mainly processes operating systems and application programs, etc.; and the modem is used to process wireless communication. It can be understood that the above-mentioned modem can also not be integrated into the processor, but can be implemented by a separate chip.

[0109] Since the electronic device is an electronic device corresponding to the efficient online adaptation method of multiple device cooperation of the embodiment of the application, and the principle of solving problems of the electronic device is similar to that of the method, the implementation of the electronic device can be referred to the implementation process of the above-mentioned method embodiments, and the repeated parts will not be described here.

[0110] Embodiment 3

[0111] The embodiment of the application further provides a computer readable storage medium, wherein the storage medium stores at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, the at least one program, the code set or the instruction set are loaded and executed by a processor to implement the efficient online adaptation method of multiple device cooperation as shown in Figure 1

[0112] ​Those skilled in the art can understand that all or part of the steps of various methods in the above embodiments can be completed by instructing the relevant hardware through a program, and the program can be stored in a computer readable storage medium, including Read-Only Memory (ROM), Random Access Memory (RAM), Programmable Read-only Memory (PROM), Erasable Programmable Read Only Memory (EPROM), One-time Programmable Read-Only Memory (OTPROM), Electrically-Erasable Programmable Read-Only Memory (EEPROM), Compact Disc Read-Only Memory (CD-ROM) or other optical disk storage, magnetic disk storage, magnetic tape storage, or any other medium that can be used to carry or store data which can be read by a computer.

[0113] Since the storage medium is a storage medium corresponding to the efficient online adaptation method of the multi-device cooperation of the embodiments of the present application, and the principle of solving the problem is similar to the method, the implementation of the storage medium can refer to the implementation process of the above method embodiments, and the repeated parts will not be described again.

[0114] Embodiment 4

[0115] In some possible implementation manners, various aspects of the method of the embodiments of the present application can also be implemented in the form of a program product, which includes program codes for causing a computer device to execute the steps of the efficient online adaptation method of the multi-device cooperation according to various exemplary embodiments of the present application described above in the specification when the program product is run on the computer device. Wherein, the executable computer program codes or "codes" for executing various embodiments can be written in a high-level programming language such as C, C++, C#, Smalltalk, Java, JavaScript, Visual Basic, Structured Query Language (for example, Transact-SQL), Perl, or in various other programming languages.

[0116] It should be understood that various aspects of the application can be implemented in hardware, software, firmware or a combination of them. In the above embodiments, various steps or methods can be implemented in software or firmware which is stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, and in another embodiment, any of the following technologies, known in the art, or their combination, can be used: discrete logic circuitry having logic gates for implementing logic functions upon an application of data signals, application-specific integrated circuits having appropriate combinational logic gates, programmable gate arrays (PGA), field-programmable gate arrays (FPGA), and so on.

[0117] In the description of the specification, the description of the terms "one embodiment", "some embodiments", "an example", "a specific example", or "some examples" and the like means that the specific features, structures, materials or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present application. In the specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Also, the specific features, structures, materials or characteristics described can be combined in any appropriate manner in any one or more embodiments or examples. In addition, a person skilled in the art can combine and combine the different embodiments or examples described in the specification and the features of the different embodiments or examples, without contradiction.

[0118] The above embodiments are only for the purpose of illustrating the technical concept and characteristics of the present application, and the purpose is to enable those skilled in the art to understand the content of the present application and to implement it, and cannot limit the protection scope of the present application. Any equivalent changes or modifications made in accordance with the essence of the present application should be covered within the protection scope of the present application.

Claims

1. A method for efficient online adaptation of multi-device collaboration, characterized in that, For the accumulation, sharing and dynamic utilization of knowledge among multiple devices, different devices share the same shared knowledge base, including the following steps: For the device with rich computing resources, the first model in the device is processed by using an online collaborative adaptation mode based on forward and backward propagation: when the domain offset is detected, the learned weights of the model are dynamically saved to the shared knowledge base; by introducing a new parameter matrix in the first model parameters, new knowledge is captured in the adaptation process; For the low-power device, the second model in the device is processed by using an online collaborative adaptation mode using only forward propagation: the model directly uses the shared knowledge base for online adaptation; The expression of the parameters of the first model is: In the formula, These are the original parameters of the model; It is accumulated in a shared knowledge base shared by multiple devices. This knowledge; These are the learnable weighting coefficients for different knowledge normalizations, i.e. ; It is a newly introduced learnable parameter matrix; The expression of the parameters of the second model is: wherein, is the original parameter of the model, is the accumulated knowledge in the shared knowledge base shared by multiple devices, is the learnable weighting coefficient of different knowledge, i.e. , for supporting the reuse of shared knowledge across devices.​ The first model in the device is processed by using an online collaborative adaptation mode based on forward and backward propagation, including: a batch of target images to be adapted a first model in an input device; detecting whether a domain shift is currently occurring, and if a domain shift is occurring, saving learned knowledge to a shared knowledge base ; According to the input image, a prediction result is generated, and the model is updated by using an online optimization target and a backward propagation mechanism until all image data are processed, and the model stops online adaptation; For the low-power edge device, the second model in the device is processed by using an online collaborative adaptation mode using only forward propagation, including: a batch of target images to be adapted a second model in an input device; Computing a distribution of image data of a current input Similarity to distribution in shared knowledge base ; According to the calculated similarity, the learned knowledge in the similar domain is obtained from the shared knowledge base for model updating until all image data are processed, and the model stops online adaptation.

2. The method of claim 1, wherein, detecting whether a domain shift is currently occurring, and if a domain shift is occurring, saving learned knowledge to a shared knowledge base comprising: Utilize output features of a model backbone network to measure current input image data distribution ; computing a difference between the current input image data distribution and the historical data distribution ;​ If the difference is greater than a preset threshold, it is determined that the current input image data has a domain shift, and the learned knowledge is saved to the shared knowledge base corresponding to the domain distribution .

3. The method of claim 2, wherein, the current input image data distribution The formula for the calculation is as follows: wherein, is a backbone network output feature, is an input data, is a data volume; with is a mean and variance of a feature, by with consists of.

4. The method of claim 2, wherein, said historical data distribution The moving average of the input image data distribution is expressed as: wherein is the equilibrium factor; measuring the difference between the image data distribution by KL divergence from the historical data distribution ​ knowledge The expression of the knowledge is: wherein, is the accumulated knowledge in the shared knowledge base shared by multiple devices ; is the normalized knowledge of different knowledge, i.e. ; is the newly introduced learnable parameter matrix.

5. The method of claim 1, wherein, The expression of the model updating by using an online optimization target and a backward propagation mechanism is: wherein represents an arbitrary online self-supervised or unsupervised optimization objective; are parameters of the first model; is a target dataset to be adapted; is a normalized vector of learnable weighting coefficients; Among them, different devices can adaptively adjust according to the data input by themselves To achieve efficient adaptation to application scenarios.

6. The method of claim 1, wherein, The calculation formula of the similarity is as follows: In the formula, is the difference of data distribution; is the difference of data distribution; is the difference of data distribution; is the difference of data distribution; is a small quantity to prevent numerical overflow.

Images