Multi-device collaborative efficient online adaptation method

Abstract

The invention discloses a multi-device collaborative efficient online adaptation method, which comprises the following steps: for devices with rich computing power resources, processing a first model in the devices by adopting an online collaborative adaptation mode based on forward and back propagation: when domain offset is detected, dynamically storing weights learned by the models into a shared knowledge base; new knowledge is captured in the adaptation process by introducing a new parameter matrix into the first model parameters; and for equipment with low computing power, processing a second model in the equipment by adopting an online collaborative adaptation mode which only utilizes forward propagation, and carrying out online adaptation on the model by directly utilizing an existing shared knowledge base. According to the method, the knowledge base is dynamically updated and shared, so that accumulation, sharing and dynamic utilization of knowledge among multiple devices are realized; and different online cooperative adaptation modes are adopted for processing for equipment with different computing power, so that the energy efficiency and practicability of online adaptation in practical application are greatly improved. The method can be widely applied to the technical field of transfer learning.

Description

Technical Field

[0001] The present invention relates to the technical field of transfer learning, and in particular to an efficient online adaptation method for multi-device collaboration. Background Art

[0002] Existing deep learning relies on the assumption that training and test data are independent and identically distributed, which makes it difficult to cope with the unknown data distribution offset in dynamically changing scenarios, which can easily lead to a significant drop in deep model performance. Therefore, how to adapt to test data online and achieve stable generalization in dynamic scenarios is a key prerequisite for the deployment and application of deep models in scenarios such as smart cities and smart transportation.

[0003] Online adaptation technology aims to adapt to scenarios using only unlabeled online data. The main methods include: 1) online adaptation based on self-supervision; 2) online adaptation based on unsupervision. Self-supervision-based online adaptation methods focus on information reconstruction learning from scene data, that is, by reducing the reconstruction loss of the scene online, the deep model can understand and generalize to new application scenarios; however, this type of method introduces additional self-supervision branches, which significantly increases the training and deployment costs of the model and is inefficient. Unsupervised online adaptation methods analyze and optimize the decision uncertainty of the model based on Shannon entropy, helping the model to form clearer decision boundaries in new scenarios, thereby improving the model's cross-scenario decision-making ability. Some methods further analyze the impact of different samples on online adaptation, construct a reliable and non-redundant active sample screening mechanism, and solve the online adaptation problem only on the retained sample subset, improving the performance and efficiency of adaptation.

[0004] However, in practical applications, there are often multiple cooperating devices, such as smartphones, surveillance cameras, and driverless cars. However, existing online adaptation methods are limited to the independent adaptation of a single device, and the efficiency and performance in practical applications are limited. There are the following bottlenecks: 1) There is a high degree of redundancy in the adaptation of different devices to similar scenes, which seriously wastes computing resources and leads to poor efficiency and performance of online adaptation; 2) Existing online adaptation methods are prone to catastrophic forgetting, making it difficult to accumulate the learned knowledge, unable to support cross-device knowledge sharing and utilization, and difficult to collaborate with multiple devices; 3) Online adaptive learning relies on the back-propagation mechanism with high computing cost, but the computing power of edge devices is limited, and the contradiction between the two is prominent, making it difficult to apply online adaptation technology to edge devices. Summary of the invention

[0005] In order to solve at least one of the technical problems existing in the prior art to a certain extent, the object of the present invention is to provide an efficient online adaptation method, electronic device and medium for multi-device collaboration.

[0006] The first technical solution adopted by the present invention is:

[0007] An efficient online adaptation method for multi-device collaboration is used to realize the accumulation, sharing and dynamic utilization of knowledge among multiple devices. Different devices share the same shared knowledge base, including the following steps:

[0008] For devices with abundant computing resources, the first model in the device is processed using an online collaborative adaptation mode based on forward and back propagation: when a domain shift is detected, the weights learned by the model are dynamically saved to a shared knowledge base, achieving knowledge accumulation without forgetting in cross-domain learning; by introducing a new parameter matrix △θ in the first model parameters new , to capture new knowledge during the adaptation process, thereby decoupling the knowledge reuse and new knowledge learning processes;

[0009] For edge devices with low computing power, an online collaborative adaptation mode that only uses forward propagation is used to process the second model in the device: the model directly uses the existing shared knowledge base for online adaptation, thereby avoiding the problem of high computing cost of knowledge learning.

[0010] It should be noted that the first model mentioned above refers to the model deployed in devices with abundant computing resources, and the second model refers to the model deployed in devices with low computing power. That is, "first" and "second" are used to distinguish between models deployed in devices with abundant computing resources and edge devices with low computing power.

[0011] Furthermore, the expression of the parameters of the first model is:

[0012]

[0013] In the formula, θ 0 are the original parameters of the model; is the M pieces of knowledge accumulated in the shared knowledge base shared by multiple devices; α i is the learnable weight coefficient for normalizing different knowledge, that is, △θ new is a newly introduced learnable parameter matrix.

[0014] Furthermore, the expression of the parameters of the second model is:

[0015]

[0016] In the formula, θ 0 are the original parameters of the model, is the M pieces of knowledge accumulated in the shared knowledge base shared by multiple devices; γ i is the learnable weight coefficient for normalizing different knowledge, that is, Used to support shared knowledge reuse across devices.

[0017] Furthermore, the processing of the first model in the device using an online collaborative adaptation mode based on forward and backward propagation includes:

[0018] A batch of target images to be adapted a first model in the input device;

[0019] Detect whether domain shift has occurred. If so, save the learned knowledge to the shared knowledge base.

[0020] The prediction results are generated based on the input image, and the model is updated using the online optimization target and back-propagation mechanism until all image data are processed and the model stops online adaptation.

[0021] Furthermore, the detection is performed to determine whether a domain shift has occurred. If a domain shift has occurred, the learned knowledge is saved to a shared knowledge base. include:

[0022] Use the output features of the model backbone network to measure the distribution of the current input image data

[0023] Calculate the current input image data distribution With historical data distribution φ d The Difference

[0024] If the difference If it is greater than the preset threshold, it is determined that the current input image data has domain shift, and the learned knowledge △θ M+1 And the corresponding domain distribution φ d Save to shared repository

[0025] Furthermore, the currently input image data distribution The calculation formula is as follows:

[0026]

[0027] In the formula, B(·) is the output feature of the backbone network, x i is the input data, N is the amount of data; and is the mean and variance of the feature, Depend on and composition.

[0028] Furthermore, the historical data distribution φ d is the moving average of the input image data distribution, and the expression is:

[0029]

[0030] Where λ is the balance factor;

[0031] Measuring image data distribution through KL divergence With historical data distribution φ d The difference between

[0032] Knowledge learned △θ M+1 The expression is:

[0033]

[0034] In the formula, is the M pieces of knowledge accumulated in the shared knowledge base shared by multiple devices; α i is the learnable weight coefficient for normalizing different knowledge, that is, △θ new is a newly introduced learnable parameter matrix.

[0035] Furthermore, the expression for updating the model using the online optimization objective and the back-propagation mechanism is:

[0036]

[0037] In the formula, represents any online self-supervised or unsupervised optimization objective; θ is the parameter of the first model; x is the target dataset to be adapted; α is the normalized learnable weight coefficient vector;

[0038] Among them, different devices can adaptively adjust α, △θ according to their own input data new , in order to achieve efficient adaptation to application scenarios.

[0039] Furthermore, for edge devices with low computing power, the second model in the device is processed using an online collaborative adaptation mode that only uses forward propagation, including:

[0040] A batch of target images to be adapted a second model in the input device;

[0041] Calculate the current input image data distribution Distributed in a shared knowledge base similarity;

[0042] According to the calculated similarity, the knowledge learned from similar domains is obtained from the shared knowledge base to update the model until all image data are processed and the model stops online adaptation.

[0043] Furthermore, the calculation formula of the similarity is as follows:

[0044]

[0045] Where D(·,·) is the difference in data distribution; φ i is △θ in the shared knowledge base i Data distribution information during learning φ d ;∈ is a very small amount to prevent numerical overflow.

[0046] The second technical solution adopted by the present invention is:

[0047] An electronic device comprises a processor and a memory, wherein the memory stores at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by the processor to implement an efficient online adaptation method for multi-device collaboration as described above.

[0048] The third technical solution adopted by the present invention is:

[0049] A computer-readable storage medium stores at least one instruction, at least one program, a code set or an instruction set, wherein the at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by a processor to implement an efficient online adaptation method for multi-device collaboration as described above.

[0050] The fourth technical solution adopted by the present invention is:

[0051] A computer program product or a computer program includes computer instructions stored in a computer-readable storage medium. A processor of a computer device can read the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the above method.

[0052] The beneficial effects of the present invention are as follows: the present invention proposes an efficient online adaptation framework for multi-device collaboration, and adopts a dynamically updated shared knowledge base to realize the accumulation, sharing and dynamic utilization of knowledge among multiple devices; in addition, different online collaborative adaptation modes are adopted to process devices with different computing powers, which greatly improves the energy efficiency and practicality of online adaptation in real applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the embodiments of the present invention or the drawings of related technical solutions in the prior art are introduced below. It should be understood that the drawings introduced below are only for the convenience of clearly describing some embodiments of the technical solutions of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0054] Figure 1 is a flow chart of an efficient online adaptation method for multi-device collaboration in an embodiment of the present invention;

[0055] Figure 2 is a flow chart of an online collaborative adaptation mode based on forward and backward propagation in an embodiment of the present invention;

[0056] Figure 3 It is a flowchart of an online collaborative adaptation mode using only forward propagation in an embodiment of the present invention. DETAILED DESCRIPTION

[0057] The embodiments of the present invention are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and are not to be construed as limitations of the present invention. For the step numbers in the following embodiments, they are only provided for the convenience of explanation, and the order between the steps is not limited in any way, and the execution order of each step in the embodiment can be adaptively adjusted according to the understanding of those skilled in the art.

[0058] In the description of the present invention, it should be understood that descriptions involving orientations, such as up, down, front, back, left, right, etc., and orientations or positional relationships indicated are based on the orientations or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be understood as a limitation on the present invention.

[0059] In the description of the present invention, "several" means one or more, "more" means more than two, "greater than", "less than", "exceed" etc. are understood as not including the number itself, and "above", "below", "within" etc. are understood as including the number itself. If there is a description of "first" or "second", it is only used for the purpose of distinguishing the technical features, and cannot be understood as indicating or implying the relative importance or implicitly indicating the number of the indicated technical features or implicitly indicating the order of the indicated technical features.

[0060] In the description of the present invention, unless otherwise clearly defined, terms such as setting, installing, connecting, etc. should be understood in a broad sense, and technicians in the relevant technical field can reasonably determine the specific meanings of the above terms in the present invention based on the specific content of the technical solution.

[0061] In response to the existing technical problems, the present invention proposes an efficient online adaptation framework for multi-device collaboration, constructs a dynamic parameter preservation mechanism based on a shared knowledge base, realizes the accumulation, sharing and dynamic utilization of knowledge among multiple devices, and further proposes an online collaborative adaptation method based on forward and backward propagation and one based only on forward propagation, to improve the energy efficiency and practicality of online adaptation in real-world applications.

[0062] Example 1

[0063] like Figure 1 As shown, this embodiment provides an efficient online adaptation method for multi-device collaboration, which is used to realize the accumulation, sharing and dynamic utilization of knowledge among multiple devices, and different devices share the same shared knowledge base, including the following steps:

[0064] For devices with abundant computing resources, the first model in the device is processed using an online collaborative adaptation mode based on forward and back propagation: when a domain shift is detected, the weights learned by the model are dynamically saved to a shared knowledge base, achieving knowledge accumulation without forgetting in cross-domain learning; by introducing a new parameter matrix △θ in the first model parameters new , to capture new knowledge during the adaptation process, thereby decoupling the knowledge reuse and new knowledge learning processes;

[0065] For edge devices with low computing power, an online collaborative adaptation mode that only uses forward propagation is used to process the second model in the device: the model directly uses the existing shared knowledge base for online adaptation, thereby avoiding the problem of high computing cost of knowledge learning.

[0066] In this embodiment, the online collaborative adaptation mode based on forward and backward propagation is aimed at devices with rich computing resources, involving the learning, accumulation, sharing and utilization process of cross-domain knowledge. In this mode, a dynamic knowledge preservation mechanism based on the knowledge base is constructed to measure the difference between the historical data distribution and the current test data distribution, and dynamically save the learned weights to the knowledge base when the domain shifts, so as to achieve cross-domain learning without forgetting knowledge accumulation. Among them, different devices share the same knowledge base, which effectively supports cross-device knowledge sharing and subsequent knowledge utilization, and solves the problem of catastrophic forgetting that is prone to occur in existing technical solutions. Secondly, a weighting coefficient is introduced for each shared knowledge, and the problem of reuse of shared knowledge is converted into an adaptive learning problem of weighting coefficients. Furthermore, a new parameter matrix △θ is introduced new New knowledge is captured during the adaptation process, thereby decoupling the knowledge reuse and new knowledge learning processes and alleviating the optimization conflicts between the two processes. The above parameters can be optimized using online optimization objectives and back-propagation mechanisms, achieving dynamic utilization of knowledge among multiple devices, improving the efficiency and performance of adaptation, and solving the problems of high redundancy in the adaptation of different devices to similar scenarios, serious waste of computing resources, and poor efficiency and performance of online adaptation in existing technical solutions.

[0067] In this embodiment, only the online collaborative adaptation mode of forward propagation is used for edge devices with low computing power, involving the process of utilizing shared knowledge. The main idea is to directly utilize the existing shared knowledge for online adaptation, thereby avoiding the problem of high computing power cost of knowledge learning. In this mode, this embodiment approximates the weighted coefficient optimization problem of knowledge utilization as a domain matching process, and based on the similarity analysis of data distribution, gives a higher utilization weight to the knowledge learned when adapting in similar domains, thereby realizing online dynamic adaptation without optimization. Based on this mode, the computing power cost of online adaptation is reduced, and the problem of limited computing power of edge devices and difficulty in applying online adaptation technology to edge devices in existing technical solutions is solved; thus, the online adaptation technology is effectively expanded to edge devices with limited computing resources such as smartphones.

[0068] The embodiment of the present invention innovatively expands the independent online adaptation of a single device to the efficient online adaptation of multiple devices in collaboration, breaking through the catastrophic forgetting problem of online adaptation, realizing the accumulation, sharing and dynamic use of knowledge among multiple devices, improving the efficiency and performance of online adaptation, and making the expensive online adaptation technology applicable to resource-constrained edge devices. The embodiment of the present invention has the characteristics of low communication cost and high scalability, which will have an important impact on the model deployment and online adaptation technology widely used in edge computing, Internet of Things and distributed systems, and improve the energy efficiency and practicality of model online adaptation in applications.

[0069] The above method is explained in detail below with reference to the accompanying drawings and specific embodiments.

[0070] (1) Online collaborative adaptation model based on forward and back propagation

[0071] This model is aimed at devices with rich computing resources and involves the learning, accumulation, sharing and utilization of cross-domain knowledge. Figure 2 , on a device using this mode, the algorithm consists of the following steps:

[0072] Step 1: Take a batch of target images to be adapted Input model. The formal expression of model parameters θ is as follows:

[0073]

[0074] Among them, θ 0 are the original parameters of the model, is the M pieces of knowledge accumulated in the knowledge base shared by multiple devices, α i is the learnable weight coefficient for normalizing different knowledge, that is, △θ new is a newly introduced learnable parameter matrix. The above scheme decouples the use and update of knowledge into α, △θnew The learning process effectively supports the utilization of knowledge across devices and the learning of new knowledge.

[0075] Step 2: Detect whether domain shift has occurred. If so, save the learned knowledge to the shared knowledge base.

[0076] Specifically, domain detection migration includes the following processes:

[0077] 1) Use the output features of the model backbone network to measure the current input data distribution The calculation formula is as follows:

[0078]

[0079] Among them, B(·) is the output feature of the backbone network, and is the mean and variance of the feature, Depend on and composition.

[0080] 2) Measure the difference between the current input data distribution and the historical data distribution. Historical data distribution φ d is the moving average of the input data distribution, formally expressed as:

[0081]

[0082] Among them, λ is the balance factor, which can be set to 0.8 in implementation. The difference measure of data distribution D(·,·) is the KL divergence between data distributions under the assumption that data distribution follows Gaussian distribution. The calculation formula is as follows:

[0083]

[0084] Where H is the dimension of the statistic. is greater than a certain threshold, that is When , it is considered that the current input data has domain offset. The above domain offset detection scheme does not need to save the original data, has low computational overhead and storage cost, and can effectively maintain the efficiency of online adaptation.

[0085] Specifically, the knowledge △θ M+1 And the corresponding domain distribution φ d Save to shared repository The formal representation of the learned knowledge is as follows:

[0086]

[0087] This knowledge is saved and immediately shared with other devices, and an additional learnable weighting coefficient α is introduced in the model parameters θM+1 , thus updating the formal representation of the parameter θ to: Since △θ M+1 Solidification after saving effectively alleviates the catastrophic forgetting problem under continuous online adaptation, realizes the gradual accumulation of knowledge, and improves the stability of online adaptation. Furthermore, the utilization and update of knowledge are decoupled into α, △θ new The learning process has the following benefits: a) It can avoid Frequent updates effectively reduce cross-device The expensive communication cost caused by synchronization realizes the collaborative online adaptation of multiple devices with efficient communication; b) Multiple devices are only based on the knowledge base For collaboration, any device can join or exit the collaborative group at any time (using the latest or non-latest shared knowledge base), achieving fully asynchronous multi-device collaborative adaptation, and improving the practicality and scalability of the invented patent in real-world scenarios.

[0088] Step 3: Generate prediction results and use online optimization objectives and back-propagation mechanisms to update the model, which can be formally expressed as:

[0089]

[0090] in Represents any online self-supervised or unsupervised optimization goal. This patent focuses on breaking through the collaborative bottleneck of online adaptation between multiple devices, and can be flexibly combined with existing advanced online optimization goals. Preferably, the entropy minimization loss can be used for parameter optimization, and its mathematical form is where f θ (x i ) is the model output result. Based on the above loss, the model is optimized and the same knowledge base is shared among devices. However, different devices can adaptively adjust α, △θ according to their own input data new , achieving efficient adaptation to application scenarios.

[0091] Step 4: Repeat steps 1 to 3 above until all test data are input and the model stops online adaptation.

[0092] (2) Online Co-adaptation Model Using Only Forward Propagation

[0093] This mode is targeted at edge devices with low computing power and involves the use of shared knowledge. As an optional implementation, see Figure 3 , on a device using this mode, the algorithm consists of the following steps:

[0094] Step 1: Take a batch of target images to be adapted Input model. The formal expression of model parameters θ is as follows:

[0095]

[0096] Among them, θ 0 are the original parameters of the model, is the M pieces of knowledge accumulated in the knowledge base shared by multiple devices, γ i is the learnable weight coefficient for normalizing different knowledge, that is, Supports shared knowledge reuse across devices.

[0097] Step 2: Measure the current input data distribution Distributed in a shared knowledge base Among them, the current input data distribution The calculation formula is the same as above, φ i is △θ in the shared knowledge base i Data distribution information during learning φ d . Data distribution similarity ρ i The calculation formula is as follows:

[0098]

[0099] Where D(·,·) is the data distribution difference measure defined above, and ∈ is a very small value to prevent numerical overflow.

[0100] Step 3: Based on the distribution similarity metric, we give priority to using the knowledge learned from similar domains in the shared knowledge base. Specifically, the formal definition of γ is:

[0101] γ=softmax(ρ)

[0102] By innovatively transforming the model online adaptation problem into a dynamic utilization process of shared knowledge and approximating the weighted coefficient optimization problem as a domain matching process, this embodiment effectively avoids the high computing cost caused by knowledge learning and coefficient optimization, reduces the computing cost of online adaptation, and realizes online dynamic adaptation without optimization, thereby expanding the costly online adaptation technology to edge devices with limited computing resources such as smartphones.

[0103] Step 4: Repeat steps 1 to 3 above until all test data are input and the model stops online adaptation.

[0104] In general, the present invention aims to break through the bottleneck of existing online adaptation technology, which is limited to independent adaptation of a single device, has a prominent catastrophic forgetting problem, is difficult to use useful knowledge of other devices, has poor online adaptation performance and efficiency, and cannot be applied to edge devices. To this end, the present invention proposes an efficient online adaptation method for multi-device collaboration, which realizes the accumulation, sharing and dynamic utilization of knowledge among multiple devices. There are two collaboration modes: 1) Online collaborative adaptation mode based on forward and back propagation: This mode reconstructs the model parameters into Thus, the reuse and update of knowledge are decoupled into α, △θ new The learning process alleviates the optimization conflict between the two processes and supports cross-device knowledge utilization and new knowledge learning. Furthermore, a dynamic knowledge preservation mechanism based on the knowledge base is constructed to store the learned new knowledge △θ in the domain shift. M+1 Saving to a shared knowledge base achieves the gradual accumulation of knowledge, alleviates the catastrophic forgetting problem under continuous online adaptation, supports efficient knowledge sharing across devices, and improves the stability and efficiency of online adaptation. 2) Online collaborative adaptation mode using only forward propagation: This mode reconstructs the model parameters into This dynamically utilizes the shared knowledge of different devices and converts γ i The optimization process is converted into domain similarity analysis for solution, which effectively avoids the high computing cost caused by knowledge learning and coefficient optimization, thereby expanding the costly online adaptation technology to edge devices with limited computing resources such as smartphones. In summary, the method of the present invention innovatively expands the independent online adaptation of a single device to the efficient online adaptation of multiple devices in collaboration, breaking through the catastrophic forgetting problem of online adaptation, realizing the accumulation, sharing and dynamic utilization of knowledge among multiple devices, improving the efficiency and performance of online adaptation, and making the costly online adaptation technology applicable to resource-constrained edge devices. The present invention has the characteristics of low communication cost and high scalability, which will have an important impact on model deployment and online adaptation technologies widely used in edge computing, Internet of Things, distributed systems and other fields, and improve the energy efficiency and practicality of model online adaptation in applications.

[0105] Example 2

[0106] An embodiment of the present invention further provides an electronic device, the electronic device comprising a processor and a memory, the memory storing at least one instruction, at least one program, a code set or an instruction set, the at least one instruction, the at least one program, the code set or the instruction set being loaded and executed by the processor to implement the following Figure 1 An efficient online adaptation method for multi-device collaboration is shown.

[0107] It is understood that the memory may include a random access memory (RAM) or a read-only memory (ROM). Optionally, the memory includes a non-transitory computer-readable storage medium. The memory may be used to store instructions, programs, codes, code sets, or instruction sets. The memory may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for at least one function, instructions for implementing the above-mentioned various method embodiments, etc.; the data storage area may store data created according to the use of the server, etc.

[0108] The processor may include one or more processing cores. The processor uses various interfaces and lines to connect the various parts of the entire server, and executes various functions of the server and processes data by running or executing instructions, programs, code sets or instruction sets stored in the memory, and calling data stored in the memory. Optionally, the processor can be implemented in at least one hardware form of digital signal processing (DSP), field programmable gate array (FPGA), and programmable logic array (PLA). The processor can integrate one or a combination of a central processing unit (CPU) and a modem. Among them, the CPU mainly processes the operating system and application programs; the modem is used to process wireless communications. It can be understood that the above-mentioned modem may not be integrated into the processor, but implemented separately through a chip.

[0109] Since the electronic device is an electronic device corresponding to an efficient online adaptation method for multi-device collaboration in an embodiment of the present invention, and the principle of solving the problem by the electronic device is similar to that of the method, the implementation of the electronic device can refer to the implementation process of the above-mentioned method embodiment, and the repeated parts will not be repeated.

[0110] Example 3

[0111] The embodiment of the present invention further provides a computer-readable storage medium, wherein the storage medium stores at least one instruction, at least one program, code set or instruction set, and the at least one instruction, the at least one program, the code set or instruction set is loaded and executed by a processor to implement the following Figure 1 An efficient online adaptation method for multi-device collaboration is shown.

[0112] Those skilled in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing related hardware through a program, and the program can be stored in a computer-readable storage medium, and the storage medium includes a read-only memory (ROM), a random access memory (RAM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), a one-time programmable read-only memory (OTPROM), an electronically erasable rewritable read-only memory (EEPROM), a compact disc (CD-ROM) or other optical disc storage, magnetic disk storage, magnetic tape storage, or any other computer-readable medium that can be used to carry or store data.

[0113] Since the storage medium is the storage medium corresponding to an efficient online adaptation method for multi-device collaboration in an embodiment of the present invention, and the principle of solving the problem by the storage medium is similar to that of the method, the implementation of the storage medium can refer to the implementation process of the above-mentioned method embodiment, and the repeated parts will not be repeated.

[0114] Example 4

[0115] In some possible implementations, various aspects of the method of the embodiment of the present invention may also be implemented in the form of a program product, which includes a program code, and when the program product is run on a computer device, the program code is used to enable the computer device to execute the steps of a multi-device collaborative efficient online adaptation method according to various exemplary embodiments of the present application described above in this specification. Among them, the executable computer program code or "code" for executing each embodiment can be written in a high-level programming language such as C, C++, C#, Smalltalk, Java, JavaScript, Visual Basic, structured query language (e.g., Transact-SQL), Perl, or in various other programming languages.

[0116] It should be understood that the various parts of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, a plurality of steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0117] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0118] The above embodiments are only for illustrating the technical concept and features of the present invention, and their purpose is to enable ordinary technicians in the field to understand the content of the present invention and implement it accordingly, and they cannot be used to limit the protection scope of the present invention. Any equivalent changes or modifications made based on the essence of the content of the present invention should be included in the protection scope of the present invention.

Claims

1. An efficient online adaptation method for multi-device collaboration, characterized in that: It is used to realize the accumulation, sharing and dynamic utilization of knowledge among multiple devices. Different devices share the same shared knowledge base, including the following steps: For devices with abundant computing resources, an online collaborative adaptation mode based on forward and back propagation is used to process the first model in the device: when domain shift is detected, the weights learned by the model are dynamically saved to the shared knowledge base; a new parameter matrix is ​​introduced into the first model parameters to capture new knowledge during the adaptation process; For devices with low computing power, the second model in the device is processed in an online collaborative adaptation mode that only uses forward propagation: the model directly uses the shared knowledge base for online adaptation.

2. The efficient online adaptation method for multi-device collaboration according to claim 1, characterized in that: The expressions of the parameters of the first model are: In the formula, θ 0 are the original parameters of the model; is the M pieces of knowledge accumulated in the shared knowledge base shared by multiple devices; α i is the learnable weight coefficient for normalizing different knowledge, that is, △θ new is a newly introduced learnable parameter matrix.

3. The efficient online adaptation method for multi-device collaboration according to claim 1, characterized in that: The expressions of the parameters of the second model are: In the formula, θ 0 are the original parameters of the model, is the M pieces of knowledge accumulated in the shared knowledge base shared by multiple devices; γ i is the learnable weight coefficient for normalizing different knowledge, that is, Used to support shared knowledge reuse across devices.

4. The efficient online adaptation method for multi-device collaboration according to claim 1, characterized in that: The process of processing the first model in the device using an online collaborative adaptation mode based on forward and backward propagation includes: A batch of target images to be adapted a first model in the input device; Detect whether domain shift has occurred. If so, save the learned knowledge to the shared knowledge base. The prediction results are generated based on the input image, and the model is updated using the online optimization target and back-propagation mechanism until all image data are processed and the model stops online adaptation.

5. The efficient online adaptation method for multi-device collaboration according to claim 4, characterized in that: The detection is to determine whether a domain shift has occurred. If a domain shift has occurred, the learned knowledge is saved to a shared knowledge base. include: Use the output features of the model backbone network to measure the distribution of the current input image data Calculate the current input image data distribution With historical data distribution φ d The Difference If the difference If it is greater than the preset threshold, it is determined that the current input image data has domain shift, and the learned knowledge △θ M+1 And the corresponding domain distribution φ d Save to shared repository 6. The efficient online adaptation method for multi-device collaboration according to claim 5, characterized in that: The current input image data distribution The calculation formula is as follows: In the formula, B(·) is the output feature of the backbone network, x i is the input data, N is the amount of data; and is the mean and variance of the feature, Depend on and composition.

7. The efficient online adaptation method for multi-device collaboration according to claim 5, characterized in that: The historical data distribution φ d is the moving average of the input image data distribution, and the expression is: Where λ is the balance factor; Measuring image data distribution through KL divergence With historical data distribution φ d The difference between Knowledge learned △θ M+1 The expression is: In the formula, is the M pieces of knowledge accumulated in the shared knowledge base shared by multiple devices; α i is the learnable weight coefficient for normalizing different knowledge, that is, △θ new is a newly introduced learnable parameter matrix.

8. The efficient online adaptation method for multi-device collaboration according to claim 4, characterized in that: The expression for updating the model using the online optimization objective and the back-propagation mechanism is: In the formula, represents any online self-supervised or unsupervised optimization objective; θ is the parameter of the first model; x is the target dataset to be adapted; α is the normalized learnable weight coefficient vector; Among them, different devices can adaptively adjust α, △θ according to their own input data new , in order to achieve efficient adaptation to application scenarios.

9. The efficient online adaptation method for multi-device collaboration according to claim 1, characterized in that: For edge devices with low computing power, the second model in the device is processed by adopting an online collaborative adaptation mode using only forward propagation, including: A batch of target images to be adapted a second model in the input device; Calculate the current input image data distribution Distributed in a shared knowledge base similarity; According to the calculated similarity, the knowledge learned from similar domains is obtained from the shared knowledge base to update the model until all image data are processed and the model stops online adaptation.

10. The efficient online adaptation method for multi-device collaboration according to claim 9, characterized in that: The calculation formula of the similarity is as follows: Where D(·,·) is the difference in data distribution; φ i is △θ in the shared knowledge base i Data distribution information during learning φ d ;∈ is a very small amount to prevent numerical overflow.

Images