Data processing method, device, equipment, and storage medium

Abstract

The present application discloses a data processing method and apparatus, a device, and a storage medium; wherein, the method includes: determining a neighbor device, receiving the model parameters of a first inference model transmitted by the neighbor device; determining the fusion weight of the model parameters; according to the fusion weight, fusing the model parameters with the corresponding model parameters of its own second inference model to obtain a fused second inference model; wherein, the first inference model is at least partially structurally consistent with the second inference model; re-determining the neighbor device to iteratively execute the above steps until the end condition is reached, and using the currently obtained fused second inference model as the target inference model; the target inference model is used to perform machine learning tasks on input data; in this way, the energy consumption caused by data transmission in machine learning scenarios can be reduced by sharing model parameters between devices, that is, knowledge sharing.

Description

Technical Field

[0001] The present application relates to artificial intelligence technology, and is related to but not limited to data processing methods and devices, equipment, and storage media. Background Art

[0002] With the continuous evolution of communication technology and the vigorous development of Internet of Things technology, the number of connected IoT devices is increasing. Based on massive IoT sensing and computing devices, various types, levels and diversified IoT services will be formed. In the 5G mobile communication network, the number of connected IoT nodes will reach 100 million, and the deployment density will reach one million per square kilometer. The access demand of massive nodes has challenged the traditional centralized computing architecture, and transmission energy consumption has become one of the main limiting bottlenecks. At the same time, information processing technology represented by artificial intelligence has developed rapidly, the cost of smart chips has continued to decrease, and smart services have been transferred from the cloud to the end side, which has promoted the outbreak of AIoT application scenarios represented by autonomous driving, smart industry, and smart medical care. How to reduce data transmission energy consumption in these application scenarios and other machine learning scenarios has become a technical problem to be solved. Summary of the invention

[0003] In view of this, the data processing method, apparatus, device, and storage medium provided in the present application can reduce the energy consumption caused by data transmission in machine learning scenarios by sharing model parameters between devices, that is, knowledge sharing.

[0004] According to one aspect of an embodiment of the present application, a data processing method is provided, including: determining a neighbor device, receiving model parameters of a first inference model transmitted by the neighbor device; determining a fusion weight of the model parameters; according to the fusion weight, fusing the model parameters with corresponding model parameters of its own second inference model to obtain a fused second inference model; wherein the first inference model is consistent with at least part of the structure of the second inference model; re-determining the neighbor device to iteratively execute the above steps until an end condition is reached, and using the currently obtained fused second inference model as a target inference model; the target inference model is used to perform machine learning tasks on input data.

[0005] In this way, since what is obtained is the model parameters of the first inference model of the neighboring device rather than the sample data set of the neighboring device, the data transmission energy consumption in the machine learning scenario can be reduced, and the target inference model of the electronic device can be obtained by the fusion of model parameters.

[0006] The data processing device provided in the embodiment of the present application includes: a determining unit, configured to determine a neighbor device;

[0007] A receiving unit, used to receive the model parameters of the first inference model transmitted by the neighboring device; the determining unit, also used to determine the fusion weight of the model parameters; a fusion unit, used to fuse the model parameters with the corresponding model parameters of its own second inference model according to the fusion weight to obtain a fused second inference model; wherein the first inference model is at least partially structurally consistent with the second inference model; an iterative unit, used to trigger the determining unit to re-determine the neighboring device, and to trigger the receiving unit and the fusion unit to iteratively execute corresponding steps until an end condition is reached, and the currently obtained fused second inference model is used as the target inference model; the target inference model is used to perform machine learning tasks on input data.

[0008] An electronic device provided in an embodiment of the present application includes a memory and a processor, wherein the memory stores a computer program that can be run on the processor, and when the processor executes the program, the method described in the embodiment of the present application is implemented.

[0009] The computer-readable storage medium provided in the embodiment of the present application stores a computer program thereon, and when the computer program is executed by a processor, the method provided in the embodiment of the present application is implemented.

[0010] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] The drawings herein are incorporated into the specification and constitute a part of the specification. These drawings illustrate embodiments consistent with the present application and are used together with the specification to illustrate the technical solution of the present application. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0012] The flowcharts shown in the accompanying drawings are only exemplary and do not necessarily include all the contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps can be decomposed, and some operations / steps can be combined or partially combined, so the actual execution order may change according to actual conditions.

[0013] Figure 1 A schematic diagram of the implementation flow of the data processing method provided in the embodiment of the present application;

[0014] Figure 2 A schematic diagram of the implementation flow of the data processing method provided in the embodiment of the present application;

[0015] Figure 3A schematic diagram of the "end-edge-cloud" hierarchical architecture provided for an embodiment of the present application;

[0016] Figure 4 A schematic diagram of another hierarchical architecture of "end-edge-cloud" provided in an embodiment of the present application;

[0017] Figure 5 A schematic diagram of the structure of the smart device provided in the embodiment of the present application;

[0018] Figure 6 A schematic diagram of the implementation flow of another data processing method provided in an embodiment of the present application;

[0019] Figure 7 A schematic diagram of initializing an adjacency matrix provided in an embodiment of the present application;

[0020] Figure 8 A schematic diagram of an updated adjacency matrix provided in an embodiment of the present application;

[0021] Fig. 9 This is a schematic diagram of the structure of a data processing device according to an embodiment of the present application;

[0022] Fig.10 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0023] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the specific technical solution of the present application will be further described in detail below in conjunction with the drawings in the embodiments of the present application. The following embodiments are used to illustrate the present application, but are not used to limit the scope of the present application.

[0024] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0025] In the following description, reference is made to “some embodiments”, which describe a subset of all possible embodiments, but it will be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0026] The embodiment of the present application provides a data processing method, which is applied to an electronic device. During the implementation process, the electronic device can be various types of devices with information processing capabilities, for example, the electronic device can include mobile phones, home gateways, cameras, speakers, laptops, personal computers, servers and other Internet of Things devices. The functions implemented by the method can be implemented by calling program codes by a processor in the electronic device. Of course, the program codes can be stored in a computer storage medium. It can be seen that the electronic device at least includes a processor and a storage medium.

[0027] Figure 1 A schematic diagram of the implementation flow of the data processing method provided in the embodiment of the present application is shown in FIG. Figure 1 As shown, the method may include the following steps 101 to 105:

[0028] Step 101: An electronic device determines a neighboring device.

[0029] The so-called neighbor device refers to a device with a large correlation with the electronic device. Therefore, the determined neighbor device may be one or more. The correlation between devices can be characterized by the similarity between the devices. In some embodiments, the electronic device can obtain the current similarity with at least one device, and then select the device whose current similarity meets the similarity condition from the at least one device as a neighbor device.

[0030] Furthermore, in some embodiments, the similarity condition is that the current similarity is greater than a specific threshold.

[0031] Step 102: The electronic device receives model parameters of the first inference model transmitted by the neighboring device.

[0032] In some embodiments, the electronic device may send a request message to a neighbor device, where the request message is used to request model parameters of a first inference model of the neighbor device; the neighbor device returns its own model parameters to the electronic device based on the request message.

[0033] In some embodiments, the model parameters of the first inference model are trained based on the local data set of the neighboring device (i.e., the first sample data set mentioned below). It can be understood that the so-called local data set is data generated by the device itself, rather than data generated by other devices, that is, the personalized data set of the device itself.

[0034] Step 103: The electronic device determines the fusion weight of the model parameters.

[0035] In some embodiments, the current similarity between the electronic device and the neighboring device can be used as the fusion weight of the model parameters of the first inference model; in other embodiments, the model parameters of the first inference model and the second inference model can be used to perform inference on the local data set of the electronic device (i.e., the second sample data set mentioned below), and based on the inference results, the prediction accuracy of the first inference model on the second sample data set is determined; and the prediction accuracy is used as the fusion weight of the model parameters of the first inference model.

[0036] In some embodiments, the model parameters of the first inference model may be substituted for the corresponding model parameters in the second inference model to obtain a replaced inference model, and then the replaced inference model may be used to perform inference on the second sample data set to obtain the prediction accuracy of the first inference model on the second sample data set.

[0037] In other embodiments, the model parameters of the first inference model may be fused with the corresponding model parameters in the second inference model to obtain a fused inference model, and then the fused inference model may be used to perform inference on the second sample data set to obtain the prediction accuracy of the first inference model on the second sample data set.

[0038] Step 104: The electronic device fuses the model parameters with corresponding model parameters of its own second inference model according to the fusion weights to obtain a fused second inference model; wherein at least part of the structure of the first inference model is consistent with that of the second inference model.

[0039] It is understandable that at least part of the structure of the inference model is consistent, and its purpose is to facilitate subsequent model fusion. For example, the inference model with image classification capability includes a feature extraction model and a feature classification model. Then in some embodiments, the first inference model and the second inference model have the same structure of the feature extraction model, for example, both are 5-layer artificial intelligence (AI) models that compress the input 1024*1024 size picture into a 256-dimensional feature vector.

[0040] In some embodiments, the model parameters of the second inference model are trained based on the local data set (i.e., the second sample data set) of the electronic device. When fusing the model parameters, the fusion weight of the model parameters of the second inference model is greater than or equal to the fusion weight of the model parameters of the first inference model. For example, the fusion weight of the model parameters of the second inference model is 1, and the fusion weight of the model parameters of the first inference model is less than 1. Of course, the fusion weight of the model parameters of the first inference model of the neighboring device can be determined based on the prediction accuracy of the first inference model in the local data set of the electronic device.

[0041] For example, assuming that there are c determined neighbor devices (including the electronic device), the model parameters of the fused second inference model can be obtained by the following formula (1):

[0042]

[0043] Among them, w i,t+1 is the model parameter after fusion of device i (i.e. the electronic device), w k,t+1 is the model parameter of neighbor device k, is the update weight of the neighbor device, where n k =am k,i ,

[0044] In step 105, the electronic device re-determines the neighboring device to iteratively execute the above steps 102 to 104 until the end condition is reached, and uses the currently obtained fused second inference model as the target inference model; the target inference model is used to perform machine learning tasks on the input data.

[0045] In this application, the inference model can be a neural network model or a support vector machine, etc. In short, the model is used to complete machine learning tasks. Machine learning tasks can be various, such as image recognition, object detection, speech recognition or semantic segmentation.

[0046] In some embodiments, the current similarity between the electronic device and the corresponding neighbor device can be updated based on the prediction accuracy of the first inference model of the neighbor device on the local data set of the electronic device; then, the electronic device re-determines the neighbor device based on the current similarity with each device, and then executes steps 102 to 104 until the end condition is reached.

[0047] In the present application, there is no limitation on the termination condition. In some embodiments, the termination condition may be that the prediction accuracy of the currently obtained fused second inference model is greater than or equal to a set accuracy threshold. In other embodiments, the termination condition may also be that the number of iterations reaches a set number threshold.

[0048] In an embodiment of the present application, an electronic device determines its neighboring device, and then obtains model parameters of a first reasoning model of the neighboring device; determines a fusion weight of the model parameters; based on the fusion weight, fuses the model parameters with corresponding model parameters of its own second reasoning model to obtain a fused second reasoning model; then, re-determines its neighboring device to iteratively execute the above steps until an end condition is reached, and uses the currently obtained fused second reasoning model as the target reasoning model to perform machine learning tasks on input data; in this way, since the model parameters of the first reasoning model of the neighboring device are obtained instead of the sample data set of the neighboring device, the amount of data transmission in the machine learning scenario can be reduced, thereby reducing the data transmission energy consumption in the scenario, and the target reasoning model of the electronic device is obtained through the fusion of model parameters.

[0049] The present application embodiment further provides a data processing method, Figure 2 A schematic diagram of the implementation flow of the data processing method provided in the embodiment of the present application is shown in FIG. Figure 2 As shown, the method may include the following steps 201 to 209:

[0050] Step 201: The electronic device uses its own second sample data set to train model parameters of a second initial inference model to obtain its own second inference model, that is, a personalized model of the electronic device.

[0051] In some embodiments, the structure of the second inference model of the electronic device is consistent with part or all of the structure of the inference model of other devices (such as the size of the convolution kernel, the number of model layers, the number of parameters, etc.), which facilitates subsequent model fusion.

[0052] Step 202: The electronic device obtains the current similarity between the electronic device and at least one device.

[0053] In some embodiments, the electronic device may maintain an adjacency matrix, wherein the values ​​(ie, elements) represent the similarity between the corresponding two devices. Therefore, the electronic device may obtain the adjacency matrix and obtain the current similarity with the at least one device from the adjacency matrix.

[0054] After updating the adjacency matrix, both the electronic device and other devices in the adjacency matrix will widely advertise it, for example, by sending the updated adjacency matrix through broadcast or multicast. In this way, the electronic device receives the updated adjacency matrix sent by at least one of the at least one devices through broadcast or multicast, thereby obtaining the latest adjacency matrix more quickly.

[0055] Step 203: The electronic device selects a device whose current similarity satisfies a similarity condition from the at least one device as a neighbor device.

[0056] In this application, there is no limitation on the similarity condition. In some embodiments, the similarity condition may be that the current similarity is greater than a specific threshold; in other embodiments, the similarity condition may be K maximum current similarities, where K is greater than 0. K may be any specific value, which is not limited in this application.

[0057] It can be understood that in the embodiment of the present application, only devices whose similarity between devices meets similarity conditions are selected as neighboring devices, and model parameters are fused based on this; in this way, the reasoning accuracy of the final target reasoning model can be improved while reducing the complexity of the fusion.

[0058] Step 204: The electronic device receives the model parameters of the first inference model transmitted by the neighboring device; wherein the model parameters of the first inference model are obtained by training the model parameters of the first initial inference model based on the first sample data set of the neighboring device.

[0059] It can be understood that before the fusion of model parameters, the first inference model and the second inference model are both obtained after training based on the corresponding local sample data sets; in this way, the number of fusion iterations of model parameters can be reduced, saving computing overhead.

[0060] Step 205: The electronic device uses the model parameters of the first inference model and the second inference model to perform inference on the second sample data set to obtain the prediction accuracy of the first inference model on the second sample data set.

[0061] In some embodiments, the model parameters of the first inference model may be substituted for the corresponding model parameters in the second inference model to obtain a replaced inference model, and then the replaced inference model may be used to perform inference on the second sample data set to obtain the prediction accuracy of the first inference model on the second sample data set.

[0062] In other embodiments, the model parameters of the first inference model may be fused with the corresponding model parameters in the second inference model to obtain a fused inference model, and then the fused inference model may be used to perform inference on the second sample data set to obtain the prediction accuracy of the first inference model on the second sample data set.

[0063] In step 206, the electronic device uses the prediction accuracy as the fusion weight of the model parameters of the first inference model, and fuses the model parameters of the first inference model with the corresponding model parameters of its own second inference model according to the fusion weight to obtain a fused second inference model; wherein, at least part of the structure of the first inference model is consistent with that of the second inference model.

[0064] It can be understood that in the embodiment of the present application, the model parameters of the first inference model of the neighboring device and the second inference model are used to perform inference on the local data set of the electronic device, and the prediction accuracy of the first inference model of the neighboring device on the local data set of the electronic device is obtained based on the inference result, and the prediction accuracy is used as the fusion weight of the model parameters of the first inference model. It is easy to understand that the higher the prediction accuracy, the more similar the data set of the neighboring device used to train the first inference model is to the data set of the electronic device used to train the second inference model; then, the model parameters are fused based on the prediction accuracy, and the fused second inference model obtained does not lose personalization.

[0065] Step 207 , the electronic device determines whether the end condition is met; if so, execute step 208 ; otherwise, execute step 209 .

[0066] As mentioned above, there is no restriction on the termination condition. In some embodiments, the termination condition may be that the number of iterations reaches a set number threshold; in other embodiments, the termination condition may be that the prediction accuracy of the currently obtained fused second inference model on the local third sample data set is greater than or equal to the set accuracy threshold. It should be noted that the third sample data set may include the second sample data set or may not include it. When included, for example, the third sample data set is the second sample data set; as another example, the third sample data set includes the second sample data set and a local test set. When not included, for example, the third sample data set is a local test set.

[0067] Step 208: The electronic device uses the currently obtained fused second inference model as a target inference model, where the target inference model is used to perform a machine learning task on the input data.

[0068] Step 209 : The electronic device updates the current similarity with the corresponding neighbor device according to the prediction accuracy; then, the process returns to step 203 .

[0069] Theoretically, by calculating the similarity between the personalized data of different devices, the similarity between the different devices can be obtained. However, considering that the device data may contain user privacy data in actual applications, in the embodiment of the present application, the similarity between two devices can be randomly initialized, and in the subsequent process, the current similarity between the corresponding neighbor devices can be updated based on the prediction accuracy of the first inference model of the neighbor device on the second sample data set of the electronic device; in this way, while solving the problem that the local data set of the neighbor device is difficult to obtain, the prediction accuracy of the second inference model after fusion is not lost.

[0070] Furthermore, in some embodiments, the electronic device may update the current similarity between the electronic device and the neighbor device in the adjacency matrix according to the prediction accuracy.

[0071] In the present application, there is no limitation on the method of updating the current similarity. In some embodiments, the electronic device may replace the current similarity value with the neighboring device with the prediction accuracy of the first inference model on the second sample data set; in other embodiments, the electronic device may also replace the current similarity value with the neighboring device with the weighted average of the current similarity and the prediction accuracy of the first inference model on the second sample data set.

[0072] In some embodiments, the method further includes: the electronic device notifying at least the at least one device of the updated current similarity between the electronic device and the neighboring device, so that the at least one device updates the corresponding current similarity.

[0073] Furthermore, in some embodiments, the electronic device may notify at least one of the at least one device of the updated current similarity with the neighbor device through a broadcast method or a multicast method.

[0074] Furthermore, in some embodiments, the electronic device notifies at least one device of the updated adjacency matrix by broadcasting or multicasting. For example, the electronic device broadcasts the updated adjacency matrix by broadcasting, so that other devices on the adjacency matrix update their own maintained adjacency matrix based on the matrix after receiving the updated adjacency matrix. For another example, the electronic device sends the updated adjacency matrix to neighboring devices by multicasting, so that the neighboring devices update their own maintained adjacency matrix based on the matrix after receiving the updated adjacency matrix.

[0075] It can be understood that, whether it is the electronic device or other devices on the adjacency matrix, after updating the adjacency matrix maintained by itself, the updated adjacency matrix will be notified to each device through broadcast or multicast method; this enables the electronic device to obtain a more accurate adjacency matrix faster, thereby reducing the number of iterations of model parameter fusion.

[0076] Based on the decentralized architecture, computing and transmission are organically integrated, breaking through the transmission bottleneck, which is beneficial to meet the urgent needs of AIoT application scenarios in terms of latency, bandwidth, power consumption, cost, privacy, etc., and helps to provide technical support for the further development of various scenarios; at the same time, active perception of the environment and intelligent collaboration between devices are formed to provide differentiated and personalized services for different users, which can effectively improve the user experience. Therefore, designing a decentralized distributed learning method for IoT devices has important application value and research significance.

[0077] However, in some embodiments, the distributed learning method focuses on how to use data distributed on multiple devices to train a global general model, thereby proposing a model fusion method based on a centralized architecture to achieve joint training of distributed data. However, the existing methods do not take into account that when the external environment is different, the focus or ultimate goal of different IoT devices is different. Compared with homogeneous services, users need more personalized, differentiated and customized services.

[0078] For example, in a service experience scenario based on mobile phone location prediction, different users have different action patterns. For example, users with rich social activities may appear in multiple locations in a day, while users with single social activities may appear in relatively fixed locations in a day. Therefore, a general model cannot provide accurate services for specific users, thus affecting the service experience formed based on location prediction.

[0079] For example, in the smart speaker voice recognition service in the home IoT scenario, different users may use different dialects or have different language habits, and the general Mandarin recognition model is difficult to meet the recognition needs of dialect users.

[0080] For example, in a smart photo album scenario, the end-side device needs to identify specific people, objects, and scenes. However, the recognition requirements of different users are different, which leads to the need for the end-side recognition models to have different focuses. Therefore, a general recognition model is difficult to meet the personalized service requirements of actual application scenarios.

[0081] In summary, when facing the personalized service needs of users, it is necessary to give the end-side devices a higher degree of freedom, which is not suitable for a centralized learning architecture. Therefore, there is currently no recognized decentralized distributed learning method that meets the actual situation.

[0082] Based on this, an exemplary application of an embodiment of the present application in a practical application scenario will be described below.

[0083] In the embodiments of the present application, the proposed decentralized distributed learning method for IoT devices organically integrates computing and transmission based on a decentralized architecture, breaks through the transmission bottleneck, is beneficial to meeting the urgent needs of AIoT application scenarios in terms of latency, bandwidth, power consumption, cost, and privacy, and helps provide technical support for the further development of various scenarios; at the same time, it forms active perception of the environment and intelligent collaboration between devices, provides differentiated and personalized services for different users, and can effectively improve user experience.

[0084] In addition, in the embodiment of the present application, Figure 3 and Figure 4The effective supplement of the "end-edge-cloud" hierarchical architecture shown above mainly forms an adaptive collaboration mechanism between intelligent devices based on AI models (i.e., reasoning models) at the end-side device layer, which can also be applied to other layers with AI models to form an IoT service architecture with inherent intelligence. Compared with cloud-side servers, end-side and edge-side devices are closer to user personalized data, so it is necessary to form customized AI models at the end-side device layer and edge device layer in order to provide differentiated and personalized services to different target users or target groups.

[0085] In order to solve the above technical problems, in an embodiment of the present application, a decentralized distributed learning method for IoT devices is provided, and the details of the method are as follows:

[0086] Prerequisite: There is a decentralized architecture with multiple smart devices, each device has a portion of local data, and the local data between different devices is non-independent and identically distributed (Non-IID).

[0087] Goal: To make the local classification model of each smart device more accurate and provide differentiated services to different users.

[0088] Among them, smart devices such as Figure 5 As shown in the figure, it refers to a smart IoT device with the following capabilities (the dashed lines indicate optional capabilities). Figure 5 As shown, the smart device 50 includes a communication module 501, a data storage module 502, a resource allocation module 503, a perception module 504, a distributed learning module 505, a privacy protection / encryption module 506 and an AI capability module 507; wherein the distributed learning module 505 includes an AI routing control unit, an AI model fusion unit and an AI model compression unit; the AI ​​capability module 507 includes an AI model training unit and an AI reasoning unit; smart devices include but are not limited to smart phones, smart home gateways, smart cameras, smart speakers and other smart Internet of Things devices.

[0089] like Figure 6 As shown, the detailed steps of a decentralized distributed learning method for an IoT device provided in an embodiment of the present application include the following steps S1 to S4:

[0090] Step S1, initializing the model of each device (i.e., smart device, hereinafter collectively referred to as device). Assume that there are N devices {E1…E N}, and its local personalized data sets are {D1…D N}; Each local data set contains two parts: training set and test set D i ={D i,train ,D i,test},i∈[1,N]. The corresponding personalized model is trained by the training set of each device data Among them, the structure of the personalized model of each device (such as the size of the convolution kernel, the number of model layers, the number of parameters, etc.) should be consistent.

[0091] The purpose of consistent model structure in step S1 is to facilitate subsequent model fusion. For example, the AI ​​capability model on the device can be composed of multiple models in series (such as image classification capability can be composed of a feature extraction model and a feature classification model). In this case, only the AI ​​capability model structure that needs to be integrated between devices is limited to be consistent (such as limiting the feature extraction model of each device to a 5-layer AI model that compresses the input 1024*1024 size image into a 256-dimensional feature vector).

[0092] Step S2, initializing the adjacency matrix AM. The adjacency matrix can represent the correlation between devices and is an important element for routing control of devices in a peer-to-peer architecture. The value of the i-th row and j-th column of AM is am i,j Indicates device E i With device E j Theoretically, it can be calculated by calculating the similarity between different personalized data. The calculation method is as follows:

[0093] Assume D A With D B For personalized datasets of two different devices, then D A With D B The calculation formula of the similarity between is shown in formula (2):

[0094] sim(D A ,D B )=exp(-dis(D A ,D B ))∈[0,1] (2);

[0095] in, C BA , C BA ′、C AB , C AB ′ is a linear transformation matrix, which can be solved by gradient descent and other methods.

[0096] In step S2, considering that the device data in actual applications may contain user privacy data, it is difficult to transmit, collect and solve the similarity of data. Therefore, the adjacency matrix can be randomly initialized and updated in the subsequent learning process. Figure 7As shown, assuming that the current scene contains 4 smart devices, device 1 has a data set containing three categories of data [0, 1, 2], device 2 has a data set containing two categories of data [0, 1], device 3 has a data set containing one category of data [2], and device 4 has a data set containing one category of data [1]. The method for randomly initializing the adjacency matrix is ​​as follows:

[0097] like Figure 7 As shown, in a peer-to-peer environment, the adjacency matrix weights may first be set to the same value (as shown in the figure) or randomly assigned.

[0098] Step S3, model fusion. Based on the adjacency matrix generated in step S2, each device updates the personalized model by acquiring the models of similar devices to itself and performing weighted model fusion, and updates the adjacency matrix at the same time. Since the adjacency matrix initialized in step S2 cannot reflect the actual correlation between devices, step S3 is carried out in two stages, namely step S3.1 and step S3.2.

[0099] Step S3.1, update the adjacency matrix. The weights in the adjacency matrix represent the similarity between devices. Assume that the current device E i Model with initialization When E i Will Send to its neighbor device E j After E j In its own training set D j,train Perform reasoning on it and calculate In D j,train The prediction accuracy acc ji As the value of the i-th column and j-th row of the adjacency matrix. That is, am i,j =acc ji In order to obtain a faster and more accurate adjacency matrix, various communication strategies such as broadcast method and multicast method can be used to update the adjacency matrix.

[0100] Step S3.2: local model fusion. For each device E i ,i∈[1,N], select the node with E according to the adjacency matrix AM i The most similar c neighboring devices are selected and the model parameters are transmitted. For example, the following steps S3.2.1 and S3.2.2 are included:

[0101] Step S3.2.1, using the accuracy of the neighbor device model on the local training data as a weight, fuse it with the local model. The fusion method is shown in the following formula (3):

[0102]

[0103] Among them, wi,t+1 is the model parameter after fusion of device i, w k,t+1 are the model parameters of the neighboring device, is the update weight of the neighbor device, where n k =am k,i ,

[0104] Step S3.2.2, using the accuracy obtained in step S3.2.1 as the weight, update the adjacency matrix.

[0105] Step S4, repeat step S3 until the end condition is met. The end condition can be reaching a certain number of updates or reaching a set accuracy threshold. The final update result is as follows Figure 8 shown.

[0106] In an embodiment of the present application, a centerless distributed learning method for an Internet of Things device is provided, and through the adaptive model sharing between intelligent Internet of Things devices (i.e., intelligent devices), intelligent collaboration between devices is realized, and personalized, differentiated and customized services can be provided to users. First, the embodiment of the present application obtains the correlation between devices by maintaining the adjacency matrix, and performs local fusion of similar models according to the adjacency matrix, so that similar devices can share knowledge. Secondly, through knowledge sharing between similar devices, the energy consumption caused by data transmission is reduced. Compared with the centralized architecture, the centerless distributed method provided by the embodiment of the present application can greatly reduce data transmission, protect user privacy, and is more suitable for resource-constrained Internet of Things devices. Finally, the method provided by the embodiment of the present application is applicable to the situation where the local data between different devices is not independent and identically distributed, and can be applied to the intelligent Internet of Things devices in the actual Internet of Things environment, so as to provide personalized, differentiated and customized services for different users.

[0107] Related distributed learning methods focus on how to use data distributed on multiple devices to train a global general model, and thus propose a model fusion method based on a centralized architecture to achieve joint training of distributed data. However, this method does not take into account that when the external environment is different, the focus or ultimate goal of different IoT devices is not the same. Compared with homogeneous services, users need to obtain personalized, differentiated and customized services. The centerless distributed learning method for IoT devices provided in the embodiment of the present application is based on a decentralized architecture, organically integrates computing and transmission, breaks through the transmission bottleneck, and is beneficial to meeting the urgent needs of AIoT application scenarios for latency, bandwidth, power consumption, cost, and privacy, and helps to provide technical support for the further development of various scenarios; at the same time, it forms active perception of the environment and intelligent collaboration between devices, provides differentiated and personalized services for different users, and can effectively improve user experience.

[0108] It should be noted that although the steps of the method in the present application are described in a specific order in the drawings, this does not require or imply that the steps must be performed in this specific order, or that all the steps shown must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps, etc.; or, steps in different embodiments may be combined into a new technical solution.

[0109] Based on the foregoing embodiments, the embodiments of the present application provide a model device, and each unit included in the device can be implemented by a processor; of course, it can also be implemented by a specific logic circuit; in the implementation process, the processor can be a central processing unit (CPU), a microprocessor (MPU), a digital signal processor (DSP) or a field programmable gate array (FPGA), etc.

[0110] Fig. 9 Schematic diagram of the structure of the data processing device of the embodiment of the present application, such as Fig. 9 As shown, the device 900 includes:

[0111] A determining unit 901, configured to determine a neighboring device;

[0112] A receiving unit 902, configured to receive model parameters of a first inference model transmitted by the neighboring device;

[0113] The determination unit 901 is further used to determine the fusion weight of the model parameters;

[0114] A fusion unit 903 is used to fuse the model parameters with corresponding model parameters of its own second reasoning model according to the fusion weight to obtain a fused second reasoning model; wherein at least part of the structure of the first reasoning model is consistent with that of the second reasoning model;

[0115] The iteration unit 904 is used to trigger the determination unit 901 to re-determine the neighboring device, and to trigger the receiving unit 902 and the fusion unit 903 to iteratively execute the corresponding steps until the end condition is reached, and the currently obtained fused second inference model is used as the target inference model; the target inference model is used to perform machine learning tasks on the input data.

[0116] In some embodiments, the model parameters of the first inference model are obtained by training the model parameters of the first initial inference model based on the first sample data set of the neighboring device; the device 900 also includes a training unit, which is used to train the model parameters of the second initial inference model using its own second sample data set before fusing the model parameters with the corresponding model parameters of its own second inference model to obtain the second inference model.

[0117] In some embodiments, the determination unit 901 is configured to obtain a current similarity with at least one device; and select, from the at least one device, a device whose current similarity satisfies a similarity condition as a neighbor device.

[0118] In some embodiments, the determination unit 901 is used to perform reasoning on its own second sample data set using the received model parameters of the first inference model and the second inference model to obtain the prediction accuracy of the first inference model on the second sample data set; and use the prediction accuracy as the fusion weight of the model parameters of the first inference model.

[0119] In some embodiments, the device 900 also includes an update unit and a notification unit, wherein the update unit is used to update the current similarity with the corresponding neighbor device based on the prediction accuracy; the notification unit is used to notify the at least one device of the updated current similarity with the neighbor device so that the at least one device updates the corresponding current similarity.

[0120] In some embodiments, the receiving unit 902 is used to: obtain an adjacency matrix, wherein the numerical values ​​in the adjacency matrix represent the current similarity between the corresponding two devices; obtain the current similarity with the at least one device from the adjacency matrix; and the updating unit is used to update the current similarity with the neighboring device in the adjacency matrix according to the prediction accuracy.

[0121] In some embodiments, the receiving unit 902 is configured to receive an updated adjacency matrix sent by at least one of the at least one device via a broadcast method or a multicast method.

[0122] The description of the above device embodiment is similar to the description of the above method embodiment, and has similar beneficial effects as the method embodiment. For technical details not disclosed in the device embodiment of the present application, please refer to the description of the method embodiment of the present application for understanding.

[0123] It should be noted that in the embodiments of this application Fig. 9 The model device shown and Figure 5 The division of modules or units in the smart device shown is schematic and is only a logical functional division. There may be other division methods in actual implementation. In addition, each functional unit in each embodiment of the present application may be integrated into a processing unit, or may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of a software functional unit. It may also be implemented in the form of a combination of software and hardware.

[0124] It should be noted that in the embodiment of the present application, if the above method is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the embodiment of the present application can be essentially or partly embodied in the form of a software product that contributes to the relevant technology. The computer software product is stored in a storage medium, including several instructions to enable an electronic device to execute all or part of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a U disk, a mobile hard disk, a read-only memory (ROM), a magnetic disk or an optical disk. In this way, the embodiment of the present application is not limited to any specific combination of hardware and software.

[0125] An embodiment of the present application provides an electronic device, Fig.10 Schematic diagram of the hardware entity of the electronic device of the embodiment of the present application, such as Fig.10 As shown, the electronic device 100 includes a storage 1001 and a processor 1002, wherein the storage 1001 stores a computer program that can be run on the processor 1002, and the processor 1002 implements the steps of the method provided in the above embodiment when executing the program.

[0126] It should be noted that the storage 1001 is configured to store instructions and applications executable by the processing 1002, and can also cache data to be processed or processed by the processing 1002 and various modules in the electronic device 100 (for example, image data, audio data, voice communication data, and video communication data), which can be implemented through flash memory (FLASH) or random access memory (Random Access Memory, RAM).

[0127] An embodiment of the present application provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps in the method provided in the above embodiment are implemented.

[0128] An embodiment of the present application provides a computer program product including instructions, which, when executed on a computer, enables the computer to execute the steps of the method provided in the above method embodiment.

[0129] It should be noted here that the description of the above storage medium and device embodiments is similar to the description of the above method embodiments, and has similar beneficial effects as the method embodiments. For technical details not disclosed in the storage medium, storage medium and device embodiments of this application, please refer to the description of the method embodiments of this application for understanding.

[0130] It should be understood that "one embodiment" or "an embodiment" or "some embodiments" mentioned throughout the specification means that specific features, structures or characteristics related to the embodiment are included in at least one embodiment of the present application. Therefore, "in one embodiment" or "in one embodiment" or "in some embodiments" appearing throughout the specification may not necessarily refer to the same embodiment. In addition, these specific features, structures or characteristics can be combined in one or more embodiments in any suitable manner. It should be understood that in various embodiments of the present application, the size of the sequence number of the above-mentioned processes does not mean the order of execution, and the execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present application. The above-mentioned sequence numbers of the embodiments of the present application are only for description and do not represent the advantages and disadvantages of the embodiments. The above description of each embodiment tends to emphasize the differences between the various embodiments, and the same or similar aspects can be referenced to each other. For the sake of brevity, this article will not repeat them.

[0131] The term "and / or" in this article is only a description of the association relationship of associated objects, indicating that there may be three relationships. For example, object A and / or object B can represent three situations: object A exists alone, object A and object B exist at the same time, and object B exists alone.

[0132] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.

[0133] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. The embodiments described above are only schematic. For example, the division of the modules is only a logical function division. There may be other division methods in actual implementation, such as: multiple modules or components can be combined, or can be integrated into another system, or some features can be ignored, or not executed. In addition, the coupling, direct coupling, or communication connection between the components shown or discussed can be through some interfaces, and the indirect coupling or communication connection of devices or modules can be electrical, mechanical or other forms.

[0134] The modules described above as separate components may or may not be physically separated, and the components displayed as modules may or may not be physical modules; they may be located in one place or distributed on multiple network units; some or all of the modules may be selected according to actual needs to achieve the purpose of the present embodiment.

[0135] In addition, all functional modules in the embodiments of the present application may be integrated into one processing unit, or each module may be a separate unit, or two or more modules may be integrated into one unit; the above-mentioned integrated modules may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0136] A person skilled in the art can understand that all or part of the steps of implementing the above method embodiment can be completed by hardware related to program instructions, and the aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps of the above method embodiment; and the aforementioned storage medium includes: mobile storage devices, read-only memories (ROM), magnetic disks or optical disks, etc., various media that can store program codes.

[0137] Alternatively, if the above-mentioned integrated unit of the present application is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application can essentially or in other words, the part that contributes to the relevant technology can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for enabling an electronic device to execute all or part of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as mobile storage devices, ROMs, magnetic disks, or optical disks.

[0138] The methods disclosed in several method embodiments provided in this application can be arbitrarily combined without conflict to obtain new method embodiments.

[0139] The features disclosed in several product embodiments provided in this application can be arbitrarily combined without conflict to obtain new product embodiments.

[0140] The features disclosed in several method or device embodiments provided in this application can be arbitrarily combined without conflict to obtain new method embodiments or device embodiments.

[0141] The above is only an implementation method of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A data processing method, characterized in that: The method comprises: Determine a neighbor device, and receive model parameters of a first inference model transmitted by the neighbor device; Determining fusion weights of the model parameters; According to the fusion weight, the model parameters are fused with corresponding model parameters of the second inference model to obtain a fused second inference model; wherein at least part of the structure of the first inference model is consistent with that of the second inference model; Re-determine the neighboring device to iteratively execute the above steps until the end condition is reached, and use the currently obtained fused second inference model as the target inference model; the target inference model is used to perform machine learning tasks on the input data; The step of determining a neighbor device includes: Obtaining the current similarity with at least one device, comprising: obtaining an adjacency matrix, wherein the values ​​in the adjacency matrix represent the current similarity between the corresponding two devices; obtaining the current similarity with the at least one device from the adjacency matrix; From the at least one device, select a device whose current similarity satisfies a similarity condition as a neighbor device; The method further comprises: Performing reasoning on its own second sample data set using the received model parameters of the first reasoning model and the second reasoning model to obtain a prediction accuracy of the first reasoning model on the second sample data set; According to the prediction accuracy, updating the current similarity between the neighboring devices in the adjacency matrix; The updated current similarity between the neighbor device and the at least one device is notified so that the at least one device updates the corresponding current similarity.

2. The method according to claim 1, characterized in that The determining of the fusion weight of the model parameters includes: The prediction accuracy is used as a fusion weight of the model parameters of the first inference model.

3. The method according to claim 1, characterized in that: The obtaining of the adjacency matrix comprises: An updated adjacency matrix is ​​received, which is sent by at least one of the at least one devices through a broadcast method or a multicast method.

4. The method according to any one of claims 1 to 3, characterized in that: The model parameters of the first inference model are obtained by training the model parameters of the first initial inference model based on the first sample data set of the neighboring device; Before fusing the model parameters with corresponding model parameters of the second inference model of the method, the method further includes: The model parameters of the second initial reasoning model are trained using the second sample data set to obtain the second reasoning model.

5. A data processing device, characterized in that: include: A determination unit, used for determining a neighbor device; A receiving unit, configured to receive model parameters of a first inference model transmitted by the neighboring device; The determining unit is further used to determine the fusion weight of the model parameters; A fusion unit, configured to fuse the model parameters with corresponding model parameters of its own second inference model according to the fusion weights to obtain a fused second inference model; wherein at least part of the structure of the first inference model is consistent with that of the second inference model; an iterative unit, used to trigger the determination unit to re-determine the neighboring device, and to trigger the receiving unit and the fusion unit to iteratively perform corresponding steps until an end condition is reached, and to use the currently obtained fused second inference model as a target inference model; the target inference model is used to perform a machine learning task on the input data; The determining unit is used to obtain the current similarity with at least one device; and select a device whose current similarity satisfies a similarity condition as a neighboring device from the at least one device; The receiving unit is further configured to obtain an adjacency matrix, wherein the values ​​in the adjacency matrix represent the current similarity between the corresponding two devices; and obtain the current similarity with the at least one device from the adjacency matrix; The determining unit is further configured to perform reasoning on its own second sample data set using the received model parameters of the first reasoning model and the second reasoning model to obtain a prediction accuracy of the first reasoning model on the second sample data set; The data processing device further includes: an updating unit and a notifying unit; The updating unit is used to update the current similarity between the neighboring device and the adjacency matrix according to the prediction accuracy; The updating unit is used to notify the at least one device of the updated current similarity with the neighboring device, so that the at least one device updates the corresponding current similarity.

6. An electronic device comprising a memory and a processor, wherein the memory stores a computer program that can be run on the processor, characterized in that: When the processor executes the program, the method according to any one of claims 1 to 4 is implemented.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 4 is implemented.

Images