Data fusion method, device, electronic device and storage medium for distributed system

Abstract

The present application discloses a data fusion method, device, electronic device and storage medium for a distributed system, and relates to the field of computer technology. At least one modal data of an edge node is obtained, and each modal data is mapped using a modal mapping model to obtain modal mapping data. Then, the data prototype of the modal data is obtained and transmitted to the central node, and then the global data prototype obtained by the central node performing an aggregation operation on the data prototype is obtained, thereby effectively protecting the privacy data of different edge nodes. The modal mapping data is compared and mapped with the global data prototype to obtain pooled modal data and input into the data fusion model for training to obtain fusion parameters. The fusion parameters are then sent to the central node, and the optimization parameters obtained by the central node based on the fusion parameters are received, so as to continue to update the data fusion model based on the optimization parameters. The problem of large differences in data modalities is effectively solved, thereby improving the accuracy of data processing results.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a data fusion method, device, electronic device, and storage medium for a distributed system. Background Art

[0002] A distributed system is a network composed of multiple nodes that communicate with each other and work collaboratively to achieve a common goal. With the continuous advancement of science and technology, the framework of distributed systems has provided strong support for concepts such as smart cities, intelligent transportation, and smart campuses. For example, a smart campus deploys distributed IoT data collection devices at enterprise nodes within the campus to collect relevant data from each enterprise. This data is then sent to the cloud for data fusion and processing, and the results are used to assist different enterprises in intelligent management.

[0003] However, related technologies usually transmit the data collected by distributed devices to the cloud for centralized processing. This process does not take into account data privacy issues, and the data modality differences caused by differences in collection devices are not taken into account. Therefore, the data processing results obtained by the cloud based on these data are not very accurate. Summary of the Invention

[0004] The present application aims to solve at least one of the technical problems existing in the prior art. To this end, the embodiments of the present application provide a data fusion method, device, electronic device and storage medium for a distributed system, which can improve the accuracy of data processing results.

[0005] In a first aspect, an embodiment of the present application provides a data fusion method for a distributed system, wherein the distributed system includes a central node and at least one edge node, each of the edge nodes being communicatively connected to the central node; the method includes:

[0006] Acquire at least one modal data of the edge node, and map each modal data using a pre-trained modal mapping model to obtain corresponding modal mapping data;

[0007] Obtaining a data prototype corresponding to each of the modal data, and transmitting the data prototype to the central node;

[0008] Obtaining a global data prototype corresponding to each mode obtained by the central node performing an aggregation operation on the data prototype;

[0009] Comparing and mapping the modal mapping data with the global data prototype to obtain pooled modal data;

[0010] The pooled modal data is input into a data fusion model for training to obtain fusion parameters, the fusion parameters are sent to the central node, and the optimization parameters obtained by the central node based on the fusion parameters are received, and the data fusion model is updated based on the optimization parameters.

[0011] In some embodiments of the present application, obtaining a data prototype corresponding to each type of modal data includes:

[0012] Obtaining a total number of data samples of the modal data in the edge node;

[0013] performing a sum operation on the modal mapping data of the same modal data to obtain a modal data prototype;

[0014] The modal data prototype is divided by the total number of data samples to obtain the data prototype.

[0015] In some embodiments of the present application, obtaining the global data prototype corresponding to each modality obtained by the central node performing an aggregation operation on the data prototype includes:

[0016] Using the central node to sum the total number of data samples of each edge node to obtain a global total number of samples;

[0017] Using the central node, the total number of data samples of the edge node is divided by the total number of global samples to obtain an edge coefficient of the edge node;

[0018] Multiplying the edge coefficient by the data prototype using the central node to obtain a reference data prototype of the edge node;

[0019] The global data prototype is obtained by the central node performing a sum operation on the reference data prototype of each edge node.

[0020] In some embodiments of the present application, the comparing and mapping the modality mapping data with the global data prototype to obtain pooled modality data includes:

[0021] Acquire global modal data according to the global data prototype; wherein the global modal data includes each type of modal data of each edge node;

[0022] Comparing and mapping each of the modal mapping data with the global modal data in sequence to obtain corresponding comparative mapping data;

[0023] An average pooling operation is performed on the contrast mapping data to obtain the pooled modal data.

[0024] In some embodiments of the present application, inputting the pooled modality data into a data fusion model for training to obtain fusion parameters includes:

[0025] Performing modal learning based on the pooled modal data to obtain modal parameters of each modal data;

[0026] Calculating the modal weight corresponding to each modal data according to the global modal data;

[0027] Multiplying the modal weight by the modal parameter to obtain fused data of the modal data;

[0028] performing a summing operation on the fused data of each modality data to obtain modality fusion data;

[0029] The modality fusion data is input into the data fusion model for training to obtain the fusion parameters.

[0030] In some embodiments of the present application, inputting the pooled modality data into a data fusion model for training to obtain fusion parameters, sending the fusion parameters to the central node, receiving optimization parameters obtained by the central node based on the fusion parameters, and updating the data fusion model based on the optimization parameters includes:

[0031] Calculating a model index of the data fusion model based on the modality fusion data, the global data prototype, and the modality mapping data;

[0032] Updating the fusion parameters of the data fusion model according to the model indicators, and sending the fusion parameters to the central node;

[0033] Receiving the optimization parameter obtained by the central node performing an average aggregation operation on the fusion parameter of each edge node;

[0034] If the data fusion model does not meet the preset conditions, the optimized parameters are used as new fusion parameters and the data fusion model is repeatedly trained.

[0035] In some embodiments of the present application, each of the modal data includes a data label; and calculating the model index of the data fusion model based on the modal fusion data, the global data prototype, and the modal mapping data includes:

[0036] Inputting the modality fusion data into the data fusion model for prediction to obtain a prediction label;

[0037] Calculating the cross entropy loss of the data fusion model according to the data label and the predicted label;

[0038] Calculating the contrast loss of the edge node according to the global data prototype and the modal mapping data;

[0039] The model index is calculated based on the cross entropy loss and the contrast loss.

[0040] In a second aspect, an embodiment of the present application further provides a data fusion device for a distributed system, applying the data fusion method for a distributed system as described in the embodiment of the first aspect of the present application, including:

[0041] A mapping module, configured to obtain at least one modal data of the edge node and map each modal data using a pre-trained modal mapping model to obtain corresponding modal mapping data;

[0042] A prototype module, configured to obtain a data prototype corresponding to each of the modal data and transmit the data prototype to the central node;

[0043] A global module is used to obtain a global data prototype corresponding to each modality obtained by the central node performing an aggregation operation on the data prototype;

[0044] A comparison module, configured to compare and map the modal mapping data with the global data prototype to obtain pooled modal data;

[0045] An updating module is used to input the pooled modal data into a data fusion model for training to obtain fusion parameters, send the fusion parameters to the central node, receive optimization parameters obtained by the central node based on the fusion parameters, and update the data fusion model based on the optimization parameters.

[0046] In a third aspect, an embodiment of the present application further provides an electronic device comprising a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, it implements the data fusion method of the distributed system as described in the embodiment of the first aspect of the present application.

[0047] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium, wherein the storage medium stores a program, and the program is executed by a processor to implement the data fusion method of a distributed system as described in the embodiment of the first aspect of the present application.

[0048] The embodiments of the present application include at least the following beneficial effects:

[0049] The embodiments of the present application provide a data fusion method, device, electronic device and storage medium for a distributed system, wherein the distributed system includes a central node and at least one edge node, and each edge node is communicatively connected to the central node. In the method, at least one modal data of the edge node is first obtained, and each modal data is mapped using a pre-trained modal mapping model to obtain corresponding modal mapping data. Then, a data prototype corresponding to each modal data is obtained, and the data prototype is transmitted to the central node, and then a global data prototype corresponding to each modality obtained by the central node performing an aggregation operation on the data prototype is obtained. In this way, the collected modal data is not directly transmitted, but information is transmitted by transmitting the data prototype, which can effectively protect the privacy data of different edge nodes. The modal mapping data is then compared and mapped with the global data prototype to obtain pooled modal data and input into the data fusion model for training to obtain fusion parameters. The fusion parameters are then sent to the central node, and the optimization parameters obtained by the central node based on the fusion parameters are received, so that the data fusion model is updated based on the optimization parameters. In this way, data of different modalities are compared and mapped, and the modal completion of different edge nodes is aligned for model training and updating, which solves the problem of large differences in data modalities. A high-precision data fusion model suitable for different modal data in different edge nodes is obtained, thereby improving the accuracy of data processing results.

[0050] Additional aspects and advantages of the present application will be given in part in the description below, and in part will become obvious from the description below, or will be learned through practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the description of the embodiments in conjunction with the following drawings, in which:

[0052] Figure 1 This is a flow chart of a data fusion method for a distributed system provided by one embodiment of the present application;

[0053] Figure 2 yes Figure 1 Flow diagram of step S102;

[0054] Figure 3 yes Figure 1 Flow diagram of step S103;

[0055] Figure 4 yes Figure 1 Flow chart of step S104;

[0056] Figure 5 yes Figure 1 Flow diagram of step S105;

[0057] Figure 6 yes Figure 1 Another flow chart of step S105;

[0058] Figure 7 yes Figure 6 Flow diagram of step S601;

[0059] Figure 8 This is a flow chart of a data fusion method for a distributed system provided by an embodiment of the present application;

[0060] Figure 9 This is a schematic diagram of a data fusion device module of a distributed system provided by one embodiment of the present application;

[0061] Figure 10 This is a schematic diagram of the structure of an electronic device provided by an embodiment of the present application.

[0062] Reference numerals: mapping module 100 , prototype module 200 , global module 300 , comparison module 400 , update module 500 , electronic device 1000 , processor 1001 , memory 1002 . DETAILED DESCRIPTION

[0063] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0064] The following describes in detail embodiments of the present application. Examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present application and are not to be construed as limiting the present application.

[0065] In the description of this application, it should be understood that descriptions involving orientations, such as up, down, front, back, left, right, etc., indicating orientations or positional relationships, are based on the orientations or positional relationships shown in the accompanying drawings. They are only for the convenience of describing this application and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation. Therefore, they cannot be understood as limitations on this application.

[0066] In the description of this application, "several" means one or more, "many" means more than two, "greater than," "less than," and "exceed" are understood to exclude the number itself, while "above," "below," and "within" are understood to include the number itself. The terms "first" and "second" are used solely to distinguish technical features and are not to be construed as indicating or implying relative importance, or as implicitly specifying the number or order of the technical features indicated.

[0067] In the description of this application, unless otherwise clearly defined, terms such as setting, installing, and connecting should be understood in a broad sense, and technicians in the relevant technical field can reasonably determine the specific meanings of the above terms in this application based on the specific content of the technical solution.

[0068] A distributed system is a network composed of multiple nodes that communicate with each other and work collaboratively to achieve a common goal. With the continuous advancement of science and technology, the framework of distributed systems has provided strong support for concepts such as smart cities, intelligent transportation, and smart campuses. For example, a smart campus deploys distributed IoT data collection devices at enterprise nodes within the campus to collect relevant data from each enterprise. This data is then sent to the cloud for data fusion and processing, and the results are used to assist different enterprises in intelligent management.

[0069] However, the data collected by various enterprises within the park may involve trade secrets or personal privacy. Furthermore, due to their diverse business scopes, each enterprise uses different types of equipment to collect data for daily management. Therefore, the different modal data collected by these different devices may belong to different parameter spaces. Directly processing data from these modalities using a homogeneous model may yield inaccurate predictions. If each enterprise's data is processed and analyzed only internally, their data processing capabilities and available reference data are limited, making it difficult to train an accurate and appropriate model to assist in daily management. Therefore, a method is urgently needed to collaboratively train a global data model without leaking private data from each enterprise.

[0070] Federated learning, a related technology, provides a distributed training model where all participants can jointly train models, eliminating the need for centralized data processing. However, traditional federated learning typically assumes that data processed by different clients belongs to the same modality. Furthermore, this technology pays little attention to protecting the privacy of data from different enterprises within a campus. Data is typically transmitted to the cloud for centralized processing, and the differences in data modalities caused by different collection equipment across enterprises during the information collection process are ignored. Consequently, it is difficult to obtain high-precision data models that are applicable to different enterprises.

[0071] Based on this, embodiments of the present application provide a data fusion method, apparatus, electronic device, and storage medium for a distributed system. First, at least one modal data set is acquired from an edge node. Each modal data set is mapped using a pre-trained modal mapping model to obtain corresponding modal mapping data. A data prototype corresponding to each modal data set is then acquired and transmitted to a central node. The central node then aggregates the data prototypes to obtain a global data prototype corresponding to each modality. This method effectively protects the privacy of data from different edge nodes by transmitting data prototypes rather than directly transmitting the collected modal data. The modal mapping data is then compared and mapped with the global data prototype to obtain pooled modal data, which is then input into a data fusion model for training to obtain fusion parameters. The fusion parameters are then sent to the central node, which receives optimized parameters derived from the fusion parameters and updates the data fusion model based on the optimized parameters. This method compares and maps data from different modalities, aligning the modal completions of different edge nodes for model training and updating. This addresses the issue of large data modality differences, resulting in a high-precision data fusion model applicable to data from different modalities at different edge nodes, thereby improving the accuracy of data processing results.

[0072] The embodiments of the present application provide a data fusion method, device, electronic device, and storage medium for a distributed system, which are specifically illustrated by the following embodiments. First, the data fusion method for a distributed system in the embodiments of the present application is described.

[0073] The data fusion method for a distributed system provided in an embodiment of the present application relates to the field of computer technology, and in particular to the field of artificial intelligence technology. The data fusion method for a distributed system provided in an embodiment of the present application can be applied to a terminal, can be applied to a server side, or can be a computer program running in a terminal or a server side. For example, a computer program can be a native program or a software module in an operating system; it can be a local application, that is, a program that needs to be installed in an operating system before it can run, such as a client that supports data fusion of a distributed system, that is, a program that can be run only by downloading it into a browser environment. In short, the above-mentioned computer program can be any form of application, module or plug-in. Among them, the terminal communicates with the server through a network. The data fusion method for the distributed system can be executed by a terminal or a server, or by a terminal and a server in collaboration.

[0074] In some embodiments, the terminal can be a smartphone, tablet computer, laptop computer, desktop computer, or smartwatch. The server can be an independent server, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms; it can also be a service node in a blockchain system, where each service node in the blockchain system forms a peer-to-peer (P2P) network. The P2P protocol is an application layer protocol running on top of the Transmission Control Protocol (TCP). The server can be installed with a server for the data fusion system of the distributed system, through which the terminal can interact. For example, the server can be installed with corresponding software, which can be an application that implements the data fusion method of the distributed system, but is not limited to the above forms. The terminal and the server can be connected via a communication connection method such as Bluetooth, Universal Serial Bus (USB), or a network, which is not limited in this embodiment.

[0075] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronic devices, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, and the like. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present invention can also be practiced in distributed computing environments in which tasks are performed by remote processing devices connected via a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media, including storage devices.

[0076] The following describes a data fusion method for a distributed system in an embodiment of the present application. The distributed system includes a central node and at least one edge node, each of which is in communication with the central node. It will be appreciated that the distributed system in this application uses a smart park as an example, where the central node is the cloud system of the smart park, and the edge nodes are the clients of each enterprise in the smart park. This application can also be applied to distributed systems such as smart transportation and smart cities, and this embodiment does not limit this.

[0077] Figure 1 This is an optional flow chart of the data fusion method for a distributed system provided in an embodiment of the present application. Figure 1 The method may include but is not limited to steps S101 to S105. Figure 1 The order of step S101 to step S105 is not specifically limited, and the order of steps can be adjusted or some steps can be reduced or added according to actual needs.

[0078] Step S101: Acquire at least one modal data of an edge node, and use a pre-trained modal mapping model to map each modal data to obtain corresponding modal mapping data.

[0079] In some embodiments, edge nodes are clients participating in data collection in various enterprises in the smart park, such as visible light and infrared cameras. The clients of each enterprise are randomly sampled to obtain N edge nodes, and each edge node collects at least one modal data, where the modal data includes text data, audio data, and image data. Assume that the modal data collected by different edge nodes has M modalities. For edge node i, the modal data it contains corresponds to M i species, i∈[1,N], M i ∈[1,M].

[0080] In some embodiments, each modality data is mapped using a pre-trained modality mapping model to obtain corresponding modality mapping data. Specifically, the pre-trained modality mapping model can be a deep learning model, of which there are M types. For example, the text mapping model BERT is used to map text data to obtain text mapping data; the audio mapping model ViT is used to map audio data to obtain audio mapping data; and the video mapping model ResNet-50 is used to map video data to obtain video mapping data. This embodiment does not impose any restrictions on this.

[0081] Step S102: Obtain the data prototype corresponding to each modal data, and transmit the data prototype to the central node.

[0082] In some embodiments, calculations are performed on each modal data type at each edge node, and the corresponding data prototype is obtained and transmitted to the central node, rather than directly transmitting the modal data collected by the edge node. This protects the privacy of each enterprise's data, and performing calculations on the enterprise's local server can effectively reduce communication overhead and costs.

[0083] Step S103: obtaining the global data prototype corresponding to each modality obtained by the central node performing an aggregation operation on the data prototype.

[0084] In some embodiments, the central node receives the data prototypes transmitted by each edge node, and performs aggregation operations on the data prototypes of different modes of all edge nodes respectively, so as to obtain the global data prototype corresponding to each mode, that is, M global data prototypes, and then transmits the M global data prototypes back to each edge node.

[0085] Step S104: Compare and map the modal mapping data with the global data prototype to obtain pooled modal data.

[0086] In some embodiments, a contrastive learning approach is used to sequentially contrast and map the modal mapping data corresponding to each modal data type in the edge node with M types of global data prototypes. This means that each modal mapping data type can generate M types of contrastive mapping data, which can then be used to calculate pooled modal data. This completes and aligns the modal data in each edge node, effectively addressing the modal differences between different edge nodes and thus resolving the issue of unstable model training caused by modality loss in the data collected by various enterprises, thereby improving training accuracy.

[0087] In step S105 , the pooled modal data is input into the data fusion model for training to obtain fusion parameters, the fusion parameters are sent to the central node, and the optimization parameters obtained by the central node based on the fusion parameters are received, and the data fusion model is updated based on the optimization parameters.

[0088] In some embodiments, the data fusion model is a high-precision model obtained by joint training of different edge nodes. It is applicable to the modal data of different clients in various enterprises in the smart park, and can help enterprises achieve intelligent management and effectively improve enterprise management efficiency. Specifically, each edge node inputs the pooled modal data into the data fusion model and iteratively trains the data fusion model using a preset loss function to obtain fusion parameters and send them to the central node. After receiving the fusion parameters of each edge node, the central node performs an average aggregation operation to obtain the optimized parameters and transmits them back to the edge node. The edge node then continues to train and update the data fusion model based on the optimized parameters.

[0089] As you can understand, as edge nodes continuously receive new data, the data fusion model is constantly updated. This allows for comparative mapping of different modal data, aligning the modal completions of different edge nodes for model training and updating. This addresses the issue of large data modality disparity, resulting in a high-precision data fusion model applicable to data of different modalities at different edge nodes.

[0090] Reference Figure 2 As shown, in some embodiments of the present application, obtaining the data prototype corresponding to each modal data in the above step S102 may include but is not limited to the following steps S201 to S203.

[0091] Step S201: Obtain the total number of data samples of modal data in the edge node.

[0092] In some embodiments, the modal data collected by the edge node is represented as a modal data set. Specifically, the modal data set of edge node i is recorded as in, represents the kth modal data of the mth mode, Represents modal data Correspondingly, the total number of data samples can be expressed as |D i |, so k∈[1,|D i |]. It is understandable that, assuming that the modal data collected by edge node 1 includes text data and audio data If there are two modalities, D1 includes |D1| text data and |D1| audio data, and the total number of data samples of modal data in edge node 1 is |D1|.

[0093] Step S202 , performing a sum operation on the modal mapping data of the same modal data to obtain a modal data prototype.

[0094] In some embodiments, the pre-trained modality mapping model can be expressed as φ = {φ m},m∈[1,M], using φ m For the kth modal data of the mth modal in edge node i Mapping is performed to obtain modal mapping data, which is recorded as Right now Furthermore, the modal mapping data of the same modal data are summed up to obtain the modal data prototype. For example, the modal mapping data of the mth mode are summed up, that is, This embodiment does not impose any limitation on this.

[0095] Step S203: Divide the modal data prototype by the total number of data samples to obtain the data prototype.

[0096] In some embodiments, the modal data prototype is divided by the total number of data samples to obtain the corresponding data prototype. Specifically, for the data prototype of the mth modality in the edge node i, it is denoted as c i,m ,therefore

[0097] Reference Figure 3 As shown, in some embodiments of the present application, obtaining the global data prototype corresponding to each modality obtained by the central node performing an aggregation operation on the data prototype in the above step S103 may include but is not limited to the following steps S301 to S304.

[0098] Step S301: Use the central node to add up the total number of data samples of each edge node to obtain the total number of global samples.

[0099] In some embodiments, the central node adds up the total number of data samples of each edge node to obtain the total number of global samples, which is recorded as |D|. Therefore, |D|=|D1|+...+|D N |.

[0100] Step S302: Using the central node, the total number of data samples of the edge node is divided by the total number of global samples to obtain the edge coefficient of the edge node.

[0101] In some embodiments, the central node is used to divide the total number of data samples of different edge nodes by the total number of global samples in turn to obtain the edge coefficient corresponding to the edge node. For example, for edge node i, its edge coefficient is expressed as

[0102] Step S303: Multiply the edge coefficient by the data prototype using the central node to obtain a reference data prototype of the edge node.

[0103] In some embodiments, the edge coefficients are multiplied by the data prototypes in sequence using the central node to obtain the reference data prototype of the edge node. For example, the reference data prototype of the mth mode in the edge node i is recorded as

[0104] Step S304: obtaining a global data prototype obtained by the central node performing a sum operation on the reference data prototype of each edge node.

[0105] In some embodiments, based on the same modality data, the central node performs a sum operation on the reference data prototypes of N edge nodes to obtain the global data prototype corresponding to the modality data and transmit it back to each edge node. For example, for the global data prototype of m modalities, it is recorded as Right now For example, the global data prototype for text mode is recorded as The global data prototype for the audio modality is recorded as The global data prototype for the image modality is recorded as This embodiment does not impose any limitation on this.

[0106] Reference Figure 4 As shown, in some embodiments of the present application, in the above step S104, the modal mapping data is compared and mapped with the global data prototype to obtain pooled modal data, which may include but is not limited to the following steps S401 to S403.

[0107] Step S401: Acquire global modal data according to the global data prototype.

[0108] In some embodiments, after receiving the global data prototype corresponding to each modality from the central node, the edge node can obtain global modal data based on the global data prototype, and the global modal data includes each modal data of each edge node. Exemplarily, for the modal data of N edge nodes, there are three modalities: text data, audio data, and image data. Therefore, each edge node will obtain the corresponding three global data prototypes from the central node. The global modal data obtained from the three global data prototypes includes text modality, audio modality, and image modality. This embodiment does not impose any restrictions on this.

[0109] Step S402 : performing comparison mapping on each modality mapping data and the global modality data in sequence to obtain corresponding comparison mapping data.

[0110] In some embodiments, each modality mapping data is sequentially compared and mapped with the global modality data to obtain corresponding comparison mapping data. Specifically, the mapping function is defined as ψ={ψ i,s}, i∈[1,N], s∈[1,M], so for the contrast mapping data of m modes in edge node i That is For example, assuming that the modal data of edge node i has only one modality, image data, when M=3, the obtained contrast mapping data has three corresponding modes, namely and This embodiment does not impose any limitation on this.

[0111] Step S403: performing an average pooling operation on the contrast mapping data to obtain pooled modal data.

[0112] In some embodiments, for edge node i, the contrast mapping data corresponding to each modality is average pooled to obtain pooled modality data, which is recorded as Specifically Therefore, the modal data is completed and aligned in each edge node through comparative mapping, which solves the problem of large modal differences caused by modal missing in edge nodes.

[0113] Reference Figure 5 As shown, in some embodiments of the present application, in the above step S105, the pooled modality data is input into the data fusion model for training to obtain fusion parameters, which may include but is not limited to the following steps S501 to S505.

[0114] Step S501 : performing modal learning based on the pooled modal data to obtain modal parameters of each modal data.

[0115] In some embodiments, modal learning is performed based on the pooled modal data to obtain modal parameters of each modal data. Specifically, a multilayer perceptron (MLP) can be used for modal learning. The modal parameters of the mth modal in the edge node i are recorded as Therefore, modal learning based on pooled modal data can be written as This embodiment does not impose any limitation on this.

[0116] Step S502: Calculate the modal weight corresponding to each modal data based on the global modal data.

[0117] In some embodiments, the modal weight corresponding to each modal data is calculated based on the global modal data. Specifically, the total number of modal types is obtained based on the global modal data, which is recorded as M. Furthermore, in this embodiment, the weighted average mode is adopted, so the modal weight corresponding to each modal data is The modal weight of the m mode is denoted as Π m ,Right now This implementation does not impose any restrictions on this.

[0118] Step S503: multiply the modal weight by the modal parameter to obtain fused data of the modal data.

[0119] In some embodiments, for the m-mode modal data in edge node i, the modal weight is multiplied by the modal parameter to obtain the fused data of the m-mode data, that is,

[0120] Step S504 : performing a sum operation on the fusion data of each modality data to obtain modality fusion data.

[0121] In some embodiments, in edge node i, the fusion data of each modality data is summed up to obtain modality fusion data, which is recorded as h i ,therefore

[0122] Step S505: input the modal fusion data into the data fusion model for training to obtain fusion parameters.

[0123] Multimodal fusion is achieved through a weighted averaging model to obtain modal fusion data and input it into a data fusion model for training. In some embodiments, the results obtained by the data fusion model and the actual results are calculated according to a preset loss function to obtain a loss value. Based on the loss value, a model update is performed for T = 10 rounds through stochastic gradient descent to obtain fusion parameters. It is understood that those skilled in the art can set the value of T according to actual needs, and this embodiment does not limit this.

[0124] Reference Figure 6As shown, in some embodiments of the present application, in the above step S105, the pooled modal data is input into the data fusion model for training to obtain fusion parameters, the fusion parameters are sent to the central node, and the optimization parameters obtained by the central node based on the fusion parameters are received, and the data fusion model is updated based on the optimization parameters, which may include but is not limited to the following steps S601 to S604.

[0125] Step S601 : calculating the model index of the data fusion model based on the modal fusion data, the global data prototype and the modal mapping data.

[0126] In some embodiments, a model index of the data fusion model is calculated based on the modal fusion data, the global data prototype, and the modal mapping data. It is understood that the model index is used to guide the update of the data fusion model. Specifically, the model index can be a model loss, which is not limited in this embodiment.

[0127] Step S602: update the fusion parameters of the data fusion model according to the model indicators, and send the fusion parameters to the central node.

[0128] In some embodiments, for edge node i, the fusion parameter w of the data fusion model is updated according to the model index i The fusion parameters can be fully connected network parameters. Specifically, the data fusion model is updated for T rounds according to the model indicators, that is, after the fusion parameters are updated for T rounds, the fusion parameters are sent to the central node.

[0129] Step S603: receiving the optimized parameters obtained by the central node performing an average aggregation operation on the fusion parameters of each edge node.

[0130] In some embodiments, the central node performs an average aggregation operation on the fusion parameters of each edge node to obtain the optimized parameters And optimize the parameters Back to each edge node.

[0131] Step S604: If the data fusion model does not meet the preset conditions, the optimized parameters are used as new fusion parameters and the data fusion model is repeatedly trained.

[0132] In some embodiments, if the data fusion model does not meet preset conditions, for example, the data fusion model does not reach the preset training rounds or does not reach the preset convergence conditions, the optimized parameters are used as new fusion parameters, and the data fusion model is repeatedly trained based on the modal fusion data.

[0133] Reference Figure 7 As shown, in some embodiments of the present application, each modality data also includes a data label y sCalculating the model index of the data fusion model based on the modal fusion data, the global data prototype and the modal mapping data in the above step S601 may include but is not limited to the following steps S701 to S704.

[0134] Step S701: input the modal fusion data into the data fusion model for prediction to obtain a predicted label.

[0135] In some embodiments, for edge node i, the modal fusion data is input into the data fusion model for prediction, and a prediction label can be obtained, which is recorded as w(h i ).

[0136] Step S702: Calculate the cross entropy loss of the data fusion model based on the data label and the predicted label.

[0137] In some embodiments, for edge node i, the cross entropy loss of the data fusion model is calculated based on the data label and the predicted label, which is denoted as L(w(h i ),y s ).

[0138] Step S703: Calculate the contrast loss of the edge node based on the global data prototype and the modal mapping data.

[0139] In some embodiments, the contrast loss of edge nodes is calculated based on the global data prototype and the modality mapping data:

[0140]

[0141] in, Global data prototype of m mode Through the mapping function ψ i,s Mapping is performed, and sim(·,·) represents the cosine similarity.

[0142] Step S704: Calculate the model index based on the cross entropy loss and the contrast loss.

[0143] In some embodiments, for edge node i, the model index fi is calculated based on the cross entropy loss and contrast loss i :

[0144]

[0145] Among them, |D i | is the total number of data samples at edge node i, and λ is a hyperparameter. The update of the data fusion model is guided by the model indicators.

[0146] The following is a complete example to illustrate this application:

[0147] Reference Figure 8The data fusion method flow chart of the distributed system shown in the figure shows that the modal data in N edge nodes has three modalities: text, audio, and image. The data fusion model can be an equipment health management model used to monitor the equipment operation status, thereby helping enterprises in the smart park to achieve intelligent management. For example, for edge node 1, only text modality and audio mode Two modalities, for edge node i there is only image modality A modality.

[0148] First, using the pre-trained modality mapping model φ={φ m} to map and obtain the corresponding modal mapping data For example Then calculate the data prototype corresponding to each modal data based on the modal mapping data And transmit it to the central node, use the central node to perform aggregation operations to obtain the global data prototype corresponding to each mode The three global data prototypes obtained and The modal mapping data is then compared and mapped with the global modal data obtained based on the global data prototype to obtain the comparative mapping data corresponding to each modality. Then perform average pooling on the contrast mapping data to obtain the pooled modal data corresponding to each modality Then, modal learning is performed based on the pooled modal data to obtain the modal parameters corresponding to each mode. Based on the modal parameters and modal weight π m Fuse different modal data to obtain modal fusion data According to the modal fusion data, the data fusion model is trained to obtain the fusion parameter w i And transmit it to the central node for average aggregation operation to obtain the optimized parameters The data is then transmitted back to each edge node, which continues to update the data fusion model according to the optimized parameters until the preset conditions are met, thereby obtaining a high-precision data fusion model applicable to different modal data in different edge nodes.

[0149] Therefore, a pre-trained modal mapping model is used to map different modal data, effectively reducing local computing costs. By transmitting information through the transmission of "data prototypes" rather than directly transmitting modal data collected by edge nodes, the privacy of each enterprise's data is protected. Furthermore, performing calculations on the enterprise's local servers can effectively reduce communication overhead and costs. Furthermore, a comparative learning method is used to compare and map data of different modalities, aligning the modal completions of different edge nodes for model training and updating. This solves the problem of large differences in data modalities and thus improves the accuracy of data processing results.

[0150] The embodiment of the present invention also provides a data fusion device for a distributed system, which can implement the data fusion method of the distributed system. Figure 9 As shown, in some embodiments of the present application, the data fusion device of the distributed system includes:

[0151] A mapping module 100 is configured to obtain at least one modal data of an edge node and map each modal data using a pre-trained modal mapping model to obtain corresponding modal mapping data;

[0152] The prototype module 200 is used to obtain the data prototype corresponding to each modal data and transmit the data prototype to the central node;

[0153] The global module 300 is used to obtain the global data prototype corresponding to each modality obtained by the central node performing an aggregation operation on the data prototype;

[0154] Comparison module 400, used to compare and map the modal mapping data with the global data prototype to obtain pooled modal data;

[0155] The updating module 500 is used to input the pooled modal data into the data fusion model for training, obtain fusion parameters, send the fusion parameters to the central node, receive the optimization parameters obtained by the central node based on the fusion parameters, and update the data fusion model based on the optimization parameters.

[0156] The specific implementation of the data fusion device of the distributed system of this embodiment is basically the same as the specific implementation of the data fusion method of the distributed system described above, and will not be described in detail here.

[0157] Figure 10 The electronic device 1000 provided in an embodiment of the present application is shown. The electronic device 1000 includes: a processor 1001, a memory 1002, and a computer program stored in the memory 1002 and executable on the processor 1001. When the computer program is executed, it is used to execute the above-mentioned distributed system data fusion method.

[0158] The processor 1001 and the memory 1002 may be connected via a bus or other means.

[0159] Memory 1002, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer executable programs, such as the distributed system data fusion method described in the embodiments of this application. Processor 1001 implements the distributed system data fusion method described above by executing the non-transitory software programs and instructions stored in memory 1002.

[0160] The memory 1002 may include a program storage area and a data storage area, wherein the program storage area may store an operating system and application programs required for at least one function; the data storage area may store a data fusion method for executing the above-mentioned distributed system. In addition, the memory 1002 may include a high-speed random access memory 1002, and may also include a non-volatile memory 1002, such as at least one storage device memory device, a flash memory device or other non-volatile solid-state memory device. In some embodiments, the memory 1002 may optionally include a memory 1002 remotely arranged relative to the processor 1001, and these remote memories 1002 may be connected to the electronic device 1000 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0161] The non-transient software programs and instructions required to implement the above-mentioned distributed system data fusion method are stored in the memory 1002. When executed by one or more processors 1001, the above-mentioned distributed system data fusion method is executed, for example, Figure 1 Steps S101 to S105 of the method, Figure 2 Steps S201 to S203 of the method, Figure 3 Steps S301 to S304 of the method, Figure 4 Steps S401 to S403 of the method, Figure 5 Steps S501 to S505 of the method, Figure 6 Steps S601 to S604 of the method, Figure 7 Method steps S701 to S704 in .

[0162] The embodiment of the present application also provides a storage medium, which is a computer-readable storage medium, and the storage medium stores a computer program, which implements the data fusion method of the above-mentioned distributed system when the computer program is executed by the processor. The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely arranged relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof.

[0163] The data fusion method, device, electronic device, and storage medium for a distributed system provided by the embodiments of the present application obtain at least one modal data from an edge node and map each modal data using a pre-trained modal mapping model to obtain corresponding modal mapping data. A data prototype corresponding to each modal data is then obtained and transmitted to a central node. The central node then aggregates the data prototypes to obtain a global data prototype corresponding to each modality. The modal mapping data is then compared and mapped with the global data prototype to obtain pooled modal data, which is then input into a data fusion model for training to obtain fusion parameters. The fusion parameters are then sent to the central node, and the central node receives optimized parameters obtained based on the fusion parameters, thereby updating the data fusion model based on the optimized parameters. Thus, different modal data are mapped using a pre-trained modal mapping model, effectively reducing local computational costs. By using the "data prototype" method for information transmission rather than directly transmitting modal data collected by edge nodes, the privacy data of each enterprise is protected. Furthermore, performing computations on the enterprise's local server effectively reduces communication overhead and communication costs. In addition, the contrastive learning method is used to compare and map data of different modalities, so that the modal completion of different edge nodes is aligned for model training and updating, which solves the problem of large differences in data modalities and thus improves the accuracy of data processing results.

[0164] The embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, i.e., they may be located in one place or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of this embodiment.

[0165] Those skilled in the art will appreciate that all or some of the steps and systems in the method disclosed above can be implemented as software, firmware, hardware, and appropriate combinations thereof. Some physical components or all physical components can be implemented as software executed by a processor, such as a central processing unit, a digital signal processor, or a microprocessor, or implemented as hardware, or implemented as an integrated circuit, such as an application-specific integrated circuit. Such software can be distributed on a computer-readable medium, and the computer-readable medium can include computer storage media (or non-transitory media) and communication media (or temporary media). As known to those skilled in the art, the term computer storage media includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data). Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory, or other memory technology, CD-ROM, digital versatile disks (DVD), or other optical disk storage, magnetic cassettes, magnetic tapes, storage device storage, or other magnetic storage devices, or any other medium that can be used to store desired information and can be accessed by a computer. Furthermore, as is well known to those skilled in the art, communication media typically includes computer-readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transport mechanism, and may include any information delivery media.

[0166] It should also be understood that the various implementations provided in the embodiments of this application can be arbitrarily combined to achieve different technical effects. The above is a specific description of the preferred implementation of this application, but this application is not limited to the above implementation. Those skilled in the art can also make various equivalent modifications or substitutions under the conditions that do not violate the spirit of this application.

Claims

1. A data fusion method for a distributed system, characterized in that: The distributed system includes a central node and at least one edge node, each edge node being communicatively connected to the central node; and the method includes: Acquire at least one modal data of the edge node, and map each modal data using a pre-trained modal mapping model to obtain corresponding modal mapping data; Obtaining a data prototype corresponding to each of the modal data, and transmitting the data prototype to the central node; Obtaining a global data prototype corresponding to each mode obtained by the central node performing an aggregation operation on the data prototype; Comparing and mapping the modal mapping data with the global data prototype to obtain pooled modal data; Inputting the pooled modality data into a data fusion model for training to obtain fusion parameters, sending the fusion parameters to the central node, receiving optimization parameters obtained by the central node based on the fusion parameters, and updating the data fusion model based on the optimization parameters; The obtaining of a data prototype corresponding to each modal data includes: Obtaining a total number of data samples of the modal data in the edge node; performing a sum operation on the modal mapping data of the same modal data to obtain a modal data prototype; The modal data prototype is divided by the total number of data samples to obtain the data prototype.

2. The data fusion method of a distributed system according to claim 1, characterized in that: The obtaining of the global data prototype corresponding to each modality obtained by the central node performing an aggregation operation on the data prototype includes: Using the central node to sum the total number of data samples of each edge node to obtain a global total number of samples; Using the central node, the total number of data samples of the edge node is divided by the total number of global samples to obtain an edge coefficient of the edge node; Multiplying the edge coefficient by the data prototype using the central node to obtain a reference data prototype of the edge node; The global data prototype is obtained by the central node performing a sum operation on the reference data prototype of each edge node.

3. The data fusion method of a distributed system according to claim 1, characterized in that: The comparing and mapping the modal mapping data with the global data prototype to obtain pooled modal data includes: Acquire global modal data according to the global data prototype; wherein the global modal data includes each type of modal data of each edge node; Comparing and mapping each of the modal mapping data with the global modal data in sequence to obtain corresponding comparative mapping data; An average pooling operation is performed on the contrast mapping data to obtain the pooled modal data.

4. The data fusion method for a distributed system according to claim 1 or 3, characterized in that: Inputting the pooled modality data into a data fusion model for training to obtain fusion parameters includes: Performing modal learning based on the pooled modal data to obtain modal parameters of each modal data; Calculating the modal weight corresponding to each modal data according to the global modal data; Multiplying the modal weight by the modal parameter to obtain fused data of the modal data; performing a summing operation on the fused data of each modality data to obtain modality fusion data; The modality fusion data is input into the data fusion model for training to obtain the fusion parameters.

5. The data fusion method of a distributed system according to claim 4, characterized in that: The step of inputting the pooled modality data into a data fusion model for training to obtain fusion parameters, sending the fusion parameters to the central node, receiving optimization parameters obtained by the central node based on the fusion parameters, and updating the data fusion model based on the optimization parameters includes: Calculating a model index of the data fusion model based on the modality fusion data, the global data prototype, and the modality mapping data; Updating the fusion parameters of the data fusion model according to the model indicators, and sending the fusion parameters to the central node; Receiving the optimization parameter obtained by the central node performing an average aggregation operation on the fusion parameter of each edge node; If the data fusion model does not meet the preset conditions, the optimized parameters are used as new fusion parameters and the data fusion model is repeatedly trained.

6. The data fusion method of a distributed system according to claim 5, characterized in that: Each of the modal data includes a data label; and calculating the model index of the data fusion model based on the modal fusion data, the global data prototype, and the modal mapping data includes: Inputting the modality fusion data into the data fusion model for prediction to obtain a prediction label; Calculating the cross entropy loss of the data fusion model according to the data label and the predicted label; Calculating the contrast loss of the edge node according to the global data prototype and the modal mapping data; The model index is calculated based on the cross entropy loss and the contrast loss.

7. A data fusion device for a distributed system, characterized in that: A data fusion method for a distributed system according to any one of claims 1 to 6, comprising: A mapping module, configured to obtain at least one modal data of the edge node and map each modal data using a pre-trained modal mapping model to obtain corresponding modal mapping data; A prototype module, configured to obtain a data prototype corresponding to each of the modal data and transmit the data prototype to the central node; A global module is used to obtain a global data prototype corresponding to each modality obtained by the central node performing an aggregation operation on the data prototype; A comparison module, configured to compare and map the modal mapping data with the global data prototype to obtain pooled modal data; an updating module, configured to input the pooled modal data into a data fusion model for training to obtain fusion parameters, send the fusion parameters to the central node, receive optimization parameters obtained by the central node based on the fusion parameters, and update the data fusion model based on the optimization parameters; The obtaining of a data prototype corresponding to each modal data includes: Obtaining a total number of data samples of the modal data in the edge node; performing a sum operation on the modal mapping data of the same modal data to obtain a modal data prototype; The modal data prototype is divided by the total number of data samples to obtain the data prototype.

8. An electronic device, characterized in that: The method comprises a memory and a processor, wherein the memory stores a computer program, and the processor implements the data fusion method of the distributed system according to any one of claims 1 to 6 when executing the computer program.

9. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a program, and when the program is executed by a processor, the data fusion method for a distributed system according to any one of claims 1 to 6 is implemented.

Images