Edge data processing method, system, device and medium
By dynamically updating the machine learning algorithm pool under the edge computing framework, selecting and optimizing the best algorithms required for processing data, the problem of lack of universality of machine learning algorithms in different fields in the existing technology is solved, and the data processing is improved Accuracy and robustness reduce error rates.
Patent Information
- Application Number
- CN202111105110.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-09-22
- Publication Date
- 2024-12-06
- Estimated Expiration
- 2041-09-22
AI Technical Summary
Machine learning algorithms under the existing edge computing framework lack universality when processing data in different fields, and fail to make full use of the historical data after model training to further optimize machine learning model parameters, resulting in waste of resources and inefficiency.
Under the edge computing framework, by saving the service request data sent by the user terminal at the edge node and transmitting it to the central cloud according to the data type, machine learning algorithm training is carried out when the preset storage threshold is reached, and the machine learning algorithm pool is dynamically updated, and the selection and Optimize the best algorithms required to process data, and improve the accuracy and robustness of data processing.
It realizes dynamic update of machine learning algorithm parameters under the edge computing framework, improves the accuracy of data processing, reduces error rates, enhances the robustness of machine learning algorithms, and improves the universality of processing.
Smart Images

Figure CN113886073B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of industrial Internet, and in particular to an edge data processing method, system, device and medium. Background Art
[0002] The Industrial Internet is a key infrastructure that connects the entire industrial system, the entire industrial chain, and the entire value chain, and supports the development of industrial intelligence. It is an emerging business model and application model formed by the deep integration of new-generation information technology and manufacturing. It is the core carrier for the Internet to expand from the consumer field to the production field and from the virtual economy to the real economy. It is also the commanding heights of a new round of global industrial competition.
[0003] In view of the business requests of various industrial equipment and industrial control systems in the industrial Internet, the complex industrial network environment, and the heterogeneity, heterogeneity, and heterogeneity of massive industrial data, the concept of edge computing is introduced to improve the computing and processing capabilities of data in the industrial Internet and reduce network latency in the industrial Internet. Edge computing adopts a distributed architecture, and the operation of applications, data and services previously processed by the network center node is handed over to the edge node in the network logic for processing. Edge computing decomposes large services into smaller and easier to process content, and distributes large services originally processed by the center node to the edge node. Edge nodes are deployed closer to user terminals. This feature significantly improves the data transmission speed and further reduces latency. At the same time, the edge nodes adopt a distributed computing architecture, which makes edge computing have the characteristics of distributed computing and can perform efficient data processing close to the user side. Edge computing can effectively alleviate network bandwidth pressure, enhance service responsiveness, and protect privacy data.
[0004] The edge nodes under the edge computing framework need to process a large amount of heterogeneous and heterogeneous data and need to get timely responses. The edge caching system based on machine learning can select suitable machine learning models to process data according to the characteristics of the data source, ensuring that different industrial Internet data are in. At the same time, the machine learning model can use the data requested by users to dynamically optimize the machine learning model parameters, further improve the performance of machine learning, and complete a virtuous cycle of data processing and model updating.
[0005] At present, there are several common methods for introducing machine learning algorithms to process data under the existing edge computing framework:
[0006] Existing technology 1: Medical data processing based on federated learning and edge computing.
[0007] Principle: Introduce edge computing framework in the medical field, adopt federated learning algorithm, use medical history data to train federated learning, and generate effective models to process medical data. Data is trained by exchanging encrypted parameters to ensure personal control over data, greatly reduce the risk of user data leakage, and realize medical disease data sharing and intelligent disease data processing.
[0008] Advantages: Federated learning is a relatively new machine learning algorithm at this stage. It is innovative in data processing in the medical field. Federated learning can jointly train data from different data sources while ensuring data privacy to obtain a better data processing model. It has the advantages of high security, strong big data analysis capabilities, convenient access, high efficiency and low cost.
[0009] Disadvantages: This method is only applicable to data in the medical field. It has a good effect in processing data in the medical field, but the effect is poor in other fields. It lacks a certain degree of versatility. At the same time, this method does not make full use of the historical data generated after model training to further improve the parameters of the machine learning model, wasting the available data accumulated later.
[0010] Existing technology 2: Commercial data processing based on machine learning and edge computing.
[0011] Principle: Obtain the business data to be processed through edge crawler network computing methods and user input information, preprocess the business data, use the preprocessed data to train the machine learning algorithm, and use edge distributed computing to filter data noise to reduce the pressure of the machine learning algorithm during the calculation process and improve training efficiency.
[0012] Advantages: Combining web crawler technology with machine learning, using edge crawler network computing methods can effectively obtain the data that needs to be processed. At the same time, data preprocessing can improve the training efficiency of machine learning, and data filtering technology is used to remove data noise, further improving the training efficiency of machine learning.
[0013] Disadvantages: This method is only applicable to data in the commercial field. It has a good effect in processing data in the commercial field, but the effect is poor in other fields. It lacks a certain degree of versatility. At the same time, this method does not make full use of the historical data generated after model training to further improve the parameters of the machine learning model, which wastes the available data accumulated later. In addition, the data preprocessing of crawler technology also adds a certain amount of resource overhead to the entire system. Summary of the invention
[0014] In order to solve at least one of the technical problems existing in the prior art to a certain extent, an object of the present invention is to provide an edge data processing method, system, device and medium.
[0015] The technical solution adopted by the present invention is:
[0016] An edge data processing method comprises the following steps:
[0017] After processing the service request sent by the user terminal, the edge node saves the data of the service request to the edge storage of the edge node according to the data type. When the storage capacity of the same type of data in the edge storage reaches the preset capacity storage threshold, the type data set of the same type of data is transmitted to the central cloud;
[0018] The central cloud receives and stores the type data set transmitted by the edge node, determines the machine learning algorithm that needs to be trained when the type data set reaches a preset training threshold, and trains the machine learning algorithm using the type data set;
[0019] Compare the performance of the algorithms before and after training, and synchronously update the machine learning algorithms with better performance to the machine learning algorithm pools of the central cloud and the edge nodes.
[0020] Furthermore, the edge data processing method further includes the following steps:
[0021] The user terminal sends a service request to the edge node;
[0022] The edge node determines, according to the data type of the service request, an optimal algorithm for processing the data of the type;
[0023] According to the determined optimal algorithm, obtaining, from the machine learning algorithm pool of the edge node, the user data of the service request processed by the corresponding machine learning algorithm;
[0024] After the data processing is completed, the edge node returns the processing result to the user terminal to complete the service request of the user terminal.
[0025] Further, the determining of the machine learning algorithm that needs to be trained includes:
[0026] Determine the machine learning algorithm that needs to be trained through the algorithm selector of the central cloud;
[0027] The algorithm selector includes a data type-machine learning algorithm mapping table, wherein the mapping table includes multiple data types and multiple machine learning algorithms, the data type includes at least one of an image data type, an audio data type, a video data type, or a document data type, and the machine learning algorithm includes at least one of principal component analysis, support vector machine, conditional random field, or a neural network model.
[0028] Further, the determining of the best algorithm for processing the type of data includes:
[0029] Determining, by an edge node algorithm selector, an optimal algorithm for processing the type of data;
[0030] The algorithm selector includes a data type-machine learning algorithm mapping table, wherein the mapping table includes multiple data types and multiple machine learning algorithms, the data type includes at least one of an image data type, an audio data type, a video data type, or a document data type, and the machine learning algorithm includes at least one of principal component analysis, support vector machine, conditional random field, or a neural network model.
[0031] Furthermore, comparing the performance of the algorithms before and after training, and synchronously updating the machine learning algorithm with better performance to the machine learning algorithm pools of the central cloud and the edge nodes, includes:
[0032] Compare the performance evaluation results before and after algorithm training, including the accuracy of data processing, the security of the model, and the robustness of the model;
[0033] Obtain the machine learning algorithm with better comprehensive evaluation and update it synchronously to the machine learning algorithm pool of the central cloud and the edge node.
[0034] Further, the edge node includes an edge memory, a machine learning algorithm pool, and an edge node algorithm selector;
[0035] The machine learning algorithm pool includes a plurality of machine learning algorithms, and the machine learning algorithm includes at least one of principal component analysis, support vector machine, conditional random field, and neural network model;
[0036] The edge storage is used to store the service request transmitted by the user terminal;
[0037] The machine learning algorithm pool is used to store the machine learning algorithms required to process the user data of the service request;
[0038] The edge node algorithm selector is used to select a corresponding machine learning algorithm from the machine learning algorithm pool of the edge node according to the characteristics of the service request transmitted by the user terminal.
[0039] Further, the central cloud includes a machine learning algorithm pool, a central cloud algorithm selector, and a dynamic update strategy;
[0040] The machine learning algorithm pool includes a plurality of machine learning algorithms, and the machine learning algorithm includes at least one of principal component analysis, support vector machine, conditional random field, and neural network model;
[0041] The machine learning algorithm pool is used to store the trained machine learning models in the central cloud;
[0042] The central cloud algorithm selector is used to select a corresponding machine learning algorithm from the machine learning algorithm pool according to the characteristics of the type of data set that reaches the training threshold for model training;
[0043] The dynamic update strategy is used to dynamically update the machine learning models in the machine learning algorithm pools of the central cloud and the edge nodes.
[0044] Another technical solution adopted by the present invention is:
[0045] An edge data processing system, comprising:
[0046] A data uploading module is used to, after the service request sent by the user terminal is processed, the edge node saves the data of the service request according to the data type to the edge storage of the edge node, and when the storage capacity of the same type of data in the edge storage reaches a preset capacity storage threshold, transmit the type data set of the same type of data to the central cloud;
[0047] A model training module is used for receiving and storing the type data set transmitted by the edge node in the central cloud, determining the machine learning algorithm to be trained when the type data set reaches a preset training threshold, and using the type data set to train the machine learning algorithm;
[0048] The algorithm update module is used to compare the performance of the algorithm before and after training, and synchronously update the machine learning algorithm with better performance to the machine learning algorithm pool of the central cloud and the edge node.
[0049] Another technical solution adopted by the present invention is:
[0050] An edge data processing device, comprising:
[0051] at least one processor;
[0052] at least one memory for storing at least one program;
[0053] When the at least one program is executed by the at least one processor, the at least one processor implements the above method.
[0054] Another technical solution adopted by the present invention is:
[0055] A computer-readable storage medium stores a program executable by a processor, wherein the program executable by the processor is used to execute the method described above when executed by the processor.
[0056] The beneficial effects of the present invention are: the present invention dynamically updates the edge node machine learning algorithm parameters under the edge computing framework, improves the accuracy of edge node data processing, reduces the error rate of edge node data processing, and enhances the robustness of the machine learning algorithm. BRIEF DESCRIPTION OF THE DRAWINGS
[0057] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the embodiments of the present invention or the drawings of related technical solutions in the prior art are introduced below. It should be understood that the drawings introduced below are only for the convenience of clearly describing some embodiments of the technical solutions of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.
[0058] Figure 1 is an overall system diagram for executing an edge data processing method in an embodiment of the present invention;
[0059] Figure 2 is a data flow diagram between a user terminal and an edge node in an embodiment of the present invention;
[0060] Figure 3 It is a data flow diagram between the edge node and the central cloud in an embodiment of the present invention;
[0061] Figure 4 is a flow chart of an edge data processing method in an embodiment of the present invention;
[0062] Figure 5 This is an application scenario diagram of an edge data processing method in an embodiment of the present invention. DETAILED DESCRIPTION
[0063] The embodiments of the present invention are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and are not to be construed as limitations of the present invention. For the step numbers in the following embodiments, they are only provided for the convenience of explanation, and the order between the steps is not limited in any way, and the execution order of each step in the embodiment can be adaptively adjusted according to the understanding of those skilled in the art.
[0064] In the description of the present invention, it should be understood that descriptions involving orientations, such as up, down, front, back, left, right, etc., and orientations or positional relationships indicated are based on the orientations or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be understood as a limitation on the present invention.
[0065] In the description of the present invention, "several" means one or more, "more" means more than two, "greater than", "less than", "exceed" etc. are understood as not including the number itself, and "above", "below", "within" etc. are understood as including the number itself. If there is a description of "first" or "second", it is only used for the purpose of distinguishing the technical features, and cannot be understood as indicating or implying the relative importance or implicitly indicating the number of the indicated technical features or implicitly indicating the order of the indicated technical features.
[0066] In the description of the present invention, unless otherwise clearly defined, terms such as setting, installing, connecting, etc. should be understood in a broad sense, and technicians in the relevant technical field can reasonably determine the specific meanings of the above terms in the present invention based on the specific content of the technical solution.
[0067] The edge data processing method based on a machine learning pool and a dynamic update strategy described in this embodiment is implemented in a specific system architecture as follows Figure 1 As shown, the system architecture includes a user terminal 101 , an edge node 102 , and a central cloud 103 .
[0068] The user terminal 101 is used to interact with the user, collect the user's request data, send the data to the edge node 102, receive the processing result of the edge node 102, and provide the processing result to the user in a user-readable manner. At the same time, the edge node 102 temporarily stores the data requested by the user locally, and after reaching the set storage threshold, transmits the data to the central cloud 103. The central cloud 103 collects the data of the edge nodes, and when the amount of data corresponding to a machine learning algorithm in the central cloud 103 reaches the set training threshold, the algorithm is trained. The specific implementation between the user terminal 101 and the edge node 102, and the edge node 102 and the central cloud 103 are respectively shown in Figure 2 and Figure 3 .
[0069] The specific process of data processing between the user terminal and the edge node is as follows: Figure 2 As shown, the user terminal interacts with the user through the user application 201 to provide services to the user. The user terminal receives the user's data request and requests data processing resources from the upper layer edge node.
[0070] The edge node is used to process the data request sent by the user terminal. The algorithm selector 204 selects the corresponding machine learning algorithm 206 from the machine learning algorithm pool 205 according to the data type requested by the user application 201 of the user terminal, processes the requested data, returns the processing result to the user terminal, and delivers it to the user through the user application 201. At the same time, the edge node saves the data copy 203 requested by the user in the edge storage 202 according to the data characteristics.
[0071] The data interaction between the edge node and the central cloud is as follows Figure 3 As shown. The edge memory 301 included in the edge node is used to store data 302 requested by the user terminal to the edge node; when the same feature data stored in the edge memory 301 reaches the storage threshold, the edge memory 301 sends the stored data to the cloud memory 306 in the central cloud, and clears the data in the edge memory 301. When the amount of the same feature data stored in the cloud memory 306 reaches the training threshold, the corresponding machine learning algorithm 310 is selected from the machine learning algorithm pool 308 through the algorithm selector 307, and the machine learning algorithm 310 is trained using the data set 309 that reaches the storage threshold, and the update strategy 311 determines whether to update the trained machine learning algorithm 310 to the machine learning algorithm pools 304 and 308.
[0072] like Figure 4 As shown, an edge data processing method based on a machine learning pool and a dynamic update strategy provided by an embodiment of the present invention includes the following steps:
[0073] S401: The user terminal requests data from the edge node;
[0074] S402: The algorithm selector in the edge node analyzes the type of data requested in step S401 and determines the machine learning algorithm required to process the data;
[0075] S403: The machine learning algorithm pool receives the machine learning algorithm determined in step S402;
[0076] S404: Selecting a machine learning algorithm required for the current request data through step S403;
[0077] S405: Use the machine learning algorithm of step S404 to process the data and return the processing result.
[0078] S406: while executing step S401, storing the data in the edge storage;
[0079] S407: After each request data is stored in the edge storage, determine whether the currently stored data has reached the preset storage threshold. If it has reached the preset storage threshold, execute the next step, otherwise do not execute the operation;
[0080] S408: Submitting the data in the edge storage to the central cloud;
[0081] S409: The cloud saves the data transmitted by the edge storage to the storage of the central cloud;
[0082] S410: Clearing historical data in the edge storage;
[0083] S411: Determine whether the amount of data in the central cloud storage reaches the threshold for training the machine learning model. If the training threshold is reached, execute the next step, otherwise do not execute the operation;
[0084] S412: The algorithm selector determines the machine learning algorithm to be trained based on the characteristics of the current training data set;
[0085] S413: Selecting a machine learning model from the central cloud machine learning pool according to the determined machine learning algorithm to be trained;
[0086] S414: Training machine learning models;
[0087] S415: Determine whether the trained machine learning model can be updated to the central cloud machine learning pool based on the adjustment strategy. If the update conditions are met, execute the next step, otherwise do not execute the operation;
[0088] S416: Update the central cloud machine learning algorithm pool;
[0089] S417: Update training threshold;
[0090] S418: Update the machine learning algorithm of the edge node.
[0091] The application scenarios of the edge data processing method of a machine learning pool and a dynamic update strategy described in this embodiment are as follows: Figure 5 As shown. The deployment and implementation of the edge data processing method is based on the edge computing architecture. The edge node is connected to the user terminal and the central cloud at the same time. The user terminal includes various application scenarios to meet user requests in various scenarios. The edge node is used to execute the service request of the user terminal, and the central cloud is used to store data and train machine learning algorithms. The edge data processing method can be applied in smart factories, smart transportation, smart medical care, smart homes and other fields, and has strong versatility.
[0092] In summary, the method of this embodiment has the following beneficial effects compared with the prior art:
[0093] (1) This embodiment provides a machine learning pool and an edge data processing method with a dynamic update strategy. The method is used to dynamically update edge node machine learning algorithm parameters under an edge computing framework, improve the accuracy of edge node data processing, reduce the error rate of edge node data processing, and enhance the robustness of the machine learning algorithm.
[0094] (2) The method of this embodiment can meet different data requests of users and dynamically select the best machine learning algorithm for processing data according to the data characteristics of the request, which not only further improves the accuracy of edge node data processing, but also improves the versatility of edge node data processing.
[0095] This embodiment also provides an edge data processing system, including:
[0096] A data uploading module is used to, after the service request sent by the user terminal is processed, the edge node saves the data of the service request according to the data type to the edge storage of the edge node, and when the storage capacity of the same type of data in the edge storage reaches a preset capacity storage threshold, transmit the type data set of the same type of data to the central cloud;
[0097] A model training module is used for receiving and storing the type data set transmitted by the edge node in the central cloud, determining the machine learning algorithm to be trained when the type data set reaches a preset training threshold, and using the type data set to train the machine learning algorithm;
[0098] The algorithm update module is used to compare the performance of the algorithm before and after training, and synchronously update the machine learning algorithm with better performance to the machine learning algorithm pool of the central cloud and the edge node.
[0099] An edge data processing system of this embodiment can execute an edge data processing method provided by the method embodiment of the present invention, can execute any combination of implementation steps of the method embodiment, and has the corresponding functions and beneficial effects of the method.
[0100] This embodiment also provides an edge data processing device, including:
[0101] at least one processor;
[0102] at least one memory for storing at least one program;
[0103] When the at least one program is executed by the at least one processor, the at least one processor implements the following Figure 4 The method shown.
[0104] An edge data processing device of this embodiment can execute an edge data processing method provided by the method embodiment of the present invention, can execute any combination of implementation steps of the method embodiment, and has the corresponding functions and beneficial effects of the method.
[0105] The present application also discloses a computer program product or a computer program, which includes a computer instruction stored in a computer-readable storage medium. A processor of a computer device can read the computer instruction from the computer-readable storage medium, and the processor executes the computer instruction, so that the computer device executes Figure 4 The method shown.
[0106] This embodiment also provides a storage medium, which stores instructions or programs that can execute an edge data processing method provided by an embodiment of the method of the present invention. When the instructions or programs are run, any combination of implementation steps of the method embodiment can be executed, and the corresponding functions and beneficial effects of the method can be obtained.
[0107] In some selectable embodiments, the function / operation mentioned in the block diagram may not occur in the order mentioned in the operation diagram. For example, depending on the function / operation involved, the two boxes shown in succession can actually be executed substantially simultaneously or the boxes can sometimes be executed in reverse order. In addition, the embodiment presented and described in the flow chart of the present invention is provided by way of example, for the purpose of providing a more comprehensive understanding of technology. The disclosed method is not limited to the operation and logic flow presented herein. Selectable embodiments are expected, wherein the order of various operations is changed and the sub-operation of a part for which is described as a larger operation is performed independently.
[0108] In addition, although the present invention is described in the context of functional modules, it should be understood that, unless otherwise specified, one or more of the functions and / or features described may be integrated into a single physical device and / or software module, or one or more functions and / or features may be implemented in separate physical devices or software modules. It is also understood that a detailed discussion of the actual implementation of each module is unnecessary for understanding the present invention. More specifically, in view of the properties, functions, and internal relationships of the various functional modules in the device disclosed herein, the actual implementation of the module will be understood within the conventional skills of the engineer. Therefore, those skilled in the art can implement the present invention set forth in the claims without excessive experimentation using ordinary techniques. It is also understood that the specific concepts disclosed are merely illustrative and are not intended to limit the scope of the present invention, which is determined by the full scope of the appended claims and their equivalents.
[0109] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc., which can store program codes.
[0110] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in conjunction with such instruction execution systems, devices or apparatuses. For the purposes of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in conjunction with such instruction execution systems, devices or apparatuses.
[0111] More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or more wires (electronic device), a portable computer disk case (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be a paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, deciphering or, if necessary, processing in another suitable manner, and then stored in a computer memory.
[0112] It should be understood that the various parts of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, a plurality of steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.
[0113] In the above description of this specification, the description with reference to the terms "one embodiment / example", "another embodiment / example" or "certain embodiments / examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.
[0114] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the claims and their equivalents.
[0115] The above is a specific description of the preferred implementation of the present invention, but the present invention is not limited to the above embodiments. Those skilled in the art may make various equivalent modifications or substitutions without violating the spirit of the present invention. These equivalent modifications or substitutions are all included in the scope defined by the claims of this application.
Claims
1. An edge data processing method, characterized in that: The following steps are involved: After processing the service request sent by the user terminal, the edge node saves the data of the service request to the edge storage of the edge node according to the data type. When the storage capacity of the same type of data in the edge storage reaches the preset capacity storage threshold, the type data set of the same type of data is transmitted to the central cloud; The central cloud receives and stores the type data set transmitted by the edge node, determines the machine learning algorithm that needs to be trained when the type data set reaches a preset training threshold, and trains the machine learning algorithm using the type data set; Compare the performance of the algorithms before and after training, and synchronously update the machine learning algorithms with better performance to the machine learning algorithm pools of the central cloud and the edge nodes; The edge data processing method further comprises the following steps: The user terminal sends a service request to the edge node; The edge node determines, according to the data type of the service request, an optimal algorithm for processing the data of the type; According to the determined optimal algorithm, obtaining, from the machine learning algorithm pool of the edge node, the user data of the service request processed by the corresponding machine learning algorithm; After the data processing is completed, the edge node returns the processing result to the user terminal to complete the service request of the user terminal; The determining of the machine learning algorithm that needs to be trained includes: Determine the machine learning algorithm that needs to be trained through the algorithm selector of the central cloud; The algorithm selector includes a data type-machine learning algorithm mapping table, wherein the mapping table includes multiple data types and multiple machine learning algorithms, the data type includes at least one of an image data type, an audio data type, a video data type, or a document data type, and the machine learning algorithm includes at least one of a principal component analysis, a support vector machine, a conditional random field, or a neural network model; The determining of the best algorithm for processing the type of data comprises: Determining, by an edge node algorithm selector, an optimal algorithm for processing the type of data; The algorithm selector includes a data type-machine learning algorithm mapping table, wherein the mapping table includes multiple data types and multiple machine learning algorithms, the data type includes at least one of an image data type, an audio data type, a video data type, or a document data type, and the machine learning algorithm includes at least one of a principal component analysis, a support vector machine, a conditional random field, or a neural network model; The edge node includes an edge storage, a machine learning algorithm pool, and an edge node algorithm selector; The machine learning algorithm pool includes a plurality of machine learning algorithms, and the machine learning algorithm includes at least one of principal component analysis, support vector machine, conditional random field, and neural network model; The edge storage is used to store the service request transmitted by the user terminal; The machine learning algorithm pool is used to store the machine learning algorithms required to process the user data of the service request; The edge node algorithm selector is used to select a corresponding machine learning algorithm from the machine learning algorithm pool of the edge node according to the characteristics of the service request transmitted by the user terminal.
2. The edge data processing method according to claim 1, characterized in that: The comparing algorithm performance before and after training, and synchronously updating the machine learning algorithm with better performance to the machine learning algorithm pool of the central cloud and the edge node, includes: Compare the performance evaluation results before and after algorithm training, including the accuracy of data processing, the security of the model, and the robustness of the model; Obtain the machine learning algorithm with better comprehensive evaluation and update it synchronously to the machine learning algorithm pool of the central cloud and the edge node.
3. The edge data processing method according to claim 1, characterized in that: The central cloud includes a machine learning algorithm pool, a central cloud algorithm selector, and a dynamic update strategy; The machine learning algorithm pool includes a plurality of machine learning algorithms, and the machine learning algorithm includes at least one of principal component analysis, support vector machine, conditional random field, and neural network model; The machine learning algorithm pool is used to store the trained machine learning models in the central cloud; The central cloud algorithm selector is used to select a corresponding machine learning algorithm from the machine learning algorithm pool according to the characteristics of the type of data set that reaches the training threshold, for model training; The dynamic update strategy is used to dynamically update the machine learning models in the machine learning algorithm pools of the central cloud and the edge nodes.
4. An edge data processing system, applied to an edge data processing method according to any one of claims 1 to 3, characterized in that: include: A data uploading module is used to, after the service request sent by the user terminal is processed, the edge node saves the data of the service request according to the data type to the edge storage of the edge node, and when the storage capacity of the same type of data in the edge storage reaches a preset capacity storage threshold, transmit the type data set of the same type of data to the central cloud; A model training module is used for receiving and storing the type data set transmitted by the edge node in the central cloud, determining the machine learning algorithm to be trained when the type data set reaches a preset training threshold, and using the type data set to train the machine learning algorithm; The algorithm update module is used to compare the performance of the algorithm before and after training, and synchronously update the machine learning algorithm with better performance to the machine learning algorithm pool of the central cloud and the edge node.
5. An edge data processing device, characterized in that: include: at least one processor; at least one memory for storing at least one program; When the at least one program is executed by the at least one processor, the at least one processor implements the method according to any one of claims 1 to 3.
6. A computer-readable storage medium storing a program executable by a processor, characterized in that: The program executable by the processor is used to implement the method according to any one of claims 1 to 3 when executed by the processor.
