Data processing method and device

Abstract

The embodiments of this specification provide a data processing method and device, which relate to the field of computer technology. The method is applied to edge nodes in an end-edge cloud architecture, and the end-edge cloud architecture includes interconnected end nodes, edge nodes, and cloud nodes. The method includes: constructing a feature data set, the feature data set includes at least one initial feature data, the initial feature data is collected by the end node and uploaded to the edge node; inputting the feature data set into a deep learning model to generate at least one intermediate feature data; returning at least one intermediate feature data to the end node and / or cloud node, so that the end node and / or cloud node performs model reasoning or model training based on at least one intermediate feature data. The solution provided in this specification controls the number of initial feature data simultaneously input into the deep learning model, thereby enabling the data processing link on the edge node to be compatible with both the streaming computing strategy and the batch computing strategy.

Description

Technical Field

[0001] One or more embodiments of the present specification relate to the field of computer technology, and in particular, to a data processing method and device. Background Art

[0002] With the rise of edge computing, edge computing and cloud computing can be combined to form a distributed computing architecture (end-edge-cloud architecture) that collaborates with terminal devices, edge computing, and cloud computing. In the end-edge-cloud architecture, various deep learning models can be deployed on edge nodes, and services can be provided to end nodes and cloud nodes.

[0003] In related technologies, deep learning models deployed on edge nodes usually complete model reasoning and data reflow based on streaming computing strategies. However, in application scenarios where data real-time requirements are not high, streaming computing strategies will occupy more system resources, resulting in resource waste. Therefore, how to make the current streaming computing links on edge nodes compatible with batch computing strategies has become an urgent problem to be solved. Summary of the invention

[0004] In view of this, one or more embodiments of the present specification provide a data processing method and device.

[0005] To achieve the above objectives, one or more embodiments of this specification provide the following technical solutions:

[0006] According to a first aspect of one or more embodiments of this specification, a data processing method is proposed, which is applied to an edge node in an edge-edge-cloud architecture, where the edge-edge-cloud architecture includes an end node, an edge node, and a cloud node connected to each other, and the method includes:

[0007] Constructing a feature data set, the feature data set includes at least one initial feature data, the initial feature data is collected by the end node and uploaded to the edge node;

[0008] Inputting the feature data set into the deep learning model to generate at least one intermediate feature data;

[0009] At least one intermediate feature data is returned to the end node and / or the cloud node, so that the end node and / or the cloud node performs model inference or model training according to the at least one intermediate feature data.

[0010] According to a second aspect of one or more embodiments of this specification, a data processing device is provided, which is applied to an edge node in an edge-edge-cloud architecture. The edge-edge-cloud architecture includes an end node, an edge node, and a cloud node connected to each other. The device includes:

[0011] A construction module is used to construct a feature data set, where the feature data set includes at least one initial feature data, which is collected by the end node and uploaded to the edge node;

[0012] A generation module, used to input the feature data set into the deep learning model to generate at least one intermediate feature data;

[0013] A reflow module is used to reflow at least one intermediate feature data to an end node and / or a cloud node, so that the end node and / or the cloud node performs model inference or model training based on the at least one intermediate feature data.

[0014] According to a third aspect of one or more embodiments of this specification, an electronic device is provided, including:

[0015] processor;

[0016] a memory for storing processor-executable instructions;

[0017] The processor implements the method of the first aspect by running executable instructions.

[0018] According to a fourth aspect of one or more embodiments of the present specification, a computer-readable storage medium is provided, on which computer instructions are stored, and when the instructions are executed by a processor, the steps of the method of the first aspect are implemented.

[0019] The data processing method provided in this specification can be applied to edge nodes in an end-edge cloud architecture. Among them, the end-edge cloud architecture includes interconnected end nodes, edge nodes and cloud nodes. Specifically, the method includes: constructing a feature data set, the feature data set includes at least one initial feature data, the initial feature data is collected by the end node and uploaded to the edge node; inputting the feature data set into a deep learning model to generate at least one intermediate feature data; returning at least one intermediate feature data to the end node and / or cloud node, so that the end node and / or cloud node performs model reasoning or model training based on at least one intermediate feature data. The solution provided in this specification controls the number of initial feature data simultaneously input into the deep learning model by constructing the initial feature data uploaded by the end node into a feature data set with unfixed capacity, thereby enabling the data processing link on the edge node to be compatible with both streaming computing strategies and batch computing strategies. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 It is a schematic diagram of a system architecture provided by an exemplary embodiment.

[0021] Figure 2 It is a flowchart of a data processing method provided by an exemplary embodiment.

[0022] Figure 3 It is a schematic diagram of the interaction process among cloud nodes, edge nodes and end nodes provided by an exemplary embodiment.

[0023] Figure 4 It is a structural schematic diagram of a device provided by an exemplary embodiment.

[0024] Figure 5 It is a structural schematic diagram of a data processing device provided by an exemplary embodiment. DETAILED DESCRIPTION

[0025] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with one or more embodiments of this specification. Instead, they are merely examples of devices and methods consistent with some aspects of one or more embodiments of this specification as detailed in the appended claims.

[0026] It should be noted that: in other embodiments, the steps of the corresponding method are not necessarily performed in the order shown and described in this specification. In some other embodiments, the steps included in the method may be more or less than those described in this specification. In addition, a single step described in this specification may be decomposed into multiple steps for description in other embodiments; and multiple steps described in this specification may be combined into a single step for description in other embodiments.

[0027] To facilitate understanding, the terms that appear in this specification are first explained.

[0028] Stream computing refers to the way of processing real-time data streams one by one. In stream computing, there is no need to wait for all data to arrive before starting the calculation. Data enters the system in a continuous and persistent manner, is divided into several small blocks, and then processed one by one. Stream computing is suitable for scenarios that require real-time response and fast data processing, such as real-time monitoring, advertising, network security, etc.

[0029] Batch computing is a way of processing a set of data in batches. In batch computing, data is divided into batches and processed as a whole batch. Batch computing is usually performed on large-scale, offline data sets and can efficiently perform complex computing tasks. Batch computing is suitable for scenarios with relatively low real-time requirements, such as data analysis and machine learning training.

[0030] Edge computing is a computing and communication method that deploys software processing logic on a network architecture that is closer to user devices and data generators. Compared with traditional cloud computing or central computing models, edge computing can significantly reduce central bandwidth, save costs and reduce latency, bringing a better user experience.

[0031] With the rise of edge computing, edge computing and cloud computing can be combined to form a distributed computing architecture (end-edge-cloud architecture) that collaborates with terminal devices, edge computing, and cloud computing. In the end-edge-cloud architecture, various deep learning models can be deployed on edge nodes, and services can be provided to end nodes and cloud nodes.

[0032] In related technologies, deep learning models deployed on edge nodes usually complete model reasoning and data reflow based on streaming computing strategies. However, in application scenarios where data real-time requirements are not high, streaming computing strategies will continue to occupy more system resources, resulting in resource waste. In contrast, although batch computing strategies can complete the processing of large amounts of data in a short time, they cannot meet the requirements of some applications for real-time transmission.

[0033] Therefore, in order to enable edge nodes to achieve better resource allocation, edge nodes can select different computing strategies according to specific application scenarios. Therefore, how to make the current stream computing link on the edge node compatible with the batch computing strategy has become an urgent problem to be solved.

[0034] In view of this, the solution provided in this specification constructs a feature data set, which includes at least one initial feature data, which is collected by the end node and uploaded to the edge node; the feature data set is input into the deep learning model to generate at least one intermediate feature data; at least one intermediate feature data is returned to the end node and / or the cloud node, so that the end node and / or the cloud node performs model reasoning or model training based on the at least one intermediate feature data. The solution provided in this specification constructs the initial feature data uploaded by the end node into a feature data set with unfixed capacity, thereby controlling the number of initial feature data simultaneously input into the deep learning model, thereby enabling the data processing link on the edge node to be compatible with both the streaming computing strategy and the batch computing strategy.

[0035] For example, Figure 1 A schematic diagram of an exemplary system architecture of a data processing method or a data processing device that can be applied to the embodiments of this specification is shown.

[0036] like Figure 1 As shown, the solution provided in the embodiment of this specification can be applied to the edge-cloud architecture 100. The edge-cloud architecture is a distributed computing architecture that optimizes network performance, reduces latency, and enhances data privacy by combining terminal devices, edge computing, and cloud computing. The core idea of ​​this architecture is to process and store data at different levels, so that computing power can be more flexibly deployed on devices at all levels.

[0037] Please continue to refer to Figure 1The edge-cloud architecture 100 may include a cloud node 101, an edge node 102, and an end node 103, wherein the cloud node 101 and the edge node 102 may communicate with each other through a network, and the edge node 102 and the end node 103 may communicate with each other through a network.

[0038] Specifically, the cloud node 101 is a central node of traditional cloud computing, which has large-scale computing and storage resources. The cloud node 101 is responsible for handling large-scale data processing, analysis and storage tasks, and provides support for the edge node 102 and the end node 103.

[0039] Edge nodes 102 refer to nodes located between end nodes 103 and cloud nodes 101, and are usually servers, gateways, or other network devices distributed in various regions. Edge nodes 102 can place computing and storage resources closer to end nodes 103 to reduce data transmission delays and provide real-time data processing and decision-making capabilities.

[0040] End nodes 103 refer to terminal devices, such as sensors, IoT devices, mobile devices, etc. These devices usually have limited computing and storage capabilities, but through the edge-end cloud architecture, they can communicate with cloud nodes and edges and collaborate with remote computing resources to complete complex tasks.

[0041] In some embodiments, a deep learning model is deployed on the edge node 102, and the deep learning model can receive the initial feature data uploaded by the end node 103, and output intermediate feature data that can be called by the cloud node 101 and / or the end node 103. Depending on the application scenario, the deep learning model can process the initial feature data based on a streaming computing strategy or a batch computing strategy, thereby meeting the requirements for data real-time performance or system performance optimization in the corresponding application scenario.

[0042] Those skilled in the art will know that Figure 1 The number of cloud nodes, edge nodes, and end nodes shown in the figure is merely illustrative. Any number of cloud nodes, edge nodes, and end nodes may be provided according to actual needs, and this specification does not impose any limitation on this.

[0043] This exemplary implementation will be described in detail below with reference to the accompanying drawings and embodiments.

[0044] First, an embodiment of this specification provides a data processing method, which can be executed by any electronic device.

[0045] Figure 2 The present invention is a flowchart of a data processing method provided by an exemplary embodiment, which is applied to edge nodes in an edge-to-edge cloud architecture, wherein the edge-to-edge cloud architecture includes interconnected end nodes, edge nodes, and cloud nodes. Figure 2 As shown, the data processing method includes the following steps.

[0046] S201, the edge node constructs a feature data set, the feature data set includes at least one initial feature data, the initial feature data is collected by the end node and uploaded to the edge node.

[0047] In some embodiments, the initial feature data may be obtained by the end node collecting the user's behavior performed on the terminal device, wherein the user's behavior may be the user's specific operation on the terminal device, the frequency of the operation, the duration of the operation, and the triggered object.

[0048] For example, based on deep learning technology, more products that users may be interested in can be recommended to users on the payment page after they purchase products. At this time, the initial feature data can be a series of operations performed by the user before purchasing the product, the category of the product purchased, etc.

[0049] It is understandable that based on the in-depth analysis of user behavior, the implicit features in the user behavior can also be obtained. For example, based on the commodity categories that the user frequently browses and the browsing time, the commodity categories that the user is interested in can be determined. For example, when the user's browsing frequency and browsing time are both greater than a preset threshold, it is considered that the user is interested in the commodity category.

[0050] In some embodiments, the initial feature data may be the original data collected by the end node, or may be a feature vector generated after being processed by a preset script on the end node, which is not limited in the embodiments of this specification.

[0051] It should be noted that the feature data set can be understood as a collection of initial feature data. Corresponding to the requirements of different application scenarios, the initial feature data in the feature data set can be one or more.

[0052] Exemplarily, the feature data set may be constructed in the following manner: determining a computing strategy; and constructing the feature data set according to the computing strategy, wherein the computing strategy is a stream computing strategy or a batch computing strategy.

[0053] In some embodiments, the computing strategy can be determined based on the application identifier carried by the initial feature data. Specifically, the application identifier is generated by the end node when collecting the initial feature data, and can be used to reflect the application scenario corresponding to the initial feature data, so that the edge node can determine the computing strategy for the initial feature data according to the application scenario. Exemplarily, a mapping relationship between the application identifier and the computing strategy can be pre-established on the edge node, so that the edge node matches the corresponding computing strategy according to the mapping relationship.

[0054] In some embodiments, when the computing strategy is a streaming computing strategy, the feature data set includes one initial feature data, that is, each feature data set can only include one initial feature data for streaming computing, thereby ensuring the real-time performance of streaming computing.

[0055] In other embodiments, when the computing strategy is a batch computing strategy, constructing a feature data set according to the computing strategy includes: aggregating the initial feature data uploaded by the end node based on the time window, and the feature data set includes at least one initial feature data uploaded by the end node within the time window. That is, each feature data set can contain multiple initial feature data for batch computing, so that multiple initial feature data can be processed in parallel to avoid long-term occupation of system resources.

[0056] S202: The edge node inputs the feature data set into the deep learning model to generate at least one intermediate feature data.

[0057] In some embodiments, before the edge node inputs the feature data set into the deep learning model, it can perform preprocessing operations such as data cleaning on the initial feature data in the feature data set, so as to avoid useless data from being input into the deep learning model. After the initial feature data in the feature data set is preprocessed, it can be converted into a format that can be recognized by the deep learning model through a format conversion operation.

[0058] It should be noted that after the edge node inputs the constructed feature data set into the deep learning model, the deep learning model can dynamically adjust the number of model input parameters based on the dynamic batch capability.

[0059] When a feature data set containing only one initial feature data is received, the initial feature data is directly input into the deep learning model for inference, and the inference result is obtained in real time, thereby ensuring the real-time transmission of the inference result. When a feature data set containing multiple initial feature data is received, the multiple initial feature data are used as input parameters of the deep learning model, and inference is performed in parallel to obtain the corresponding inference results, thereby reducing the time occupied by system resources.

[0060] Exemplarily, the deep learning model can be deployed on an inference server based on the Trition architecture, which can be equipped with inference engines such as Onnx and Pytorch, but this specification does not limit this.

[0061] It should be noted that the intermediate feature data generated by the deep learning model on the edge node can be data used to assist other models in reasoning or training. Exemplarily, the deep learning model can be used to generate embedding data or perform tasks such as extracting entity intent. For example, the deep learning model can extract the user's intent from the collected initial feature data (for example, the user's behavior chain), and return the extracted intent to the cloud node and / or the end node as intermediate feature data, so that the cloud node and the end node can perform reasoning and training of models such as media content recommendation and product recommendation based on the extracted intent.

[0062] S203, the edge node flows back at least one intermediate feature data to the end node and / or the cloud node, so that the end node and / or the cloud node performs model inference or model training according to the at least one intermediate feature data.

[0063] In some embodiments, corresponding to the determination of the aforementioned computing strategy, in the process of reflowing the intermediate feature data, a reflow strategy of the intermediate feature data may be first determined, and then at least one intermediate feature data may be reflowed to the end node and / or the cloud node according to the reflow strategy. The reflow strategy includes at least one of a streaming transmission strategy and a batch transmission strategy.

[0064] It is worth noting that, unlike the aforementioned computing strategies, the reflow strategy can be a streaming transmission strategy, a batch transmission strategy, or a combination of the streaming transmission strategy and the batch transmission strategy.

[0065] Exemplarily, the edge node can determine the return strategy of the intermediate feature data in combination with the application scenarios on the end node and the cloud node. For example, if the end node needs to call the intermediate feature data for real-time model reasoning, and the cloud node only needs to store the intermediate feature data for subsequent model training, at this time, since the end node has a high real-time requirement for the intermediate feature data, and the cloud node has no real-time requirement for the intermediate feature data, the intermediate feature data can be returned to the end node through the streaming transmission strategy, and returned to the cloud node through the batch transmission strategy.

[0066] In some embodiments, when the reflow strategy is a streaming reflow strategy, each intermediate feature data generated by the deep learning model will be reflowed to the end node and / or cloud node in real time.

[0067] In other embodiments, when the reflow strategy is a batch reflow strategy, a batch reflow cache may be constructed, and then the intermediate feature data gathered in the batch reflow cache may be reflowed to the end node and / or cloud node at a specified time interval (e.g., every 5 seconds). The batch reflow cache is used to gather the intermediate feature data generated by the deep learning model.

[0068] In some embodiments, after the intermediate feature data flows back to the cloud node, the cloud node can store the intermediate feature data in a data table through a designated gateway for inference or training by a deep learning model deployed on the cloud node.

[0069] In other embodiments, after the intermediate feature data flows back to the end node, the end node may store the intermediate feature data for inference or training of a deep learning model deployed on the end node.

[0070] In some embodiments, in order to further improve the data transmission efficiency between the edge node and the cloud node and the end node, the end node may compress the initial feature data before uploading the initial feature data to the edge node, and the edge node may compress the intermediate feature data before returning the intermediate feature data to the cloud node and / or the end node.

[0071] The solution provided in the embodiments of this specification can provide a unified data link for the streaming computing strategy and batch computing strategy on the edge node, so that the edge node can be compatible with both the streaming computing / transmission strategy and the batch computing / transmission strategy during the process of model reasoning and data reflow, and the edge node can simultaneously reflow the intermediate feature data back to the cloud node and the end node for use by the cloud node and the end node.

[0072] Furthermore, the embodiments of the present specification construct feature data sets of different sizes and, based on the dynamic batch processing capabilities of the deep learning model on the edge node, achieve real-time processing of initial feature data with high real-time requirements to ensure its real-time performance, and batch processing of initial feature data with low real-time requirements, thereby reducing the long-term occupation of system resources.

[0073] In addition, the solution provided by the embodiments of this specification can also reduce the cost of data transmission between end nodes, edge nodes and cloud nodes. On the one hand, since the deep learning model on the edge node is a model for extracting intermediate feature data, its reasoning results (for example, embedding, entity, intent, etc.) will be much smaller than the initial input parameters of the deep learning model. In the process of data reflow, only the reflow of intermediate feature data can meet the needs of cloud nodes and end nodes, thereby effectively reducing the volume of data for transmission. On the other hand, by compressing the data before each transmission, the volume of data for transmission can be further reduced.

[0074] For ease of understanding, the interaction process between the cloud nodes, edge nodes and end nodes in the embodiments of this specification will be specifically described below in combination with actual scenarios.

[0075] Figure 3 It is a schematic diagram of the interaction process among cloud nodes, edge nodes and end nodes provided by an exemplary embodiment.

[0076] like Figure 3 As shown, the end node can collect the click / browse behaviors performed by the user on the terminal, and perform corresponding feature collection to obtain initial feature data. After the initial feature data is compressed, the end-side software development kit (SDK) can select a specified routing strategy according to the specific application scenario and upload the compressed initial feature data to the edge node (for example, the content distribution network gateway). The connection between the end node and the edge node can be maintained until the edge node returns the intermediate feature data.

[0077] After receiving the initial feature data, the edge node can determine the corresponding computing strategy (for example, streaming computing strategy or batch computing strategy) for the initial feature data and construct the corresponding feature data set. In the case where the computing strategy is a batch computing strategy, the initial feature data can be aggregated through a time window.

[0078] After completing the construction of the feature data set, the initial feature data in the feature data set can be preprocessed by data analysis, data filtering, etc., so as to eliminate useless data and reduce the amount of data. The feature data set is then converted into a format that can be recognized by the deep learning model on the edge node and sent to the deep learning model.

[0079] Based on the dynamic batch processing capability, the deep learning model performs streaming inference calculations on feature data sets containing a single initial feature data, and performs batch inference calculations on feature data sets containing multiple initial feature data, thereby obtaining intermediate feature data. The intermediate feature data obtained can be compressed to reduce the amount of data that needs to be transmitted when the data flows back.

[0080] The edge node can determine the return strategy of the intermediate feature data according to the specific application scenario, such as streaming transmission strategy and batch transmission strategy. When the return strategy is a streaming transmission strategy, the intermediate feature data will be returned to the cloud node and / or the end node in real time. When the return strategy is a batch transmission strategy, the intermediate feature data will be stored in the batch return cache, and the data in the batch return cache will be returned to the cloud node and / or the end node at the specified event interval.

[0081] After the intermediate feature data flows back to the cloud node, the cloud node can store the intermediate feature data in the data table through the specified gateway for inference or training of the deep learning model deployed on the cloud node. After the intermediate feature data flows back to the end node, the end node can store the intermediate feature data for inference or training of the deep learning model deployed on the end node. It is worth noting that since the edge node has no direct request capability to the end node, the communication link used when the edge node flows the intermediate feature data back to the end node is still the communication link established by the end node when uploading the initial feature data.

[0082] Figure 4 This is a schematic diagram of a device structure provided by an exemplary embodiment. Figure 4 At the hardware level, the device includes a processor 402, an internal bus 404, a network interface 406, a memory 408, and a non-volatile memory 410, and may also include hardware required for other application scenarios. One or more embodiments of this specification can be implemented based on software, such as the processor 402 reading the corresponding computer program from the non-volatile memory 410 into the memory 408 and then running it. Of course, in addition to the software implementation, one or more embodiments of this specification do not exclude other implementations, such as logic devices or a combination of software and hardware, etc., that is, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0083] Please refer to Figure 5 , Figure 5 A data processing device 500 is provided, which can be applied to Figure 4 The device shown is used to implement the technical solution of this specification.

[0084] Exemplarily, the data processing device 500 can be applied to an edge node in an edge-edge-cloud architecture, which includes interconnected end nodes, edge nodes, and cloud nodes.

[0085] Specifically, the data processing device 500 may include:

[0086] The construction module 501 is used to construct a feature data set, which includes at least one initial feature data, and the initial feature data is collected by the end node and uploaded to the edge node.

[0087] The generation module 502 is used to input the feature data set into the deep learning model to generate at least one intermediate feature data.

[0088] The reflow module 503 is used to reflow at least one intermediate feature data to the end node and / or the cloud node, so that the end node and / or the cloud node performs model inference or model training according to the at least one intermediate feature data.

[0089] In some embodiments, the construction module 501 is used to determine a computing strategy, where the computing strategy is a streaming computing strategy or a batch computing strategy; and to construct a feature data set according to the computing strategy.

[0090] In some embodiments, when the computing strategy is a streaming computing strategy, the feature data set includes an initial feature data.

[0091] In some embodiments, when the computing strategy is a batch computing strategy, the construction module 501 is specifically used to aggregate the initial feature data uploaded by the end nodes based on the time window, and the feature data set includes at least one initial feature data uploaded by the end nodes within the time window.

[0092] In some embodiments, the reflow module 503 is used to determine a reflow strategy, which includes at least one of a streaming transmission strategy and a batch transmission strategy; according to the reflow strategy, at least one intermediate feature data is reflowed to the end node and / or the cloud node.

[0093] In some embodiments, when the reflow strategy is a streaming reflow strategy, the reflow module 503 is specifically used to reflow each intermediate feature data generated by the deep learning model to the end node and / or cloud node in real time.

[0094] In some embodiments, when the reflow strategy is a batch reflow strategy, the reflow module 503 is specifically used to build a batch reflow cache, which is used to aggregate the intermediate feature data generated by the deep learning model; according to a specified time interval, the intermediate feature data aggregated in the batch reflow cache is reflowed to the end node and / or cloud node.

[0095] In some embodiments, the data processing device 500 further includes a compression module (not shown in the figure). The compression module is used to compress at least one intermediate feature data.

[0096] The systems, devices, modules or units described in the above embodiments may be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer, which may be in the form of a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email transceiver, a game console, a tablet computer, a wearable device or a combination of any of these devices.

[0097] In a typical configuration, a computer includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0098] Memory may include non-permanent storage in a computer-readable medium, in the form of random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of a computer-readable medium.

[0099] Computer-readable media include permanent and non-permanent, removable and non-removable media that can be used to store information by any method or technology. Information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, disk storage, quantum memory, graphene-based storage media or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include temporary computer-readable media (transitory media), such as modulated data signals and carrier waves.

[0100] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0101] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0102] The above is a description of a specific embodiment of the specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0103] The terms used in one or more embodiments of this specification are only for the purpose of describing specific embodiments, and are not intended to limit one or more embodiments of this specification. The singular forms of "a", "said" and "the" used in one or more embodiments of this specification and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used herein refers to and includes any or all possible combinations of one or more associated listed items.

[0104] It should be understood that although the terms first, second, third, etc. may be used to describe various information in one or more embodiments of this specification, these information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of one or more embodiments of this specification, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0105] The above description is merely a preferred embodiment of one or more embodiments of the present specification and is not intended to limit one or more embodiments of the present specification. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of one or more embodiments of the present specification shall be included in the scope of protection of one or more embodiments of the present specification.

Claims

1. A data processing method, applied to an edge node in an edge-to-edge cloud architecture, wherein the edge-to-edge cloud architecture includes an end node, an edge node, and a cloud node connected to each other, the method comprising: Constructing a feature data set includes: determining a computing strategy, wherein the computing strategy is a streaming computing strategy or a batch computing strategy; constructing the feature data set according to the computing strategy; the feature data set includes at least one initial feature data, wherein the initial feature data is collected by the end node and uploaded to the edge node; Inputting the feature data set into a deep learning model to generate at least one intermediate feature data, including: when the feature data set includes at least two initial feature data, using the multiple initial feature data in the feature data set as input parameters of the deep learning model, performing reasoning in parallel, and obtaining corresponding intermediate feature data respectively; the deep learning model has the ability to dynamically adjust the model input parameter quantity; Returning the at least one intermediate feature data to the end node and / or the cloud node, so that the end node and / or the cloud node performs model reasoning or model training according to the at least one intermediate feature data; The reflowing of the at least one intermediate feature data to the end node and / or the cloud node includes: determining a reflow strategy for the intermediate feature data based on an application scenario, the reflow strategy including at least one of a streaming transmission strategy and a batch transmission strategy; and reflowing the at least one intermediate feature data to the end node and / or the cloud node according to the reflow strategy.

2. According to the method of claim 1, when the computing strategy is a streaming computing strategy, the feature data set includes an initial feature data.

3. According to the method of claim 1, when the computing strategy is a batch computing strategy, constructing the feature data set according to the computing strategy comprises: The initial feature data uploaded by the end node is aggregated based on a time window, and the feature data set includes at least one initial feature data uploaded by the end node within the time window.

4. The method according to claim 1, wherein when the reflow strategy is a streaming reflow strategy, reflowing the at least one intermediate feature data to the end node and / or the cloud node according to the reflow strategy comprises: Each intermediate feature data generated by the deep learning model is reflowed to the end node and / or the cloud node in real time.

5. The method according to claim 1, wherein when the reflow strategy is a batch reflow strategy, reflowing the at least one intermediate feature data to the end node and / or the cloud node according to the reflow strategy comprises: Constructing a batch reflow cache, wherein the batch reflow cache is used to aggregate intermediate feature data generated by the deep learning model; According to a specified time interval, the intermediate feature data gathered in the batch reflow buffer is reflowed to the end node and / or the cloud node.

6. The method according to claim 1, after inputting the feature data set into a deep learning model to obtain at least one intermediate feature data, the method further comprises: The at least one intermediate feature data is compressed.

7. A data processing device, applied to an edge node in an edge-edge-cloud architecture, wherein the edge-edge-cloud architecture includes an end node, an edge node, and a cloud node connected to each other, and the device includes: A construction module is used to construct a feature data set, including: determining a computing strategy, the computing strategy is a streaming computing strategy or a batch computing strategy; constructing the feature data set according to the computing strategy; the feature data set includes at least one initial feature data, the initial feature data is collected by the end node and uploaded to the edge node; A generating module, configured to input the feature data set into a deep learning model to generate at least one intermediate feature data, including: when the feature data set includes at least two initial feature data, using the multiple initial feature data in the feature data set as input parameters of the deep learning model, performing reasoning in parallel, and obtaining corresponding intermediate feature data respectively; the deep learning model has the ability to dynamically adjust the amount of model input parameters; A reflow module is used to reflow the at least one intermediate feature data to the end node and / or the cloud node, so that the end node and / or the cloud node performs model inference or model training according to the at least one intermediate feature data; the reflowing the at least one intermediate feature data to the end node and / or the cloud node includes: determining a reflow strategy based on an application scenario, the reflow strategy including at least one of a streaming transmission strategy and a batch transmission strategy; according to the reflow strategy, reflowing the at least one intermediate feature data to the end node and / or the cloud node.

8. An electronic device comprising: processor; a memory for storing processor-executable instructions; The processor implements the method according to any one of claims 1 to 6 by running the executable instructions.

9. A computer-readable storage medium having computer instructions stored thereon, which, when executed by a processor, implement the steps of the method according to any one of claims 1 to 6.

Images