Data processing methods, devices, electronic equipment and media

Abstract

This application relates to the field of computing power network technology, and provides a data processing method, apparatus, electronic device, and medium. Based on the model information of a global perception model and the metadata information corresponding to each perception node, it determines the perception sub-model corresponding to each perception node; it distributes the corresponding perception sub-model and perception task to each perception node; and it aggregates the collected data uploaded by each perception node to obtain target data. The central master node of this application determines the perception sub-model corresponding to each perception node based on the global perception model and the metadata information corresponding to each perception node, and distributes it along with the perception task to each perception node. This allows for rapid cross-regional collection of required data through subordinate perception nodes, and the aggregation of the collected data uploaded by each perception node forms a complete dataset as target data, completing data perception of the entire computing power network and improving data perception efficiency.

Description

Technical Field

[0001] This application relates to the field of computing network technology, specifically to a data processing method, apparatus, electronic device, and medium. Background Technology

[0002] In the context of computing power networks, the computing power network operating system manages widely distributed and heterogeneous computing power and networks, including general-purpose computing power, intelligent computing, and supercomputing power, as well as access networks, bearer networks, core networks, and transmission networks. The types and quantities of resources involved in computing power networks are enormous, and they are characterized by cross-regional and cross-geographical characteristics. While the widespread adoption of virtualization, cloudification, and containerization has brought convenience, it has also made the system architecture more complex. This presents a significant challenge to unified perception and analysis under the computing power network operating system.

[0003] Traditional data sensing methods typically design independent data models and sensing access points for specific regions and scenarios, followed by centralized analysis and processing. This leads to conflicts due to the lack of standardized data models in cross-domain scenarios, and a lack of real-time sensing capabilities for diverse data in multi-layered, massive data scenarios. In distributed computing network architectures, computing resources are distributed across geographical areas, and the amount and types of data to be sensed are enormous, including computing power, network, applications, services, logs, etc. Using traditional data sensing methods for data sensing is time-consuming, resulting in low efficiency. Summary of the Invention

[0004] This application provides a data processing method, apparatus, electronic device, and medium to solve the problem that current data sensing based on traditional data sensing methods requires a lot of time, resulting in low efficiency in data sensing.

[0005] In a first aspect, embodiments of this application provide a data processing method, including:

[0006] Based on the model information of the global perception model and the metadata information corresponding to each perception node, the perception sub-model corresponding to each perception node is determined; wherein, the global perception model is determined based on the metadata information uploaded by each perception node.

[0007] The corresponding perception sub-model and perception task are sent to each of the perception nodes.

[0008] The collected data uploaded by each of the sensing nodes is aggregated to obtain target data; wherein, the aggregated data is obtained by the sensing nodes based on the corresponding sensing sub-model and the sensing task.

[0009] In one embodiment, determining the perception sub-model corresponding to each perception node based on the model information of the global perception model and the metadata information corresponding to each perception node includes:

[0010] For each sensing node, perform the following steps:

[0011] In the global perception model, delete the parts of the model information that are different from the metadata information of the current perception node to obtain the perception sub-model corresponding to the current perception node.

[0012] In one embodiment, before aggregating the collected data uploaded by each of the sensing nodes, the method further includes:

[0013] Receive the aggregated data uploaded by each of the aforementioned sensing nodes;

[0014] The aggregated data is obtained by the sensing node collecting data based on the sensing task and transforming the collected data based on the sensing sub-model corresponding to the sensing node.

[0015] In one embodiment, before determining the perception sub-model of the perception node based on the global perception model and the metadata information of the perception node, the method further includes:

[0016] The initial perception model is designed based on the metadata information uploaded by each perception node.

[0017] The initial perception model was validated, and the validation results were obtained.

[0018] If the verification result indicates that the initial perception model has passed the verification, then the initial perception model is determined as the global perception model.

[0019] In one embodiment, verifying the initial perception model to obtain a verification result includes:

[0020] Anomaly detection of model information is performed on the initial perception model to obtain a first detection result;

[0021] The initial perception model and the verification task are distributed to each of the perception nodes, and the second detection result fed back by each of the perception nodes based on the initial perception model and the verification task is received.

[0022] Based on the first detection result and the second detection result, the verification result of the initial perception model is determined.

[0023] In one embodiment, determining the verification result of the initial perception model based on the first detection result and the second detection result includes:

[0024] If both the first detection result and the second detection result are determined to be valid, then the verification result of the initial perception model is determined to be valid.

[0025] In one embodiment, the model information includes at least one of model identifier, model name, model description, and attribute relationships.

[0026] Secondly, embodiments of this application provide a data processing apparatus, including:

[0027] The determination module is used to determine the perception sub-model corresponding to each of the perception nodes based on the model information of the global perception model and the metadata information corresponding to each perception node; wherein, the global perception model is determined based on the metadata information uploaded by each of the perception nodes.

[0028] The distribution module is used to distribute the corresponding perception sub-model and perception task to each of the perception nodes.

[0029] The aggregation module is used to aggregate the collected data uploaded by each of the sensing nodes to obtain target data; wherein the aggregated data is obtained by the sensing nodes based on the corresponding sensing sub-model and the sensing task.

[0030] Thirdly, embodiments of this application provide an electronic device, including a processor and a memory storing a computer program, wherein the processor executes the program to implement the steps of the data processing method described in the first aspect.

[0031] Fourthly, embodiments of this application provide a storage medium, which is a computer-readable storage medium including a computer program. When the computer program is executed by a processor, it implements the steps of the data processing method described in the first aspect.

[0032] The data processing method, apparatus, electronic device, and storage medium provided in this application embodiment allow the central master node to determine the corresponding perception sub-model for each perception node based on the global perception model and the metadata information corresponding to each perception node, and distribute it to each perception node along with the perception task. Then, the required data can be quickly collected across regions through the subordinate perception nodes, and the collected data uploaded by each perception node after collection is aggregated to form a complete dataset as the target data, thereby completing the data perception of the entire computing power network and improving the data perception efficiency. Attached Figure Description

[0033] To more clearly illustrate the technical solutions in this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0034] Figure 1 This is one of the flowcharts illustrating the data processing method provided in the embodiments of this application;

[0035] Figure 2 This is a second schematic flowchart of the data processing method provided in the embodiments of this application;

[0036] Figure 3 This is the third flowchart illustrating the data processing method provided in the embodiments of this application;

[0037] Figure 4 This is the fourth flowchart illustrating the data processing method provided in the embodiments of this application;

[0038] Figure 5 This is a schematic diagram of the data collection process in the data processing method provided in the embodiments of this application;

[0039] Figure 6 This is a schematic diagram of the functional modules of an embodiment of the data processing device of this application;

[0040] Figure 7 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0041] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below with reference to the accompanying drawings of the embodiments. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0042] The data processing method, apparatus, electronic device, and storage medium provided by the present invention will be described in detail below with reference to embodiments.

[0043] Figure 1 This is one of the flowcharts illustrating the data processing method provided in an embodiment of this application. (Refer to...) Figure 1 This application provides a data processing method, which may include:

[0044] Step 100: Based on the model information of the global perception model and the metadata information corresponding to each perception node, determine the perception sub-model corresponding to each perception node.

[0045] It should be noted that the execution subject of the data processing method provided in this application embodiment can be a computer device, such as a server, mobile phone, tablet computer, laptop computer, PDA, in-vehicle electronic device, wearable device, ultra-mobile personal computer (UMPC), netbook, or personal digital assistant (PDA), etc. The execution subject may include a central master node, which may have one or more subordinate sensing nodes, and these sensing nodes may be distributed in different regions.

[0046] The central master node is responsible for the unified management of the global perception model and the data uploaded by each perception node, including global model management, unified data management, and perception task scheduling.

[0047] Among them, global model management mainly involves the unified design, verification, and distribution of perception models.

[0048] Unified data management primarily involves aggregating, analyzing, and sharing data collected from various sensing nodes, including metadata for the data models of each sensing center. After receiving data completion information from each sensing node, the central master node merges the data according to aggregation identifiers and rules. Analysis and sharing mainly involve processing the aggregated complete dataset and publishing and sharing the results.

[0049] The sensing task scheduling mainly involves distributing and monitoring sensing tasks to various sensing nodes. When a task is distributed, the central master node assigns a unique identifier to each task for monitoring and subsequent processing. Task monitoring primarily refers to real-time status tracking of the distributed tasks, including the reception, processing, and completion status of each sensing node, and notifying the unified data management module for further data processing after each sensing node completes a task.

[0050] The global perception model is determined based on the metadata information uploaded by each perception node, and the process of determining the global perception model may include design and verification.

[0051] The sensing nodes are used to collect and process data under the scheduling of the central master node.

[0052] The metadata information of the sensing node can include data domain, model identifier, model name, model description, attribute relationships, etc.; among which, attribute relationships include attribute identifier, Chinese name, English name, data type, data constraints, data source, responsible department, statistical scope, etc.

[0053] The data types can include integer, floating-point, boolean, character, date, etc., which are the main data types of the marked attributes.

[0054] Data constraints can include not null, unique, range of values, regular expressions, etc., which mainly restrict the data content of attributes.

[0055] The data domain is the category to which the attribute belongs, and generally includes business, resources, configuration, events, quality, operation and maintenance, and knowledge.

[0056] The perception sub-model is a sub-model that is split from the global perception model by deleting model information based on the metadata information of the perception nodes, which is the central master node.

[0057] Step 200: Distribute the corresponding perception sub-model and perception task to each perception node;

[0058] In this application, the sensing task can be a task issued by an upper-layer application; or it can be a task pre-set in the central master node and triggered when certain conditions are met. For example, a sensing task can be pre-set in the central master node and configured to execute once every preset time interval, where the preset time interval is a value set according to actual needs.

[0059] The perception task includes attribute parameters such as the collection period and the collection target.

[0060] For example, the perception task in this application includes the collection period and the collection target.

[0061] The collection period refers to the period during which data collection and processing are required.

[0062] Collection targets can include, but are not limited to, databases, files, interfaces, message queues, etc.

[0063] In this application, the central master node can formulate corresponding perception sub-models based on the global perception model and the metadata information of different perception nodes, and send the perception sub-models and perception tasks to the corresponding perception nodes.

[0064] For example, referring to the requirements of the global perception model, a perception sub-model 1 is formulated according to the requirements of perception node 1, and a perception sub-model 2 is formulated according to the requirements of perception node 2. The perception sub-model 1 and the perception task are then sent to perception node 1, and the perception sub-model 2 and the perception task are sent to perception node 2.

[0065] Step 300: Aggregate the collected data uploaded by each sensing node to obtain the target data.

[0066] In this application, the collected data is obtained by the sensing nodes collecting data according to the sensing tasks and processing the data according to the sensing sub-model corresponding to the sensing node.

[0067] After receiving the aggregated data uploaded by each sensing node, the central master node can aggregate the aggregated data according to the calculation rules in the global sensing model to form a complete dataset as the target data.

[0068] Specifically, the data aggregation method in this application can be either connection or merging.

[0069] In this context, joins primarily involve connecting different data segments using association conditions to form a wide table. For example:

[0070] D1 = {id, name, f} 11 f 12 , ..., f 1n},

[0071] D2 = {id, name, f} 21 f 22 , ..., f 2m},

[0072] …

[0073] D n ={id, name, f n1 f n2 , ..., f nj}

[0074] Through connection calculation, that is:

[0075] D = D1 + D2 + ... + D n ={id, name, f 11 ,f 12 ,…,f 1n ,…,f n1 f n2 , ...f nj}

[0076] Where D1, D2, ... D n Attribute fields with different dimensions, f nj This is a data segment in the nth row and jth column.

[0077] Connections are primarily used to perform correlation analysis on data of different types, generating a dataset with multidimensional attributes.

[0078] For example, to build a status dataset for each data center, data such as resources, alarms, performance, and logs can be obtained from each sensing center, and then the dataset containing multi-dimensional fields such as resources, alarms, performance, and logs can be generated by connecting them.

[0079] Merging primarily involves taking the union of different data segments to obtain a complete data table. For example:

[0080] D = D1∪D2∪…∪D n

[0081] Where D1, D2, ... D n Attribute fields containing the same dimension.

[0082] Merging is mainly used to combine the data from each part to form a complete dataset.

[0083] For example, virtual machine datasets for each data center can be constructed by merging virtual machine data from various sensing centers.

[0084] In this application, after obtaining the target data, the target data can also be published to achieve data sharing, so that upper-layer applications, other applications, and various sensing nodes can utilize it.

[0085] The data processing method provided in this application embodiment determines the corresponding perception sub-model for each perception node based on the global perception model and the metadata information corresponding to each perception node, and sends it to each perception node along with the perception task. Then, the required data can be quickly collected across regions through the subordinate perception nodes, and the collected data uploaded by each perception node is aggregated to form a complete dataset as the target data, thereby completing the data perception of the entire computing power network and improving the data perception efficiency.

[0086] This application adopts a distributed sensing method based on federated technology, using a collection and aggregation approach to avoid the problems of large-scale data migration and processing under centralized processing, thereby improving data sensing efficiency.

[0087] Figure 2 This is a second schematic flowchart illustrating the data processing method provided in an embodiment of this application. (Refer to...) Figure 2 In one embodiment, before determining the perception sub-model of a perception node based on the global perception model and the metadata information of the perception node, the method further includes:

[0088] Step 11: Design a perception model based on the metadata information uploaded by each perception node to obtain an initial perception model;

[0089] In this application, the central master node can design the perception model based on the metadata information uploaded by each perception node. The metadata information includes data domain, model identifier, model name, model description, attribute relationships, etc.; and the attribute relationships can include attribute identifier, Chinese name, English name, data type, data constraints, data source, responsible department, statistical scope, etc.

[0090] Therefore, based on the above metadata information, the model information of the perception model can be designed as model identifier, model name, model description, attribute relationships, etc. The attribute relationships can include attribute identifier, Chinese name, English name, data type, data constraints, data domain, mapping relationship, calculation rules, etc.

[0091] The mapping relationship is mainly used to map attribute names in the global perception model to attribute names of perception nodes. The format is data domain name.model name.attribute name. When the models of multiple perception nodes are not consistent, they are separated by vertical lines, i.e., perception node identifier.data domain name.model name.attribute name. When the model is distributed, it will be split for different perception nodes.

[0092] The calculation rules are the calculation rules for model attribute values. They can be calculated by transformation or by combining multiple attributes. Calculation expressions can be set, such as addition, subtraction, multiplication, and division algorithm formulas.

[0093] Specifically, the perception model design in this application can be based on relevant business needs, selecting model identifiers and attribute relationships that meet the requirements from the metadata information provided by each perception node, and designing a global model containing different data domain contents as the initial perception model through combination.

[0094] The design is mainly reflected in the design of attribute relationships, namely attribute identifiers, Chinese names, English names, data types, data constraints, data domains, mapping relationships, calculation rules, etc.

[0095] Step 12: Validate the initial perception model and obtain the validation results;

[0096] In this application, after obtaining the initial perception model, it is also necessary to verify the initial perception model.

[0097] The verification process can include internal verification based on the central master node and external verification based on the sensing nodes, and obtain the internal verification results and external verification results of the initial sensing model respectively.

[0098] The validation results of the initial perception model can be obtained based on the internal and external validation results.

[0099] The verification result is either the initial perception model passes the verification or the initial perception model fails the verification.

[0100] If the verification result shows that the initial perception model fails the verification, relevant personnel can be notified to investigate and adjust the model design.

[0101] Furthermore, the initial perception model was validated, and the validation results were obtained, including:

[0102] Step 121: Perform anomaly detection on the initial perception model to obtain the first detection result;

[0103] The verification of the initial sensing model in this application can include two aspects. First, internal verification: specifically, the central master node uses the metadata information provided by each sensing node to perform anomaly detection on the designed initial sensing model, determining whether all information in the initial sensing model conforms to the descriptions in the metadata information of the sensing nodes. If any information is found that does not conform to the descriptions in the metadata information, the user can be prompted to make modifications.

[0104] For example, it can detect whether the mapping relationships and calculation rules in the model information of the initial sensing node conform to the description of data type, data constraints, statistical caliber, etc. in the metadata information.

[0105] It should be noted that if any information in the model information of the initial perception model does not conform to the description in the metadata information of the perception node, such as the calculation rules cannot be implemented based on the known information, then the initial perception model is determined to have failed the verification.

[0106] If all information in the model information of the initial perception model matches the description in the metadata information of the perception node, then the initial perception model is determined to have passed the verification.

[0107] After anomaly detection is completed, the first detection result is obtained, indicating whether the initial perception model has passed or failed verification.

[0108] Step 122: Distribute the initial perception model and verification task to each perception node, and receive the second detection result fed back by each perception node based on the initial perception model and verification task;

[0109] Another aspect of the verification of the initial perception model in this application is external verification. Specifically, the central master node can send the initial perception model and a verification task for the initial perception model to each perception node through a distribution interface. The verification task instructs the perception nodes to run a small amount of data to verify the model logic and operability of the initial perception model.

[0110] Furthermore, it receives the second detection result, which indicates whether the initial perception model has passed or failed verification, after each perception node has verified the initial perception model through a verification task.

[0111] Step 123: Based on the first detection result and the second detection result, determine the verification result of the initial perception model.

[0112] After obtaining the first and second detection results, the verification results of whether the initial perception model as a whole has passed the verification can be further determined based on the first and second detection results.

[0113] Furthermore, based on the first and second detection results, the validation results of the initial perception model are determined, including:

[0114] Step 1231: If it is determined that both the first detection result and the second detection result are valid for the initial perception model, then the verification result of the initial perception model is determined to be valid.

[0115] If the first detection result indicates that the initial perception model has passed verification, and the second detection result also indicates that the initial perception model has passed verification, then since the initial perception model has passed both internal and external verification, the verification result of the initial perception model can be determined as having passed verification.

[0116] If it is determined that either the first or the second detection result indicates that the initial perception model has failed validation, then the initial perception model has failed validation, and relevant personnel can be prompted to investigate and make adjustments.

[0117] Step 13: If the verification result is that the initial perception model has passed the verification, then the initial perception model is determined as the global perception model.

[0118] If the verification result shows that the initial perception model passes the verification, then the initial perception model is determined as the global perception model.

[0119] This embodiment can construct a global perception model that meets the needs of perception nodes across regions by using the metadata information uploaded by each perception node. This allows for subsequent data collection by scheduling perception nodes across regions based on the global perception model, enabling faster processing of raw data and improving data perception efficiency.

[0120] Figure 3 This is the third flowchart illustrating the data processing method provided in this application embodiment. (Refer to...) Figure 3 In one embodiment, based on the model information of the global perception model and the metadata information corresponding to each perception node, the perception sub-model corresponding to each perception node is determined, including:

[0121] Step 101: For each sensing node, execute step 102 respectively:

[0122] Step 102: Delete the part of the model information in the global perception model that is different from the metadata information of the current perception node, and obtain the perception sub-model corresponding to the current perception node.

[0123] In this application, the central master node can split the global perception model for different perception nodes and distribute it to each perception node for data aggregation.

[0124] Specifically, when splitting the global perception model, for each perception node, the central master node can compare the model information of the global perception model with the metadata information of the perception node to determine the correspondence between the metadata information of the perception node and the model information in the global perception model.

[0125] Furthermore, based on the data domain in the metadata information, the parts of the global perception model that are different from the data domain can be deleted, and the parts of the global perception model that are different from the mapping relationship and calculation rules of the perception node can be deleted, thereby forming a perception sub-model suitable for the perception node.

[0126] This embodiment can split the global perception model into corresponding perception sub-models according to different perception nodes, so that each perception node can quickly collect data according to the perception task and integrate the data into the data required by the central node according to the perception sub-model, which can improve the data perception efficiency.

[0127] Figure 4 This is the fourth flowchart illustrating the data processing method provided in this application embodiment. (Refer to...) Figure 4 In one embodiment, before aggregating the collected data uploaded by each sensing node, the method further includes:

[0128] Step 21: Receive the aggregated data uploaded by each sensing node.

[0129] In this application, after receiving the sensing sub-model and sensing task sent by the central master node, each sensing node can collect data according to the collection cycle and collection target in the sensing task, combined with the data content it manages, to obtain the collected data.

[0130] After obtaining the collected data, the sensing nodes can use the corresponding calculation formulas to calculate the collected data according to the calculation rules in the sensing sub-model, and then transform the calculated data according to the requirements of the sensing sub-model to obtain the aggregated data.

[0131] Furthermore, the sensing nodes can upload the collected data to the corresponding central master node, so that the central master node can aggregate the data based on the collected data uploaded by each sensing node to obtain the target data.

[0132] Therefore, the central master node can receive the aggregated data uploaded by each sensing node.

[0133] In this embodiment, the sensing nodes can receive the aggregated data obtained by the data aggregation process performed by the central master node according to the scheduling of the sensing nodes. Since the sensing nodes are close to the data source, they can process the raw data faster, thus improving the speed of data sensing and thereby improving the efficiency of data sensing.

[0134] Figure 5 This is a schematic diagram of the data collection process in the data processing method provided in the embodiments of this application. Figure 5 In the process, the central master node can split the global perception model according to the requirements of the perception nodes at the beginning of the data collection process, and send the split perception sub-models and perception tasks to the perception nodes together.

[0135] The sensing nodes collect data according to the sensing tasks, perform calculations on the collected data based on the model information of the sensing sub-models, further transform the calculated data based on the model information of the sensing sub-models, and upload the transformed data to the central master node.

[0136] The central master node aggregates the collected data received from each sensing node and ends the data aggregation process.

[0137] Furthermore, this application also provides a data processing apparatus.

[0138] Reference Figure 6 , Figure 6 This is a schematic diagram of the functional modules of an embodiment of the data processing device of this application.

[0139] The data processing device includes:

[0140] The determining module 610 is used to determine the sensing sub-model corresponding to each of the sensing nodes based on the model information of the global perception model and the metadata information corresponding to each sensing node; wherein, the global perception model is determined based on the metadata information uploaded by each of the sensing nodes.

[0141] The distribution module 620 is used to distribute the corresponding perception sub-model and perception task to each of the perception nodes;

[0142] The aggregation module 630 is used to aggregate the collected data uploaded by each of the sensing nodes to obtain target data; wherein the aggregated data is obtained by the sensing nodes based on the corresponding sensing sub-model and the sensing task.

[0143] The data processing device provided in this application embodiment allows the central master node to determine the corresponding perception sub-model for each perception node based on the global perception model and the metadata information corresponding to each perception node, and to send it to each perception node along with the perception task. Then, the required data can be quickly collected across regions through the subordinate perception nodes, and the collected data uploaded by each perception node is aggregated to form a complete dataset as the target data, thereby completing the data perception of the entire computing power network and improving the data perception efficiency.

[0144] In one embodiment, the determining module 610 is used to:

[0145] For each sensing node, perform the following steps:

[0146] In the global perception model, delete the parts of the model information that are different from the metadata information of the current perception node to obtain the perception sub-model corresponding to the current perception node.

[0147] In one embodiment, the determining module 610 is further configured to:

[0148] The initial perception model is designed based on the metadata information uploaded by each perception node.

[0149] The initial perception model was validated, and the validation results were obtained.

[0150] If the verification result indicates that the initial perception model has passed the verification, then the initial perception model is determined as the global perception model.

[0151] In one embodiment, the determining module 610 includes a verification unit, the verification unit being used for:

[0152] Anomaly detection of model information is performed on the initial perception model to obtain a first detection result;

[0153] The initial perception model and the verification task are distributed to each of the perception nodes, and the second detection result fed back by each of the perception nodes based on the initial perception model and the verification task is received.

[0154] Based on the first detection result and the second detection result, the verification result of the initial perception model is determined.

[0155] In one embodiment, the verification unit includes a determining unit, the determining unit being configured to:

[0156] If both the first detection result and the second detection result are determined to be valid, then the verification result of the initial perception model is determined to be valid.

[0157] In one embodiment, the aggregation module 630 is used for:

[0158] Receive the aggregated data uploaded by each of the aforementioned sensing nodes;

[0159] The aggregated data is obtained by the sensing node collecting data based on the sensing task and transforming the collected data based on the sensing sub-model corresponding to the sensing node.

[0160] Figure 7 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 7 As shown, the electronic device may include a processor 710, a communication interface 720, a memory 730, and a communication bus 740, wherein the processor 710, the communication interface 720, and the memory 730 communicate with each other via the communication bus 740. The processor 710 can call a computer program stored in the memory 730 to execute steps of a data processing method, such as:

[0161] Based on the model information of the global perception model and the metadata information corresponding to each perception node, the perception sub-model corresponding to each perception node is determined; wherein, the global perception model is determined based on the metadata information uploaded by each perception node.

[0162] The corresponding perception sub-model and perception task are sent to each of the perception nodes.

[0163] The collected data uploaded by each of the sensing nodes is aggregated to obtain target data; wherein, the aggregated data is obtained by the sensing nodes based on the corresponding sensing sub-model and the sensing task.

[0164] Furthermore, the logical instructions in the aforementioned memory 730 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0165] On the other hand, embodiments of this application also provide a storage medium, which is a computer-readable storage medium storing a computer program. The computer program is used to cause a processor to execute the steps of the methods provided in the above embodiments, including, for example:

[0166] Based on the model information of the global perception model and the metadata information corresponding to each perception node, the perception sub-model corresponding to each perception node is determined; wherein, the global perception model is determined based on the metadata information uploaded by each perception node.

[0167] The corresponding perception sub-model and perception task are sent to each of the perception nodes.

[0168] The collected data uploaded by each of the sensing nodes is aggregated to obtain target data; wherein, the aggregated data is obtained by the sensing nodes based on the corresponding sensing sub-model and the sensing task.

[0169] The computer-readable storage medium can be any available medium or data storage device that the processor can access, including but not limited to magnetic storage (e.g., floppy disk, hard disk, magnetic tape, magneto-optical disk (MO)), optical storage (e.g., CD, DVD, BD, HVD), and semiconductor storage (e.g., ROM, EPROM, EEPROM, non-volatile memory (NAND FLASH), solid-state drive (SSD)).

[0170] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, i.e., they may be located in one location or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of the present embodiment. Persons of ordinary skill in the art will be able to understand and implement the present invention without inventive effort.

[0171] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0172] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A data processing method, characterized in that, include: Based on the model information of the global perception model and the metadata information corresponding to each perception node, the perception sub-model corresponding to each perception node is determined; wherein, the global perception model is determined based on the metadata information uploaded by each perception node. The corresponding perception sub-model and perception task are sent to each of the perception nodes. The collected data uploaded by each of the sensing nodes is aggregated to obtain target data; wherein, the collected data is obtained by the sensing nodes based on the corresponding sensing sub-model and the sensing task; The model information based on the global perception model and the metadata information corresponding to each perception node are used to determine the perception sub-model corresponding to each perception node, including: For each sensing node, perform the following steps: In the global perception model, delete the part of the model information that is different from the metadata information of the current perception node to obtain the perception sub-model corresponding to the current perception node. Before aggregating the collected data uploaded by each of the sensing nodes, the method further includes: Receive the aggregated data uploaded by each of the aforementioned sensing nodes; The collected data is obtained by the sensing node collecting data based on the sensing task, and calculating the collected data using the corresponding calculation formula based on the calculation rules in the sensing sub-model corresponding to the sensing node, and then transforming the calculated data according to the requirements of the sensing sub-model; the calculation rules are the calculation rules for model attribute values.

2. The data processing method according to claim 1, characterized in that, Before determining the perception sub-model of the perception node based on the metadata information of the global perception model and the perception node, the process further includes: The initial perception model is designed based on the metadata information uploaded by each perception node. The initial perception model was validated, and the validation results were obtained. If the verification result indicates that the initial perception model has passed the verification, then the initial perception model is determined as the global perception model.

3. The data processing method according to claim 2, characterized in that, The verification of the initial perception model to obtain the verification result includes: Anomaly detection of model information is performed on the initial perception model to obtain a first detection result; The initial perception model and the verification task are distributed to each of the perception nodes, and the second detection result fed back by each of the perception nodes based on the initial perception model and the verification task is received. Based on the first detection result and the second detection result, the verification result of the initial perception model is determined.

4. The data processing method according to claim 3, characterized in that, Determining the verification result of the initial perception model based on the first detection result and the second detection result includes: If both the first detection result and the second detection result are determined to be valid, then the verification result of the initial perception model is determined to be valid.

5. The data processing method according to claim 1, characterized in that, The model information includes at least one of the following: model identifier, model name, model description, and attribute relationships.

6. A data processing apparatus, characterized in that, include: The determination module is used to determine the perception sub-model corresponding to each of the perception nodes based on the model information of the global perception model and the metadata information corresponding to each perception node; wherein, the global perception model is determined based on the metadata information uploaded by each of the perception nodes. The distribution module is used to distribute the corresponding perception sub-model and perception task to each of the perception nodes. An aggregation module is used to aggregate the collected data uploaded by each of the sensing nodes to obtain target data; wherein, the aggregated data is obtained by the sensing nodes based on the corresponding sensing sub-model and the sensing task; The determining module is used to perform the following steps for each sensing node: delete the part of the model information in the global sensing model that is different from the metadata information of the current sensing node, and obtain the sensing sub-model corresponding to the current sensing node; The aggregation module is used to receive the aggregated data uploaded by each of the sensing nodes; wherein, the aggregated data is obtained by the sensing nodes collecting data based on the sensing task, and calculating the collected data using corresponding calculation formulas based on the calculation rules in the sensing sub-model corresponding to the sensing node, and then converting the calculated data according to the requirements of the sensing sub-model; the calculation rules are the calculation rules for model attribute values.

7. An electronic device comprising a processor and a memory storing a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the data processing method according to any one of claims 1 to 5.

8. A medium, said medium being a computer-readable storage medium, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the data processing method according to any one of claims 1 to 5.

Images