Data processing method and device, equipment and storage medium

Abstract

The application provides a data processing method and device, equipment and a storage medium, relates to the technical field of communication, and is used for improving the data processing efficiency in the application scene of a single data source and multiple data streams. The method is applied to a data gateway device in a business system, the business system further includes a data source device and an edge node device, the data gateway device is connected with the data source device and the edge node device, and the data processing method comprises the following steps: the data gateway device receives source data sent by the data source device. Further, the data gateway device determines the target number of working nodes for preprocessing the source data. Further, the data gateway device processes the source data based on the target number of working nodes, and sends the processed source data to the edge node device.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of communication, and in particular to a data processing method and device, equipment and a storage medium. BACKGROUND

[0002] In actual production scenarios, a system constructed by informationization and intelligentization means is often supported by massive data. Traditional central cloud unified data processing of data sources can ensure data security and prevent data leakage. Meanwhile, the storage and computing resources of the central cloud are more abundant, which facilitates unified management and control.

[0003] However, in some specific industries, such as the industrial control field, the real-time requirement of business applications for data is increasingly high. Data is often generated from a device data source and then displayed on a human-computer interaction terminal. The time delay therebetween is often in the order of milliseconds or even microseconds to ensure the effectiveness of device data monitoring or control. In such a scenario, traditional central cloud unified scheduling processing is often big data batch processing, and the time delay of transmitting the processed data to the corresponding terminal is difficult to ensure the real-time performance of the data. SUMMARY

[0004] The present application provides a data processing method, device, equipment and storage medium to improve the data processing efficiency in the application scenario of a single data source and multiple data streams.

[0005] To achieve the above-mentioned purpose, the present application adopts the following technical solutions:

[0006] In a first aspect, a data processing method, device, equipment and storage medium are provided, which are applied to a data gateway device in a business system. The business system further includes a data source device and an edge node device. The data gateway device is connected with the data source device and the edge node device. The data processing method includes: receiving, by the data gateway device, source data sent by the data source device. Further, the data gateway device determines a target number of working nodes for pre-processing the source data. Further, the data gateway device processes the source data based on the target number of working nodes and sends the processed source data to the edge node device.

[0007] In the data processing method provided by the present application, after receiving the source data sent by the data source device, the data gateway device determines the target number of working nodes for pre-processing the source data, creates the target number of working nodes, processes the source data, and then sends the processed source data to the corresponding edge node device. In this way, the source data can be immediately processed without waiting, which reduces the time delay of data processing. Moreover, the edge node participates in the subsequent data processing, thereby achieving the sharing of computing pressure and further reducing the time delay of data processing.

[0008] In a possible design, the data gateway device determines the target number of working nodes for preprocessing the source data, including: determining, by the data gateway device, a processing duration corresponding to the source data in a case where a sending frequency is greater than or equal to a preset frequency, the sending frequency being a frequency at which the data source device sends the source data, and the processing duration being a duration required by the data gateway device to receive the source data and send the processed source data to the edge node device; and determining the target number according to the sending frequency and the processing duration. This design provides an implementation manner in which the data gateway device determines the target number of working nodes for preprocessing the source data.

[0009] In a possible design, the data gateway device determines the target number of working nodes for preprocessing the source data, including: determining, by the data gateway device, a preprocessing procedure corresponding to the source data in a case where a sending frequency is less than a preset frequency; and determining a number of steps included in the preprocessing procedure as the target number. This design provides an implementation manner in which the data gateway device determines the target number of working nodes for preprocessing the source data.

[0010] In a possible design, the data gateway device determines the target number of working nodes for preprocessing the source data, including: determining, by the data gateway device, a data type required by the edge node device in a case where a sending frequency is less than a preset frequency; and determining a number of data types as the target number. This design provides an implementation manner in which the data gateway device determines the target number of working nodes for preprocessing the source data.

[0011] In a possible design, the data gateway device determines the target number of working nodes for preprocessing the source data, including: determining, by the data gateway device, a number of edge node devices corresponding to the source data as the target number in a case where a sending frequency is less than a preset frequency and each piece of data in the source data has a corresponding edge node device. This design provides an implementation manner in which the data gateway device determines the target number of working nodes for preprocessing the source data.

[0012] In a possible design, the data method further includes: releasing, by the data gateway device, the target number of working nodes after the data gateway device does not receive the source data sent by the data source device for more than a preset duration. This design releases the occupied computing power resources after the data source device no longer reports data, thereby avoiding waste of computing power resources.

[0013] Secondly, a data processing apparatus is provided, comprising a data gateway device deployed in a business system. The business system also includes a data source device and edge node devices. The data gateway device is connected to both the data source device and the edge node devices. The data processing apparatus includes a receiving unit, a determining unit, a processing unit, and a sending unit. The receiving unit receives source data sent by the data source device. The determining unit determines a target number of working nodes for preprocessing the source data. The processing unit processes the source data based on the target number of working nodes. The sending unit sends the processed source data to the edge node devices.

[0014] In one possible design, a determining unit is specifically used to determine the processing time corresponding to the source data when the transmission frequency is greater than or equal to a preset frequency; the transmission frequency is the frequency at which the data source device sends the source data, and the processing time is the time required for the data gateway device to send the processed source data to the edge node device after receiving the source data; the target quantity is determined based on the transmission frequency and the processing time.

[0015] In one possible design, a determining unit is specifically used to determine the preprocessing flow corresponding to the source data when the transmission frequency is less than a preset frequency; the number of steps included in the preprocessing flow is determined as the target number.

[0016] In one possible design, a determining unit is specifically used to determine the data types required by the edge node device when the transmission frequency is less than a preset frequency; and the number of data types is determined as the target number.

[0017] In one possible design, the determining unit is specifically used to determine the number of edge node devices corresponding to the source data as the target number when the transmission frequency is less than the preset frequency and each piece of data in the source data has a corresponding edge node device.

[0018] In one possible design, the processing unit is also used to release a target number of working nodes after a preset time has elapsed if no source data is received from the data source device.

[0019] Thirdly, a network device is provided, the network device including a memory and a processor; the memory and the processor are coupled, the memory being used to store computer program code including computer instructions, and when the processor executes the computer instructions, the network device performs a data processing method as provided in the first aspect or any possible design thereof.

[0020] Fourthly, a computer-readable storage medium is provided, in which instructions are stored, which, when executed on a network device, cause the network device to perform a data processing method as provided in the first aspect or any possible implementation thereof.

[0021] Fifthly, a computer program product is provided, comprising computer instructions that, when executed on a network device, enable the network device to perform a data processing method as provided in the first aspect or any possible implementation thereof. Attached Figure Description

[0022] Figure 1 A schematic diagram of the structure of a business system provided for an embodiment of this application;

[0023] Figure 2 A schematic diagram of the structure of a traffic splitting module provided for an embodiment of this application;

[0024] Figure 3 A schematic flowchart of a data processing method provided for embodiments of this application. Figure 1 ;

[0025] Figure 4 A schematic flowchart of a data processing method provided for embodiments of this application. Figure 2 ;

[0026] Figure 5 A schematic flowchart of a data processing method provided for embodiments of this application. Figure 3 ;

[0027] Figure 6 A schematic flowchart of a data processing method provided for embodiments of this application. Figure 4 ;

[0028] Figure 7 A schematic flowchart of a data processing method provided for embodiments of this application. Figure 5 ;

[0029] Figure 8 A schematic diagram of a node configuration provided for an embodiment of this application;

[0030] Figure 9 A schematic diagram of a data processing device structure provided for an embodiment of this application;

[0031] Figure 10 A schematic diagram of a network device structure provided for embodiments of this application. Figure 1 ;

[0032] Figure 11 A schematic diagram of a network device structure provided for embodiments of this application. Figure 2 . Detailed Implementation

[0033] The technical solutions in the embodiments of this application will now be described with reference to the accompanying drawings.

[0034] In the embodiments of this application, the terms "exemplary" or "for example" are used to indicate that something is an example, illustration, or description. Any embodiment or design that is described as "exemplary" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design. Specifically, the use of the terms "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.

[0035] In the description of this application, unless otherwise stated, " / " means "or". For example, A / B can mean A or B. "And / or" in this document is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, and B alone. Furthermore, "at least one" and "multiple" refer to two or more. The terms "first," "second," etc., do not limit the quantity or order of execution, and "first," "second," etc., do not necessarily imply differences.

[0036] In certain industries, such as industrial control, business applications have increasingly stringent requirements for data real-time performance. The latency requirements from data generation at the device data source to its display on the human-machine interface terminal are often in the millisecond range, or even lower, in order to ensure the effectiveness of device data monitoring or control. In such scenarios, traditional centralized cloud scheduling and processing often involves large-scale batch processing. Considering the latency of transmitting the processed data to the corresponding terminal, it is difficult to guarantee data real-time performance.

[0037] In application scenarios with a single data source and multiple data flows, message middleware is often used to distribute and forward data. However, adding message middleware increases system complexity, reduces availability, and if the message middleware fails, the entire system's data flow will be forced to stop. In addition, message middleware often uses an asynchronous message queue (MQ) model, making it less suitable for industries with high real-time requirements.

[0038] To address the aforementioned problems, this application proposes a data processing method, apparatus, device, and storage medium, applied to a data gateway device in a business system. The business system also includes a data source device and edge node devices. The data gateway device is connected to both the data source device and the edge node devices. The data processing method includes: the data gateway device receiving source data sent by the data source device; further, the data gateway device determining a target number of working nodes to preprocess the source data; further, the data gateway device processing the source data based on the target number of working nodes and sending the processed source data to the edge node devices.

[0039] In this way, after receiving source data from the data source device, the data gateway device determines the target number of worker nodes to preprocess the source data, creates the target number of worker nodes to process the source data, and then sends the processed source data to the corresponding edge node devices. This allows the source data to be processed immediately without waiting, reducing data processing latency. Furthermore, because the edge nodes participate in subsequent data processing, the computational load is distributed, further reducing data processing latency.

[0040] Figure 1 This application illustrates a business system, and the data processing method provided in this embodiment can be applied to systems such as... Figure 1 The business system shown is designed to improve data processing efficiency in application scenarios with multiple data flows from a single data source. For example... Figure 1 As shown, the business system 10 includes a data gateway device 11, a data source device 12, and multiple edge node devices 13.

[0041] The following needs to be explained: Figure 1 The example shows three edge node devices 13, which does not constitute a limitation on the number of edge node devices. It is only used to illustrate that the data processing method provided in this application embodiment can be applied to a business system that includes multiple edge node devices. Furthermore, the multiple edge node devices 13 can be edge node devices of the same type and model, or they can be edge node devices of different types and models. This application embodiment does not make any specific limitation in this regard.

[0042] The data gateway device 11 is connected to the data source device 12 and multiple edge node devices 13. The connection can be wired or wireless, and this application embodiment does not limit the connection.

[0043] like Figure 2 As shown, the data gateway device 11 includes a traffic splitting module 111 and a configuration module 112. It should be noted that each of the multiple edge node devices 13 includes a data processing module 131 and a service invocation module 132.

[0044] Among them, the data source device 12 is the single data source for the business system 10, and sends data to the data gateway device 11 in real time, either one by one or in batches.

[0045] It should be noted that the data source device 12 can be a sensor device such as a smoke sensor, a distance sensor, or a surveillance camera, used to acquire and collect data and push it to the data gateway device. The data source device 12 can also be an intermediate device connecting multiple sensor devices, used to aggregate the data collected by multiple sensor devices and push it to the data gateway device; this application does not specifically limit this.

[0046] The data gateway device 11 is a service gateway used to preprocess data and send the processed data to the corresponding edge node device 13 according to preset rules.

[0047] The configuration module 112 carries the data diversion logic pre-configured by the operation and maintenance personnel of the business system 10. The data diversion logic includes pre-processing procedures for single or multiple data items and diversion strategies.

[0048] The splitting module 111 can be used to perform data preprocessing and data splitting tasks according to the splitting strategy configured by the configuration module 112.

[0049] Understandably, in order to ensure the real-time transmission of data, the diversion module 111 processes the source data according to the amount of source data and the diversion strategy. Unlike the data caching mechanism provided by the message queue, the source data at the data gateway device 11 is processed and sent out "as soon as it arrives".

[0050] like Figure 3 As shown, the splitting module 111 includes multiple working nodes 21 and a scheduling module 22. The working nodes 21 are used to split the source data. The scheduling module dynamically adjusts the number of working nodes according to the set measurement.

[0051] Edge node device 13 is a dedicated small server with computing capabilities deployed for a specific subsystem or sub-service. It is used to process data sent by the data gateway device in real time. Since the data structure has been preprocessed on the gateway device, the edge node device only needs to process the data with simple processing logic after receiving the data.

[0052] The data processing module 131 is used to perform data querying or extraction.

[0053] The business call module 132 is for the application, client, etc. of the corresponding edge node device 13.

[0054] In some embodiments, the data processing method provided in this application can be applied to a data processing device deployed on a data gateway device, or it can be directly applied to the data gateway device. This application takes the application to a data gateway device as an example to illustrate the data processing method.

[0055] The data source device 12 can be used to push the collected source data to the data gateway device 11.

[0056] Correspondingly, the data gateway device 11 can be used to receive source data sent by the data source device 12.

[0057] Data gateway device 11 can also be used to determine the target number of working nodes that preprocess the source data.

[0058] The data gateway device 11 can also be used to create a target number of worker nodes. Furthermore, the data gateway device processes the source data based on the target number of worker nodes.

[0059] The data gateway device 11 can also be used to send processed source data to the edge node device 13.

[0060] Correspondingly, the edge node device 13 can be used to obtain corresponding feature data from the processed source data according to the configured rules after receiving the processed source data, so as to meet the user's data requirements for the business system 10.

[0061] Figure 1 This is a flowchart illustrating a data processing method according to some exemplary embodiments. In some embodiments, the above-described data processing method can be applied to, for example... Figure 3 The data gateway device 11 in the business system 10 shown. Hereinafter, this application embodiment uses the application of the data processing method to the data gateway device 11 as an example to describe the above data processing method.

[0062] like Figure 4 As shown, the data processing method provided in this application embodiment includes the following S301-S304.

[0063] S301. The data gateway device receives source data sent by the data source device.

[0064] As one possible implementation, the data source device collects data and reports it to the data gateway device based on its own configuration information.

[0065] Correspondingly, the data gateway device receives the source data reported by the data source device.

[0066] It should be noted that the configuration information of the data source device itself can be sent immediately after the data is collected, or the collected data can be sent periodically. That is, the source data sent by the data source device can be a single data point or include multiple data points. This application embodiment does not make specific limitations on this.

[0067] For example, the data source device can be a surveillance camera used for monitoring instruments, and its configuration information is to push surveillance image frames every second. Correspondingly, the data gateway device receives the surveillance image frames sent by the data source device.

[0068] S302, The data gateway device determines the target number of working nodes for preprocessing the source data.

[0069] As one possible implementation, the data gateway device determines the transmission frequency of the source data from the data source device, and determines the target number of working nodes required based on the transmission frequency of the source data.

[0070] For example, if the data gateway device determines that the transmission frequency is greater than the preset frequency, it determines the target number of working nodes as a first number; if the data gateway device determines that the transmission frequency is less than the preset frequency, it determines the target number of working nodes as a second number, wherein the first number is greater than the second number, so that the working nodes inside the data gateway device can meet the processing latency of the business system's requirements for source data.

[0071] It should be noted that the preset frequency can be set in advance in the data gateway device by the operation and maintenance personnel of the business system, and this application embodiment does not specifically limit this.

[0072] In some embodiments, the data gateway device determines a target number of working nodes to preprocess the source data in one implementation, as shown below.

[0073] When the data gateway device transmits data at a frequency lower than the preset frequency, and each piece of data in the source data has a corresponding edge node device, the number of edge node devices corresponding to the source data is determined as the target number.

[0074] For example, if the source data includes four data items: data A, data B, data C, and data D, and the edge node devices corresponding to data A and data B are edge node device 1, the edge node device corresponding to data B is edge node device 2, and the edge node device corresponding to data C is edge node device 3, then the data gateway device determines that the edge node devices corresponding to the source data are edge node device 1, edge node device 2, and edge node device 3, and thus determines that the number of edge node devices corresponding to the source data is 3, which is also the target number.

[0075] It should be noted that how the data gateway device specifically determines the target number of working nodes for preprocessing the source data can be referred to the subsequent description in the embodiments of this application, and will not be repeated here.

[0076] S303, The data gateway device processes source data based on a target number of working nodes.

[0077] As one possible implementation, the data gateway device creates a target number of working nodes based on the target number determined in step S302 above, and configures processing rules for each of the target number of working nodes. Furthermore, the data gateway device simultaneously processes the source data according to the processing rules based on each of the target number of working nodes.

[0078] It should be noted that the processing rules can be pre-set by the operation and maintenance personnel of the business system in the configuration module of the data gateway device. The scheduling module configures the processing rules for each working node according to the processing rules. For example, the processing rules may include operations such as decoding, conversion, and processing. This application embodiment does not specifically limit this.

[0079] S304. The data gateway device sends the processed source data to the edge node device.

[0080] As one possible implementation, in a data gateway device, the processing rules for each of the target number of working nodes are customized based on the data requirements of the edge node devices. Therefore, after each working node in the data gateway device preprocesses the source data, the data gateway device transmits the processed source data to the corresponding edge node device.

[0081] In some examples, edge node device 1 requires data A from the source data, edge node device 2 requires data B from the source data, and edge node device 3 requires data C from the source data. Data A is contained in the processed source data generated by worker node 1, data B is contained in the processed source data generated by worker node 2, and data C is contained in the processed source data generated by worker node 3. The data gateway device then establishes end-to-end relationships between worker node 1 and edge node device 1, worker node 2 and edge node device 2, and worker node 3 and edge node device 3. This ensures that each worker node, after receiving the processed source data, transmits the processed source data to the corresponding edge node device, thus satisfying the data requirements of the edge node devices.

[0082] In some embodiments, the data gateway device starts timing after receiving source data sent by the data source device, and restarts timing if it receives source data sent by the data gateway device again.

[0083] To avoid excessive worker nodes and wasted resources, the data gateway device releases the target number of worker nodes after a preset time period without receiving source data from the data source device, thereby ceasing the occupation of computing resources.

[0084] It should be noted that the preset duration can be set in advance in the data gateway device by the operation and maintenance personnel of the business system according to the latency requirements of the business system for the source data. This application embodiment does not make specific limitations on this.

[0085] Understandably, in the data processing method provided in this application embodiment, after receiving source data sent by the data source device, the data gateway device determines the target number of working nodes to preprocess the source data, creates the target number of working nodes to process the source data, and then sends the processed source data to the corresponding edge node devices. This allows the source data to be processed immediately without waiting, reducing data processing latency. Furthermore, since the edge nodes participate in subsequent data processing, computational pressure is distributed, further reducing data processing latency.

[0086] In one design, to meet the latency requirements of the business system for the source data, the data gateway device determines the target number of working nodes to be allocated to the source data, such as... Figure 5 As shown, the data processing method provided in this application embodiment also includes S401-S402.

[0087] S401. When the data gateway device transmits data at a frequency greater than or equal to a preset frequency, it determines the processing time corresponding to the source data.

[0088] The sending frequency is the frequency at which the data source device sends source data, and the processing time is the time required for the data gateway device to send the processed source data to the edge node device after receiving the source data.

[0089] As one possible implementation, the data gateway device first determines the transmission frequency of the source data from the data source device based on its configuration information. Further, the data gateway device compares the transmission frequency with a preset frequency, and if the transmission frequency is greater than or equal to the preset frequency, determines the processing time corresponding to the source data.

[0090] It should be noted that the processing time corresponding to the source data can be calculated by the data gateway device based on the data structure and data type of the source data, as well as the computing power of the worker nodes allocated by the data gateway device. Furthermore, since the source data reported by a single data source device has a certain degree of similarity, the processing time corresponding to the source data can also be preset in the data gateway device by the operation and maintenance personnel of the business system; this embodiment of the application does not specifically limit this.

[0091] S402. The data gateway device determines the target quantity based on the transmission frequency and processing time.

[0092] As one possible implementation, the data gateway device calculates based on a preset calculation formula, according to the transmission frequency and processing time, and further determines the calculation result as the target quantity.

[0093] It should be noted that the preset calculation formula can be transmission frequency × processing time, and the product can be rounded up; it can also be coefficient K × transmission frequency × processing time, and the product can be rounded up. This application embodiment does not specifically limit this.

[0094] For example, if the preset calculation formula is sending frequency × processing time, and rounded up, and the data source device sends source data at a sending frequency of 500 messages / second and a processing time of 10 milliseconds, then the data gateway device calculates 500 × 0.01 = 5 based on the sending frequency and processing time, and then the data gateway device determines the target quantity to be 5.

[0095] If the data source device sends source data at a frequency of 325 messages / second and the processing time is 10 milliseconds, the data gateway device calculates 325 × 0.01 = 3.25 based on the sending frequency and processing time, and then determines the target quantity to be 4.

[0096] Understandably, in the data processing method provided in this application embodiment, when the data source device sends data too frequently and a single working node cannot handle the data, the number of working nodes is determined based on the sending frequency of the data source device and the processing time of the source data, so that multiple working nodes can process the source data simultaneously. Since the number of working nodes is related to the source data to be processed, too many working nodes will not be created, thus avoiding a waste of computing resources.

[0097] In one design, to meet the latency requirements of the business system for the source data, the data gateway device determines the target number of working nodes to be allocated to the source data, such as... Figure 6 As shown, the data processing method provided in this application embodiment also includes S501-S502.

[0098] S501. When the data gateway device transmits data at a frequency lower than the preset frequency, it determines the preprocessing procedure corresponding to the source data.

[0099] As one possible implementation, the data gateway device first determines the transmission frequency of the source data from the data source device based on its configuration information. Further, the data gateway device compares the transmission frequency with a preset frequency, and if the transmission frequency is lower than the preset frequency, determines the steps included in the preprocessing flow corresponding to the source data.

[0100] It should be noted that the steps included in the preprocessing process can be obtained from the configuration module and pre-configured in the configuration module by the operation and maintenance personnel of the business system. Alternatively, the data gateway device can determine the preprocessing process based on the data type of the source data and the data requirements of the edge node devices. For example, if the source data is image data and the data requirements of the edge node devices are information captured in the images, then the preprocessing process may include image recognition and key field extraction. This application embodiment does not specifically limit this.

[0101] S502, The data gateway device determines the number of steps included in the preprocessing process as the target number.

[0102] As one possible implementation, the data gateway device determines the number of steps included in the preprocessing flow as a target number based on the steps identified in step S501 above. In this way, a target number of working nodes can be created to process each step, thereby ensuring the efficiency of processing the source data.

[0103] In one design, to meet the latency requirements of the business system for the source data, the data gateway device determines the target number of working nodes to be allocated to the source data, such as... Figure 7 As shown, the data processing method provided in this application embodiment also includes S601-S602.

[0104] S601. When the data gateway device transmits data at a frequency lower than a preset frequency, it determines the data type required by the edge node device.

[0105] As one possible implementation, the data gateway device first determines the transmission frequency of the source data from the data source device based on its configuration information. Further, the data gateway device compares the transmission frequency with a preset frequency, and if the transmission frequency is lower than the preset frequency, determines the data type required by the edge node device.

[0106] It should be noted that the data types required by edge node devices can be pre-configured in the data gateway device by the operation and maintenance personnel of the business system. For example, if the source data is a monitoring image frame, and the monitoring image frame includes images from multiple meters, and the meters display meter data, then the data type can be the meter data of each meter; as another example, the data types required by edge node devices include image data and keyword data, and this application embodiment does not specifically limit this.

[0107] S602, The data gateway device determines the number of data types as the target number.

[0108] As one possible implementation, the data gateway device determines the number of included data types as the target number based on the data types determined in step S601 above. In this way, the target number of working nodes can be created to process each data type, thereby ensuring the efficiency of processing the source data.

[0109] In one design, in conjunction with the above embodiments of this application, such as... Figure 8 As shown, the data processing method provided in this application embodiment also includes S701-S706.

[0110] S701, The data gateway device determines the sending frequency of the data source device and its relationship with the preset frequency.

[0111] It should be noted that when the transmission frequency is greater than or equal to the preset frequency, the data gateway device executes step S702; when the transmission frequency is less than the preset frequency, the data gateway device executes step S703.

[0112] S702. The data gateway device determines the target number of working nodes to preprocess the source data based on the transmission frequency and processing time.

[0113] S703, the data gateway device determines whether the source data is a single data entry.

[0114] It should be noted that when the source data is a single data entry, the data gateway device executes step S704; when the source data is not a single data entry, the data gateway device executes step S705.

[0115] S704. The data gateway device determines the target number of working nodes to preprocess the source data based on the preprocessing process corresponding to the source data or the data type required by the edge node device.

[0116] S705, The data gateway device determines whether each piece of data in the source data has a corresponding edge node device.

[0117] It should be noted that if each piece of data in the source data has a corresponding edge node device, the data gateway device executes step S706; if each piece of data in the source data does not have a corresponding edge node device, the data gateway device executes step S704.

[0118] S706. The data gateway device determines the target number of working nodes to preprocess the source data based on the number of edge node devices corresponding to the source data.

[0119] It should be noted that the specific implementation of the above steps S701-S706 can be referred to the description in the above embodiments of this application, and will not be repeated here.

[0120] In some embodiments, if the source data is a monitoring image frame, and the monitoring image is used to capture the meter data of 5 meters, wherein the 5 meters include meter A, meter B, meter C, meter D and meter E, edge node device 1 corresponds to the meter data of meter A, edge node device 2 corresponds to the meter data of meter B, edge node device 3 corresponds to the meter data of meter C, edge node device 4 corresponds to the meter data of meter D, and edge node device 5 corresponds to the meter data of meter E.

[0121] The data gateway device then determines that working node 1 needs to identify and parse the monitoring image frames, and transmits the identified and parsed data to working nodes 2-6 respectively. Figure 9 As shown, after preprocessing the data by worker node 2, the instrument data of instrument A is transmitted to edge node device 1. After preprocessing the data by worker node 3, the instrument data of instrument B is transmitted to edge node device 2. After preprocessing the data by worker node 4, the instrument data of instrument C is transmitted to edge node device 3. After preprocessing the data by worker node 5, the instrument data of instrument D is transmitted to edge node device 4. After preprocessing the data by worker node 6, the instrument data of instrument E is transmitted to edge node device 5. This further ensures that each edge node device can extract the corresponding data to meet the needs of the business system.

[0122] The foregoing mainly describes the solutions provided by the embodiments of this application from a methodological perspective. To achieve the above functions, it includes corresponding hardware structures and / or software modules for executing each function. Those skilled in the art should readily recognize that, in conjunction with the units and algorithm steps of the various examples described in the embodiments disclosed herein, the embodiments of this application can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed in hardware or by computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0123] This application embodiment can divide the user equipment into functional modules according to the above method example. For example, each function can be divided into a separate functional module, or two or more functions can be integrated into one processing module. The integrated module can be implemented in hardware or as a software functional module. Optionally, the module division in this application embodiment is illustrative and only represents one logical functional division; other division methods may be used in actual implementation.

[0124] Figure 9This is a schematic diagram of a data processing apparatus provided in an embodiment of this application. This data processing apparatus is used to execute the data processing method described above. Figure 3 As shown, the data processing device 80 includes a receiving unit 801, a determining unit 802, a processing unit 803, and a sending unit 804.

[0125] The receiving unit 801 is used to receive source data sent by the data source device. For example, such as... Figure 3 As shown, the receiving unit 801 can be used to execute S301.

[0126] The determining unit 802 is used to determine the target number of working nodes for preprocessing the source data. For example, such as... Figure 3 As shown, the determining unit 802 can be used to execute S302.

[0127] Processing unit 803 is used to process source data based on a target number of working nodes. For example, ... Figure 3 As shown, the processing unit 803 can be used to execute S303.

[0128] The sending unit 804 is used to send processed source data to the edge node device. For example, ... Figure 9 As shown, the sending unit 804 can be used to execute S304.

[0129] Optional, such as Figure 9 As shown, in the data processing apparatus 80 provided in this application embodiment, the determining unit 802 is specifically used to determine the processing time corresponding to the source data when the sending frequency is greater than or equal to a preset frequency; the sending frequency is the frequency at which the data source device sends the source data, and the processing time is the time required for the data gateway device to send the processed source data to the edge node device after receiving the source data; the target quantity is determined according to the sending frequency and the processing time.

[0130] Optional, such as Figure 9 As shown, in the data processing apparatus 80 provided in this application embodiment, the determining unit 802 is specifically used to determine the preprocessing process corresponding to the source data when the transmission frequency is less than the preset frequency; and to determine the number of steps included in the preprocessing process as the target number.

[0131] Optional, such as Figure 9 As shown, in the data processing apparatus 80 provided in this application embodiment, the determining unit 802 is specifically used to determine the data type required by the edge node device when the transmission frequency is less than the preset frequency; and to determine the number of data types as the target number.

[0132] Optional, such as Figure 9As shown, in the data processing apparatus 80 provided in this application embodiment, the determining unit 802 is specifically used to determine the number of edge node devices corresponding to the source data as the target number when the transmission frequency is less than the preset frequency and each piece of data in the source data has a corresponding edge node device.

[0133] Optional, such as Figure 10 As shown, in the data processing apparatus 80 provided in this application embodiment, the processing unit 803 is further configured to release a target number of working nodes after a preset time period has elapsed without receiving source data sent by the data source device.

[0134] In implementing the functions of the integrated modules described above using hardware, this application provides a possible structural diagram of a network device. This network device is used to execute the data processing method performed by the data processing device in the above embodiments. Figure 10 As shown, the network device 90 includes a processor 901, a memory 902, and a bus 903. The processor 901 and the memory 902 can be connected via the bus 903.

[0135] Processor 901 is the control center of the network device. It can be a single processor or a collective term for multiple processing elements. For example, processor 901 can be a general-purpose central processing unit (CPU) or other general-purpose processors. The general-purpose processor can be a microprocessor or any conventional processor.

[0136] As one embodiment, processor 901 may include one or more CPUs, for example Figure 10 CPU 0 and CPU 1 are shown in the diagram.

[0137] The memory 902 may be a read-only memory (ROM) or other type of static storage device capable of storing static information and instructions, random access memory (RAM) or other type of dynamic storage device capable of storing information and instructions, or electrically erasable programmable read-only memory (EEPROM), disk storage media or other magnetic storage devices, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but is not limited thereto.

[0138] As one possible implementation, the memory 902 can exist independently of the processor 901. The memory 902 can be connected to the processor 901 via a bus 903 and is used to store instructions or program code. When the processor 901 calls and executes the instructions or program code stored in the memory 902, it can implement the data processing method provided in the embodiments of this application.

[0139] In another possible implementation, the memory 902 can also be integrated with the processor 901.

[0140] Bus 903 can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus. This bus can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 10 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0141] It should be pointed out that, Figure 10 The structure shown does not constitute a limitation on the network device 90. Except... Figure 10 In addition to the components shown, the network device 90 may include more than Figure 9 It can show more or fewer parts, or combine certain parts, or arrange different parts.

[0142] As an example, combined Figure 10 The functions implemented by the receiving unit 801, determining unit 802, processing unit 803, and sending unit 804 in the data processing device 80 are the same as those of the receiving unit 801, determining unit 802, processing unit 803, and sending unit 804. Figure 10 The processor 901 in it has the same function.

[0143] Optional, such as Figure 11 As shown, the network device provided in this application embodiment may further include a communication interface 904.

[0144] Communication interface 904 is used to connect with other devices via a communication network. This communication network can be Ethernet, a wireless access network, a wireless local area network (WLAN), etc. Communication interface 904 may include an acquisition unit for receiving data and a transmission unit for sending data.

[0145] In one design, the communication interface of the network device provided in this application embodiment can also be integrated into the processor.

[0146] Figure 11 Another hardware structure of the network device in an embodiment of this application is shown. For example... Figure 11 As shown, network device 100 may include processor 1001 and communication interface 1002. Processor 1001 is coupled to communication interface 1002.

[0147] The functions of processor 1001 can be referred to in the description of processor 901 above. In addition, processor 1001 also has storage functions, which can be referred to in the description of memory 902 above.

[0148] The communication interface 1002 is used to provide data to the processor 1001. The communication interface 1002 can be an internal interface of the network device or an external interface of the network device (equivalent to the communication interface 904).

[0149] It should be pointed out that, Figure 11 The structure shown does not constitute a limitation on network devices, except ​ In addition to the components shown, the network device 100 may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0150] Through the above description of the embodiments, those skilled in the art will clearly understand that, for the sake of convenience and brevity, only the division of the above functional units is used as an example. In practical applications, the above functions can be assigned to different functional units as needed, that is, the internal structure of the device can be divided into different functional units to complete all or part of the functions described above. The specific working process of the system, device, and unit described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0151] This application also provides a computer-readable storage medium storing instructions. When a computer executes these instructions, the computer performs each step of the method flow shown in the above method embodiments.

[0152] Embodiments of this application provide a computer program product containing instructions that, when executed on a computer, cause the computer to perform the data processing method described in the above method embodiments.

[0153] The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of computer-readable storage media (a non-exhaustive list) include: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), registers, hard disks, optical fibers, compact disc read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing, or any other form of computer-readable storage medium in the art. An exemplary storage medium is coupled to a processor, enabling the processor to read information from and write information to the storage medium. Of course, the storage medium may also be a component of the processor. The processor and the storage medium may reside in an application-specific integrated circuit (ASIC). In the embodiments of this application, the computer-readable storage medium can be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0154] Since the apparatus, device, computer-readable storage medium, and computer program product in the embodiments of this application can be applied to the above methods, the technical effects that can be obtained can also be referred to the above method embodiments. The embodiments of this application will not be repeated here.

[0155] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A data processing method, characterized in that, A data gateway device applied in a business system, the business system further including a data source device and an edge node device, the data gateway device being connected to the data source device and the edge node device respectively, the method comprising: Receive source data sent by the data source device; If the transmission frequency is greater than or equal to a preset frequency, the processing time corresponding to the source data is determined; the transmission frequency is the frequency at which the data source device sends the source data, and the processing time is the time required for the data gateway device to receive the source data and send the processed source data to the edge node device; based on the product of the transmission frequency and the processing time, the target number of working nodes for preprocessing the source data is determined; or... If the transmission frequency is less than the preset frequency, determine the preprocessing procedure corresponding to the source data, and set the number of steps included in the preprocessing procedure as the target quantity; or determine the data type required by the edge node device, and set the quantity of the data type as the target quantity; or... If the transmission frequency is less than the preset frequency, and each piece of data in the source data has a corresponding edge node device, the number of edge node devices corresponding to the source data is determined as the target number. The source data is processed by the target number of working nodes, and the processed source data is sent to the edge node device.

2. The data processing method according to claim 1, characterized in that, The method further includes: If the source data is not received from the data source device for a preset period of time, the target number of working nodes will be released.

3. A data processing apparatus, characterized in that, A data gateway device deployed in a business system, the business system also including a data source device and an edge node device, the data gateway device being connected to the data source device and the edge node device respectively, and the data processing device including a receiving unit, a determining unit, a processing unit and a sending unit; The receiving unit is used to receive source data sent by the data source device; The determining unit is used to determine the processing time corresponding to the source data when the transmission frequency is greater than or equal to a preset frequency. The sending frequency is the frequency at which the data source device sends the source data, and the processing time is the time required for the data gateway device to send the processed source data to the edge node device after receiving the source data. The target number of working nodes for preprocessing the source data is determined based on the product of the transmission frequency and the processing time. Alternatively, if the transmission frequency is less than the preset frequency, determine the preprocessing process corresponding to the source data and set the number of steps included in the preprocessing process as the target number; or determine the data type required by the edge node device and set the number of the data types as the target number; or, if the transmission frequency is less than the preset frequency and each piece of data in the source data has a corresponding edge node device, set the number of edge node devices corresponding to the source data as the target number. The processing unit is used to process the source data based on the target number of working nodes; The sending unit is used to send the processed source data to the edge node device.

4. The data processing apparatus according to claim 3, characterized in that, The processing unit is also configured to release the target number of working nodes after a preset time period has elapsed if the source data sent by the data source device has not been received.

5. A network device, characterized in that, Including memory and processor; The memory and the processor are coupled; The memory is used to store computer program code, which includes computer instructions; When the processor executes the computer instructions, the network device performs the data processing method as described in claim 1 or 2.

6. A computer-readable storage medium storing instructions, characterized in that, When the instruction is executed on the network device, the network device performs the data processing method as described in claim 1 or 2.

Images