Data acquisition method and device, equipment and storage medium

Abstract

The invention discloses a data acquisition method, apparatus and device, and a storage medium. The data acquisition method comprises the steps of receiving a data acquisition configuration command sent by a data acquisition management node; performing data acquisition according to the data acquisition configuration command, and generating serialized streaming data from the acquired data; and sending the serialized streaming data to a data receiving end through a data processing intermediate platform. According to the invention, the data acquisition configuration command is issued in a programmable manner, so that the flexibility of data acquisition is improved; by supporting streaming data acquisition, the key requirements of large data volume and real-time data acquisition are met.

Description

Technical Field

[0001] The present invention relates to the technical field of data processing, and in particular to a data acquisition method, device, equipment and storage medium. Background Art

[0002] With the advancement of network intelligence, intelligence has been applied to a certain extent in the fields of network management and network operation, and data is the basis for realizing intelligence. At present, the main network data acquisition technologies include the following: data acquisition technology for network management, hardware acquisition and software acquisition methods of the unified packet detection DPI system, and the service-based interface-based acquisition method of 5GC NWDAF. The current acquisition scheme for network management data is relatively mature, but the requirements for the real-time acquisition and data volume of network management data are not high. For the hardware acquisition method of DPI data, the data needs to be parsed twice, the accuracy is not high, and even accurate data content cannot be obtained, and the cost is relatively high; the software acquisition method has a great impact on the device performance and is not standardized. The data acquisition method of the NWDAF service-based interface cannot meet the data acquisition requirements of large data volumes, and also has a great impact on the performance of the device. Therefore, for the application requirements of the big data base in the network evolution, such as the large-scale deployment of network AI applications characterized by data-driven, and the network digital twin based on real-time big data, these applications will be used for real-time analysis, diagnosis, and closed-loop of the network operation state, which puts forward higher requirements for data acquisition. The existing data acquisition technologies are difficult to meet the needs of future network evolution for large data volumes and real-time data acquisition. Summary of the Invention

[0003] The technical problem to be solved by the present invention is to provide a data acquisition method, device, equipment and storage medium, which improve the flexibility of data acquisition by issuing data acquisition configuration commands in a programmable manner, and solve the key requirements of large data volume and real-time data acquisition by supporting the streaming data acquisition method.

[0004] To achieve the above object, an embodiment of the present invention provides a data acquisition method, which is applied to a network node, and the method includes:

[0005] Receiving a data acquisition configuration command sent by a data acquisition management node;

[0006] Performing data acquisition according to the data acquisition configuration command, and generating serialized streaming data from the acquired data;

[0007] Sending the serialized streaming data to a data receiving end through a data processing intermediate platform.

[0008] As an improvement of the above solution, the data acquisition configuration command is encapsulated in a programmable framework, and the data acquisition configuration command includes acquisition content, acquisition frequency, and acquisition time.

[0009] As an improvement to the above solution, data collection is performed according to the data collection configuration command, and the collected data is generated into serialized streaming data, specifically including:

[0010] Data collection is performed according to the collection content, collection frequency, and collection time included in the data collection configuration command;

[0011] The collected data is serialized to generate the serialized streaming data.

[0012] As an improvement to the above solution, sending the serialized streaming data to the data receiving end through the data processing intermediate platform specifically includes:

[0013] The serialized streaming data is sent and written to the data processing intermediate platform through the serialized stream data interface, and the data processing intermediate platform sends the serialized streaming data to the corresponding data receiving end through the deserialized stream data interface based on the subscription message sent by the data receiving end.

[0014] An embodiment of the present invention further provides a data collection method, which is applied to a data processing intermediate platform, and the method includes:

[0015] Receiving a subscription message sent by a data receiving end;

[0016] Receiving serialized streaming data sent by a network node and performing distributed storage;

[0017] Based on the subscription message sent by the data receiving end, sending the serialized streaming data to the corresponding data receiving end through the deserialized stream data interface.

[0018] An embodiment of the present invention further provides a data collection device, including:

[0019] A command receiving module, configured to receive a data collection configuration command sent by a data collection management node;

[0020] A data collection module, configured to perform data collection according to the data collection configuration command and generate serialized streaming data from the collected data;

[0021] A data sending module, configured to send the serialized streaming data to the data receiving end through the data processing intermediate platform.

[0022] An embodiment of the present invention further provides a data collection device, including:

[0023] A subscription receiving module, configured to receive a subscription message sent by a data receiving end;

[0024] A data receiving module, configured to receive serialized streaming data sent by a network node and perform distributed storage;

[0025] A data sending module, configured to send the serialized streaming data to a corresponding data receiving end through a deserialized stream data interface based on a subscription message sent by the data receiving end.

[0026] An embodiment of the present invention further provides a terminal device, including a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor. When the processor executes the computer program, the data acquisition method described in any one of the above is implemented.

[0027] An embodiment of the present invention further provides a computer-readable storage medium, which includes a stored computer program. When the computer program runs, the device where the computer-readable storage medium is located is controlled to execute the data acquisition method described in any one of the above.

[0028] Compared with the prior art, the beneficial effects of a data acquisition method, device, equipment, and storage medium provided by an embodiment of the present invention are as follows: receiving a data acquisition configuration command sent by a data acquisition management node; performing data acquisition according to the data acquisition configuration command, and generating serialized streaming data from the acquired data; sending the serialized streaming data to a data receiving end through a data processing intermediate platform. By issuing a data acquisition configuration command in a programmable manner, the flexibility of data acquisition is improved in the embodiment of the present invention; by supporting the method of streaming data acquisition, the key requirements of large data volume and real-time data acquisition are solved; by using a stream data processing platform as an intermediate platform, the data consumption requirements of multiple data receiving ends are met based on one-time data acquisition and processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 is a schematic flowchart of a preferred embodiment of a data acquisition method provided by the present invention;

[0030] Figure 2 is a schematic flowchart of another preferred embodiment of a data acquisition method provided by the present invention;

[0031] Figure 3 is a schematic structural diagram of a preferred embodiment of a data acquisition device provided by the present invention;

[0032] Figure 4 is a schematic structural diagram of another preferred embodiment of a data acquisition device provided by the present invention;

[0033] Figure 5 is a schematic structural diagram of a preferred embodiment of a terminal device provided by the present invention. Detailed implementation manners

[0034] The following will clearly and completely describe the technical solutions in the embodiments of the present invention with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without making creative efforts belong to the scope of protection of the present invention.

[0035] Please refer to Figure 1 , Figure 1 , which is a schematic flowchart of a preferred embodiment of a data acquisition method provided by the present invention. The data acquisition method is applied to a network node and includes:

[0036] S1. Receive a data acquisition configuration command sent by a data acquisition management node;

[0037] S2. Perform data acquisition according to the data acquisition configuration command, and generate serialized streaming data from the acquired data;

[0038] S3. Send the serialized streaming data to a data receiving end through a data processing intermediate platform.

[0039] Specifically, the data acquisition method provided by the embodiments of the present invention is applied to a network node, which refers to a network function or node where the data source is located, such as a network function entity. The network node is built-in with a data acquisition agent, which serves as a data acquisition and data generator. The network node receives a data acquisition configuration command sent by a data acquisition management node, performs data acquisition according to the data acquisition configuration command, and generates serialized streaming data from the acquired data. The generated serialized streaming data is sent to a data receiving end through a data processing intermediate platform. Among them, the data receiving end is the data consumer. In a network digital twin system, the data consumer can be an online network digital twin instance, which generates and maintains the operation of the network digital twin instance through real-time data, ensuring that the network twin instance is a real-time mirror of the real network. At the same time, it can also be an offline model training or inference functional unit, which is used for model training and the construction of an offline network twin instance. In a network intelligent scenario, the data consumer can be an inference and training instance of a network intelligent application, which supports the real-time generation of a network intelligent inference instance through real-time data acquisition to meet the real-time requirements of the running inference instance, and supports the real-time online training requirements of network intelligent training through real-time data acquisition.

[0040] It should be noted that in the specific implementation technologies of the generation, transmission processing, distribution, and consumption functions of streaming data, Kafka can be a specific implementation technology. That is, the data collection and sending of the data collection proxy end are implemented based on the Producer mechanism of Kafka, the streaming data intermediate platform is implemented based on the streaming data processing platform solution of Kafka, and the data acquisition function of the network data consumer is implemented based on the consumer mechanism of Kafka.

[0041] In the embodiment of the present invention, the data collection configuration command is issued in a programmable manner, which improves the flexibility of data collection; by supporting the method of streaming data collection, the key requirements of large data volume and real-time data collection are solved; by using the streaming data processing platform as an intermediate platform, it is realized that based on one-time data collection and processing, the data consumption requirements of multiple data receiving ends are met.

[0042] In another preferred embodiment, the data collection configuration command is encapsulated in a programmable framework, and the data collection configuration command includes the collection content, the collection frequency, and the collection time.

[0043] Specifically, the data collection configuration command sent by the data collection management node is encapsulated in a programmable framework, and the data collection configuration command at least includes the collection content, the collection frequency, and the collection time. Exemplarily, the programmable framework can use the YANG model, etc. for data content definition.

[0044] In yet another preferred embodiment, the data collection is performed according to the data collection configuration command, and the collected data is generated into serialized streaming data, which specifically includes:

[0045] Data collection is performed according to the collection content, the collection frequency, and the collection time included in the data collection configuration command;

[0046] The collected data is serialized to generate the serialized streaming data.

[0047] Specifically, in the embodiment of the present invention, data collection is performed according to the collection content, the collection frequency, and the collection time included in the data collection configuration command, and the collected data is serialized to generate serialized streaming data, which is sent and written to the data processing intermediate platform.

[0048] In yet another preferred embodiment, the sending of the serialized streaming data to the data receiving end through the data processing intermediate platform specifically includes:

[0049] Send the serialized streaming data through the serialized stream data interface and write it to the data processing intermediate platform. The data processing intermediate platform sends the serialized streaming data to the corresponding data receiving end through the deserialized stream data interface based on the subscription message sent by the data receiving end.

[0050] Specifically, the data processing intermediate platform is responsible for receiving, distributed storage, and data distribution of stream data or other types of data, meeting the requirements of large data volume and real-time processing. In the embodiment of the present invention, the serialized streaming data is sent and written to the data processing intermediate platform through the serialized stream data interface. The data processing intermediate platform sends the serialized streaming data to the corresponding data receiving end through the deserialized stream data interface based on the subscription message sent by the data receiving end.

[0051] The embodiment of the present invention meets the requirements of real-time collection and processing of network big data by introducing a stream data collection and processing mechanism, which can be used to build a network data twin system and can also be used for the design of data collection and processing functions in network intelligent functions. Compared with the existing network data collection scheme, it adds a data stream collection and processing scheme suitable for real-time and large data volume, and at the same time realizes the flexible configurability of the data collection mode through a programmable data collection configuration interface.

[0052] Please refer to Figure 2 , Figure 2 which is a schematic flowchart of another preferred embodiment of a data collection method provided by the present invention. The data collection method is applied to the data processing intermediate platform and includes:

[0053] S1, receiving the subscription message sent by the data receiving end;

[0054] S2, receiving the serialized streaming data sent by the network node and performing distributed storage;

[0055] S3, based on the subscription message sent by the data receiving end, sending the serialized streaming data to the corresponding data receiving end through the deserialized stream data interface.

[0056] Specifically, the data acquisition method provided by the embodiments of the present invention is applied to a data processing intermediate platform. As an intermediate platform, the data processing intermediate platform receives, distributes and stores streaming data or other types of data, and is an important component supporting large amounts of data and real-time processing. The data processing intermediate platform receives the subscription messages sent by the data receiving end, and receives the serialized streaming data sent by the network node, and performs distributed storage. Among them, the network node performs data acquisition according to the data acquisition configuration command sent by the data acquisition management node, and generates serialized streaming data from the acquired data. The data acquisition configuration command sent by the data acquisition management node is encapsulated by a programmable framework, and the data acquisition configuration command at least includes the acquisition content, the acquisition frequency, and the acquisition time. Exemplarily, the programmable framework can use the YANG model or the like to define the data content. The data processing intermediate platform sends the serialized streaming data to the corresponding data receiving end through the deserialized stream data interface based on the subscription message sent by the data receiving end. Among them, the data receiving end is the data consumer. In the network data twin system, the data consumer can be an online network digital twin instance, which generates and maintains the operation of the network digital twin instance through real-time data, and ensures that the network twin instance mirrors the real network in real time. It can also be an offline model training or inference functional unit for model training and the construction of an offline network twin instance. In the network intelligence scenario, the data consumer can be an inference and training instance of a network intelligence application, which supports the real-time generation of network intelligence inference instances through real-time data acquisition, meets the real-time requirements of the running inference instances, and supports the real-time online training requirements of network intelligence training through real-time data acquisition.

[0057] It should be noted that in the specific implementation technologies of the generation, transmission processing, distribution, and consumption functions of streaming data, Kafka can be a specific implementation technology. That is, the data acquisition and sending of the data acquisition proxy end are realized based on the Producer mechanism of Kafka, the streaming data intermediate platform is realized based on the streaming data processing platform solution of Kafka, and the data acquisition function of the network data consumer is realized based on the consumer mechanism of Kafka.

[0058] In the embodiments of the present invention, data acquisition configuration commands are issued in a programmable manner, which improves the flexibility of data acquisition; by supporting the method of streaming data acquisition, the key requirements of large data volume and real-time data acquisition are solved; by using the stream data processing platform as an intermediate platform, based on one-time data acquisition and processing, the data consumption requirements of multiple data receivers are met. The embodiments of the present invention introduce a stream data acquisition and processing mechanism to meet the real-time acquisition and processing requirements of network big data, which can be used to build a network data twin system and can also be used for the design of data acquisition and processing functions in network intelligent functions. Compared with the existing network data acquisition solutions, a data stream acquisition and processing solution suitable for real-time and large data volume is added, and at the same time, through the programmable data acquisition configuration interface, the flexible configurability of the data acquisition mode is realized.

[0059] Correspondingly, the present invention also provides a data acquisition device, which can implement all processes of the data acquisition method in the above embodiments.

[0060] Please refer to Figure 3 , Figure 3 FIG. is a schematic structural diagram of a preferred embodiment of a data acquisition device provided by the present invention. The data acquisition device includes:

[0061] A command receiving module 301, configured to receive a data acquisition configuration command sent by a data acquisition management node;

[0062] A data acquisition module 302, configured to perform data acquisition according to the data acquisition configuration command and generate serialized stream data from the acquired data;

[0063] A data sending module 303, configured to send the serialized stream data to a data receiver through a data processing intermediate platform.

[0064] Preferably, the data acquisition configuration command is encapsulated in a programmable framework, and the data acquisition configuration command includes acquisition content, acquisition frequency, and acquisition time.

[0065] Preferably, the data acquisition module 302 is specifically configured to:

[0066] Perform data acquisition according to the acquisition content, the acquisition frequency, and the acquisition time included in the data acquisition configuration command;

[0067] Perform serialization processing on the acquired data to generate the serialized stream data.

[0068] Preferably, the data sending module 303 is specifically configured to:

[0069] Send the serialized streaming data through the serialized stream data interface and write it to the data processing intermediate platform. The data processing intermediate platform sends the serialized streaming data to the corresponding data receiving end through the deserialized stream data interface based on the subscription message sent by the data receiving end.

[0070] Please refer to Figure 4 , Figure 4 which is a schematic structural diagram of another preferred embodiment of a data acquisition device provided by the present invention. The data acquisition device includes:

[0071] A subscription receiving module 401 for receiving subscription messages sent by a data receiving end;

[0072] A data receiving module 402 for receiving serialized streaming data sent by a network node and performing distributed storage;

[0073] A data sending module 403 for sending the serialized streaming data to the corresponding data receiving end through the deserialized stream data interface based on the subscription message sent by the data receiving end.

[0074] In specific implementation, the working principle, control process, and achieved technical effects of the data acquisition device provided in the embodiments of the present invention are the same as those of the data acquisition method in the above embodiments, and will not be elaborated here.

[0075] Please refer to Figure 5 , Figure 5 which is a schematic structural diagram of a preferred embodiment of a terminal device provided by the present invention. The terminal device includes a processor 501, a memory 502, and a computer program stored in the memory 502 and configured to be executed by the processor 501. When the processor 501 executes the computer program, it implements the data acquisition method described in any of the above embodiments.

[0076] Preferably, the computer program can be divided into one or more modules / units (such as computer program 1, computer program 2,...). The one or more modules / units are stored in the memory 502 and executed by the processor 501 to complete the present invention. The one or more modules / units can be a series of computer program instruction segments capable of performing specific functions, and these instruction segments are used to describe the execution process of the computer program in the terminal device.

[0077] The processor 501 may be a Central Processing Unit (CPU), or may also be other general-purpose processors, Digital Signal Processors (DSPs), Application Specific Integrated Circuits (ASICs), Field-Programmable Gate Arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor, or the processor 501 may also be any conventional processor. The processor 501 is the control center of the terminal device and connects various parts of the terminal device through various interfaces and lines.

[0078] The memory 502 mainly includes a program storage area and a data storage area. Among them, the program storage area can store an operating system, application programs required for at least one function, etc., and the data storage area can store relevant data, etc. In addition, the memory 502 may be a high-speed random access memory, or may also be a non-volatile memory, such as a plug-in hard disk, a Smart Media Card (SMC), a Secure Digital (SD) card, a Flash Card, etc., or the memory 502 may also be other volatile solid-state storage devices.

[0079] It should be noted that the above terminal device may include, but is not limited to, a processor and a memory. Those skilled in the art can understand that Figure 5 the structural schematic diagram is only an example of the above terminal device and does not limit the above terminal device. It may include more or fewer components than those shown in the figure, or combine certain components, or different components.

[0080] An embodiment of the present invention also provides a computer-readable storage medium. The computer-readable storage medium includes a stored computer program. Among them, when the computer program runs, it controls the device where the computer-readable storage medium is located to execute the data acquisition method described in any one of the above embodiments.

[0081] An embodiment of the present invention provides a data acquisition method, device, equipment and storage medium. The method includes receiving a data acquisition configuration command sent by a data acquisition management node; performing data acquisition according to the data acquisition configuration command, and generating serialized streaming data from the acquired data; and sending the serialized streaming data to a data receiving end through a data processing intermediate platform. By issuing a data acquisition configuration command in a programmable manner, the embodiment of the present invention improves the flexibility of data acquisition; by supporting the method of streaming data acquisition, it solves the key requirements of large data volume and real-time data acquisition; and by using a stream data processing platform as an intermediate platform, it realizes meeting the data consumption requirements of multiple data receiving ends based on one-time data acquisition and processing.

[0082] It should be noted that the system embodiments described above are only illustrative. The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed to multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment. In addition, in the attached drawings of the system embodiments provided by the present invention, the connection relationship between modules indicates that they have a communication connection, which can be specifically implemented as one or more communication buses or signal lines. Those of ordinary skill in the art can understand and implement it without creative efforts.

[0083] The above is the preferred embodiment of the present invention. It should be pointed out that for those of ordinary skill in the art, without departing from the principle of the present invention, several improvements and refinements can be made, and these improvements and refinements are also regarded as the protection scope of the present invention.

Claims

1. A data acquisition method, characterized in that, Applied to a network node, the method includes: Receiving a data collection configuration command sent by a data collection management node; Performing data collection according to the data collection configuration command, and generating serialized streaming data from the collected data; Sending the serialized streaming data to a data receiving end through a data processing intermediate platform.

2. The data acquisition method according to claim 1, wherein The data collection configuration command is encapsulated in a programmable framework, and the data collection configuration command includes collection content, collection frequency, and collection time.

3. The data acquisition method according to claim 2, wherein The performing data collection according to the data collection configuration command and generating serialized streaming data from the collected data specifically includes: Performing data collection according to the collection content, the collection frequency, and the collection time included in the data collection configuration command; Performing serialization processing on the collected data to generate the serialized streaming data.

4. The data acquisition method according to claim 3, wherein The sending the serialized streaming data to a data receiving end through a data processing intermediate platform specifically includes: Sending and writing the serialized streaming data to the data processing intermediate platform through a serialized stream data interface, and the data processing intermediate platform sends the serialized streaming data to the corresponding data receiving end through a deserialized stream data interface based on a subscription message sent by the data receiving end.

5. A data acquisition method, characterized in that, Applied to a data processing intermediate platform, the method includes: Receiving a subscription message sent by a data receiving end; Receiving serialized streaming data sent by a network node and performing distributed storage; Based on the subscription message sent by the data receiving end, sending the serialized streaming data to the corresponding data receiving end through a deserialized stream data interface.

6. A data acquisition device, characterized in that, Including: A command receiving module, configured to receive a data collection configuration command sent by a data collection management node; A data collection module, configured to perform data collection according to the data collection configuration command and generate serialized streaming data from the collected data; A data sending module, configured to send the serialized streaming data to a data receiving end through a data processing intermediate platform.

7. A data acquisition device, characterized in that, Including: A subscription receiving module, configured to receive a subscription message sent by a data receiving end; A data receiving module, configured to receive serialized streaming data sent by a network node and perform distributed storage; A data sending module, configured to send the serialized streaming data to the corresponding data receiving end through a deserialized stream data interface based on the subscription message sent by the data receiving end.

8. A terminal device, characterized in that, Including a processor and a memory, where a computer program is stored in the memory, and the computer program is configured to be executed by the processor. When the processor executes the computer program, the data collection method according to any one of claims 1 to 4 is implemented.

9. A terminal device, characterized in that, Including a processor and a memory, where a computer program is stored in the memory, and the computer program is configured to be executed by the processor. When the processor executes the computer program, the data collection method according to claim 5 is implemented.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program. When the device where the computer-readable storage medium is located executes the computer program, the data collection method according to any one of claims 1 to 5 is implemented.

Images