Data processing method and device oriented to operating system and data processing system

Abstract

The invention provides a data processing method and device oriented to an operating system, and relates to the technical field of information, in particular to the technical field of computer networks and the like. According to the specific implementation scheme, the method comprises the steps of obtaining a position index from a pre-configured production queue; determining a storage space allocated to the position index in advance and to-be-extracted data; copying the to-be-extracted data to a storage space; and mounting the position index into the pre-configured consumption queue, so that the flow extraction processing module obtains the position index from the consumption queue, and extracts the to-be-extracted data from the storage space based on the position index, thereby reducing the complexity of data extraction operation.

Description

Technical Field

[0001] The present disclosure relates to the field of information technology, specifically to technical fields such as cloud computing, and particularly to a data processing method and apparatus, a data processing system, an electronic device, a computer-readable storage medium, and a computer program product for an operating system. Background Art

[0002] The following are usually the solutions for gateway server traffic tracking and statistics: Solution 1, perform traffic mirroring on the uplink port of a high-performance gateway server, introduce the mirrored traffic into a dedicated traffic mirroring machine, save traffic files in the traffic mirroring machine, and perform traffic statistics; this solution requires the switch to reserve dedicated ports to copy the entire service traffic, has high requirements for network infrastructure, and also has high investment costs. Solution 2, monitor the traffic of the gateway server through the packet capture function, which is complex to operate and has an impact on the performance of the gateway service itself. Summary of the Invention

[0003] The present disclosure provides a data processing method and apparatus, a system, an electronic device, a computer-readable storage medium, and a computer program product for an operating system.

[0004] According to a first aspect, a data processing method for an operating system is provided. The method includes: obtaining a position index from a pre-configured production queue; determining a storage space and data to be extracted pre-allocated for the position index; copying the data to be extracted to the storage space; and mounting the position index to a pre-configured consumption queue, so that a traffic extraction processing module obtains the position index from the consumption queue and extracts the data to be extracted from the storage space based on the position index.

[0005] According to a second aspect, a data processing method for an operating system is provided. The method includes: obtaining a position index from a pre-configured consumption queue; determining a storage space pre-allocated for the position index; extracting the data to be extracted from the storage space and storing the data to be extracted in an extraction file; and putting the position index into a pre-configured production queue.

[0006] According to a third aspect, a data processing apparatus for an operating system is provided. The apparatus includes: an obtaining unit configured to obtain a position index from a pre-configured production queue; a determining unit configured to determine a storage space and data to be extracted pre-allocated for the position index; a copying unit configured to copy the data to be extracted to the storage space; and a mounting unit configured to mount the position index to a pre-configured consumption queue, so that a traffic extraction processing module obtains the position index from the consumption queue and extracts the data to be extracted from the storage space based on the position index.

[0007] According to a fourth aspect, there is provided a data processing apparatus for an operating system, the apparatus including: a consumption unit configured to obtain a position index from a pre-configured consumption queue; an allocation unit configured to determine a storage space pre-allocated for the position index; an extraction unit configured to extract data to be extracted from the storage space and store the data to be extracted in an extraction file; and a placement unit configured to place the position index into a pre-configured production queue.

[0008] According to a fifth aspect, there is provided a data processing system, the system including: at least one traffic loading processing module and a traffic extraction processing module; each traffic loading processing module is configured to obtain a position index from a pre-configured production queue; determine a storage space pre-allocated for the position index and data to be extracted; copy the data to be extracted to the storage space; mount the position index to a pre-configured consumption queue; the traffic extraction processing module is configured to obtain a position index from the consumption queue; determine a storage space pre-allocated for the position index; based on the position index, extract data to be extracted from the storage space and store the data to be extracted in an extraction file; and place the position index into the production queue.

[0009] According to a sixth aspect, there is provided an electronic device, the electronic device including: at least one processor; and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the method described in any implementation manner of the first aspect or the second aspect.

[0010] According to a seventh aspect, there is provided a non-transitory computer-readable storage medium storing computer instructions for causing a computer to execute the method described in any implementation manner of the first aspect or the second aspect.

[0011] According to an eighth aspect, there is provided a computer program product including a computer program, and the computer program implements the method described in any implementation manner of the first aspect or the second aspect when executed by a processor.

[0012] The data processing method and apparatus for an operating system provided by an embodiment of the present disclosure first obtain a position index from a pre-configured production queue; secondly, determine the storage space allocated for the position index and the data to be extracted; then, copy the data to be extracted into the storage space; and finally, mount the position index to a pre-configured consumption queue so that a traffic extraction processing module can obtain the position index from the consumption queue and extract the data to be extracted from the storage space based on the position index. By alternately providing the position index through the production queue and the consumption queue, the loading and extraction of the data to be extracted are realized, ensuring that the extraction of the data to be extracted does not interfere with the core forwarding function of the gateway, reducing the complexity of the data extraction operation, and improving the data processing performance.

[0013] It should be understood that the content described in this part is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it used to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] The drawings are used to better understand the solution and do not constitute a limitation to the present disclosure. Among them:

[0015] Figure 1 is a flowchart according to an embodiment of the data processing method for an operating system of the present disclosure;

[0016] Figure 2 is a flowchart according to another embodiment of the data processing method for an operating system of the present disclosure;

[0017] Figure 3 is a schematic structural diagram of the present disclosure for generating retrieval data;

[0018] Figure 4 is a schematic structural diagram according to an embodiment of the data processing apparatus for an operating system of the present disclosure;

[0019] Figure 5 is a schematic structural diagram according to another embodiment of the data processing apparatus for an operating system of the present disclosure;

[0020] Figure 6 is a schematic structural diagram according to an embodiment of the data processing system of the present disclosure;

[0021] Figure 7 is a block diagram of an electronic device for implementing the data processing method for an operating system of the embodiments of the present disclosure. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0022] The exemplary embodiments of the present disclosure will be described below in conjunction with the accompanying drawings. Various details of the embodiments of the present disclosure are included to facilitate understanding, and they should be considered merely exemplary. Therefore, those of ordinary skill in the art should recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, descriptions of well-known functions and structures are omitted below for clarity and conciseness.

[0023] There are the following two solutions for gateway server traffic tracing: Solution 1 is to perform traffic mirroring on the uplink port of a high-performance gateway server; Solution 2 is to monitor the traffic of the gateway server through packet capture.

[0024] For the first solution of mirroring the gateway server traffic, although it provides an effective monitoring means, its implementation process is accompanied by a series of challenges and potential deficiencies. First of all, this method requires the physical switch to reserve dedicated ports and sufficient bandwidth to copy the entire service traffic, which not only places high requirements on the network infrastructure but also requires expensive costs invested in high-end traffic mirroring devices. Secondly, the replication of traffic in the mirroring process will bring additional load to the network, which may in turn lead to excessive consumption of bandwidth resources and the emergence of performance bottlenecks. Especially in the case of large traffic or when the bandwidth of the mirroring port does not match that of the observation port, the observation port may not be able to efficiently forward all mirrored data packets, resulting in packet loss. In addition, in order to reduce interference with normal service traffic, careful design and optimization of the configuration are required, which not only increases the complexity of deployment but also may lead to greater difficulties in later maintenance and troubleshooting. Moreover, the mirrored traffic may not fully reflect the actual traffic received by the gateway.

[0025] For the second solution, the solution of directly enabling the packet capture function on a high-performance gateway server, the resulting operational complexity and potential impact on the gateway's own performance cannot be ignored. In a distributed system, it will be particularly cumbersome to monitor the traffic of multiple gateways, posing a significant challenge to management and statistics. In addition, due to the large volume of generated traffic files, the storage of these files will occupy valuable local disk space and may interfere with normal production operations.

[0026] In view of the deficiencies in the traditional technology, the present disclosure proposes a data processing method for an operating system, and through the processing logic of this method, the effect of not interfering with the core forwarding function of the gateway can be achieved. Figure 1 Flow 100 according to an embodiment of the data processing method for an operating system of the present disclosure is shown. The above-mentioned data processing method for an operating system includes the following steps:

[0027] Step 101, obtain a position index from a pre-configured production queue.

[0028] In this embodiment, a location index is a data space access object used to improve the access speed of spatial locations. A production queue is a queue that stores at least one location index in sequence and has a function similar to the first-in, first-out of a message queue. When there are multiple location indexes stored in the production queue, the multiple location indexes are sorted in the order of entry. Each location index is an access object of a data space, and the location index in the production queue is the index after extracting information from the corresponding storage space.

[0029] In this embodiment, when obtaining a location index from the production queue, the location index with the longest storage time in the production queue can be obtained; or the location index closest to the index exit in the production queue can be obtained. Among them, the production queue has two ports, one is the index entry for sending in the location index, and the other is the index exit for taking out the location index, that is, the index exit is the port where the production queue outputs the location index.

[0030] In this embodiment, the execution entity on which the data processing method for the operating system runs can be the traffic loading processing module in the gateway server. The traffic loading processing module can be the processor in the gateway server or the working thread in the gateway server. There can be one or multiple traffic loading processing modules. When there are multiple traffic loading processing modules, each traffic loading processing module has a corresponding production queue and consumption queue, and each traffic loading processing module can implement the data processing method for the operating system as Figure 1 shown.

[0031] In this embodiment, the traffic loading processing module can implement a high-performance gateway service. The high-performance gateway service works in the user state, uses the polling mode drive instead of interrupts, and bypasses the kernel protocol stack. In this architecture, each traffic loading processing module needs to be allocated two dedicated circular queues - a production queue and a consumption queue, so as to implement the information extraction task of managing the data stream.

[0032] Step 102, determine the storage space pre-allocated for the location index and the data to be extracted.

[0033] In this embodiment, each location index in the production queue corresponds to a storage space in the gateway server. Before executing the data processing method for the operating system of the present disclosure, this storage space is a space without storing any data. Through the data processing method for the operating system of the present disclosure, the data to be extracted can be stored in the storage space corresponding to this location index.

[0034] In this embodiment, the execution entity on which the data processing method for the operating system runs can process at least one instance in the gateway server in real time. Each instance is a connection, and this connection can be a long connection or a short connection. The data to be extracted is the processed data of the execution entity. For example, the data to be extracted is the request data in the instance.

[0035] In this embodiment, each location index corresponds to a storage space, and this corresponding relationship can be stored in a correspondence table. After obtaining the location index, by querying the correspondence table, the storage space pre-allocated for the location index can be obtained.

[0036] Step 103: Copy the data to be extracted to the storage space.

[0037] In this embodiment, there are multiple storage spaces for storing data in the gateway server. The size of the storage space can be set based on development requirements. Each location index corresponds to a storage space, and the data to be extracted can be directly stored in the storage space obtained through the location index.

[0038] Step 104: Mount the location index to a pre-configured consumption queue so that the traffic extraction processing module can obtain the location index from the consumption queue and extract the data to be extracted from the storage space based on the location index.

[0039] In this embodiment, the consumption queue is a queue that stores at least one location index in sequence and has a function similar to the first-in, first-out of a message queue. When there are multiple location indexes stored in the consumption queue, the multiple location indexes are sorted in the order of entry. The location index in the consumption queue is the index of the storage space containing information.

[0040] In this embodiment, the traffic extraction processing module can be a processor in the gateway server or an information extraction processing thread constructed in the gateway server for data extraction. There is only one traffic extraction processing module. When there are multiple traffic loading processing modules, each traffic loading processing module has a corresponding production queue and consumption queue, and the traffic extraction processing module corresponds to the production queues and consumption queues of all traffic loading processing modules.

[0041] The data processing method for an operating system provided by an embodiment of the present disclosure first obtains a position index from a pre-configured production queue; then determines the storage space pre-allocated for the position index and the data to be extracted; then copies the data to be extracted into the storage space; and finally mounts the position index to a pre-configured consumption queue, so that a traffic extraction processing module can obtain the position index from the consumption queue and extract the data to be extracted from the storage space based on the position index. Thus, by alternately providing the position index through the production queue and the consumption queue, the loading and extraction of the data to be extracted are realized, ensuring that the extraction of the data to be extracted does not interfere with the core forwarding function of the gateway, reducing the complexity of the data extraction operation, and improving the data processing performance.

[0042] In some embodiments of the present disclosure, the obtaining of the position index from the pre-configured production queue includes: obtaining in real time the global switch signal of a pre-set global switch; detecting whether the global switch signal indicates that the global data stream extraction function is turned on; and in response to detecting that the global switch signal indicates that the global data stream extraction function is turned on, obtaining the position index from the pre-configured production queue.

[0043] In this alternative implementation, the gateway manages multiple types of traffic, and the global switch is a switch for global management. By pre-setting the global switch signal of the global switch for the gateway, it can be determined whether the global data extraction function of the gateway is turned on or off.

[0044] In this alternative implementation, the global switch signal is the information shown after the global switch is set. For example, the global switch signal can be 1 or 0, where 1 indicates that the global data stream extraction function is turned on, and 0 indicates that the global data stream extraction function is not turned on.

[0045] The method for obtaining the position index provided by this alternative implementation obtains in real time the global switch signal of a pre-set global switch; detects whether the global switch signal indicates that the global data stream extraction function is turned on; and in response to detecting that the global switch signal indicates that the global data stream extraction function is turned on, obtains the position index from the pre-configured production queue. Obtaining the position index through the pre-set global switch improves the reliability and security of obtaining the position index.

[0046] Optionally, the global switch signal of the global switch can also be configured by a switch operator or a user through a user interface. Specifically, the obtaining of the position index from the pre-configured production queue includes: obtaining the global switch signal of the global switch from the user interface; detecting whether the global switch signal indicates that the global data stream extraction function is turned on; and in response to detecting that the global switch signal indicates that the global data stream extraction function is turned on, obtaining the position index from the pre-configured production queue.

[0047] In some embodiments of the present disclosure, the obtaining of the position index from the pre-configured production queue further includes: obtaining in real time the global switch signal of the pre-set global switch; detecting whether the global switch signal indicates that the global data stream extraction function is turned on; in response to detecting that the global switch signal indicates that the global data stream extraction function is turned on, obtaining in real time the instance switch signal of the instance switch set for each instance; detecting whether the instance switch signal indicates that the current instance data extraction function is turned on; and in response to detecting that the instance switch signal indicates that the current instance data extraction function is turned on, obtaining the position index from the pre-configured production queue.

[0048] In this alternative implementation, the gateway manages multiple instances, each instance representing a network connection. The instance switch is the switch for managing the data of the instance. By pre-setting the instance switch signal for the instance switch, it can be determined whether the instance data extraction function of the gateway is turned on or off.

[0049] In this alternative implementation, the instance switch signal is the information shown after the instance switch is set. For example, the instance switch signal can be 1 or 0, where 1 indicates that the instance data extraction function is turned on and the data corresponding to the instance switch can be extracted; 0 indicates that the instance data extraction function is not turned on and extraction of the data corresponding to the instance switch is not allowed.

[0050] The method for obtaining the position index provided by this alternative implementation obtains in real time the global switch signal of the pre-set global switch; detects whether the global switch signal indicates that the global data stream extraction function is turned on; in response to detecting that the global switch signal indicates that the global data stream extraction function is turned on, obtains in real time the instance switch signal of the instance switch set for each instance; detects whether the instance switch signal indicates that the current instance data extraction function is turned on; and in response to detecting that the instance switch signal indicates that the current instance data extraction function is turned on, obtains the position index from the pre-configured production queue. By setting the coarse-grained global switch and the fine-grained instance switch, it is determined that the data stream of a specific instance can be extracted from the gateway server in a targeted manner. At this time, the position index is obtained, which improves the accuracy and reliability of the extraction of the data to be extracted.

[0051] Optionally, the global switch signal of the global switch and the instance switch information of the instance switch can be configured by the switch operator or user through the user interface. Specifically, the obtaining of the position index from the pre-configured production queue includes: obtaining the global switch signal of the global switch from the user interface; detecting whether the global switch signal indicates that the global data stream extraction function is turned on; in response to detecting that the global switch signal indicates that the global data stream extraction function is turned on, obtaining the instance switch signal of the instance switch from the user interface; detecting whether the instance switch signal indicates that the current instance data extraction function is turned on; in response to detecting that the instance switch signal indicates that the current instance data extraction function is turned on, obtaining the position index from the pre-configured production queue. Thus, traffic tracking can be flexibly performed at the instance level. The user can monitor specific instances according to their own needs, and then achieve precise traffic management and analysis, thereby providing the user with a highly customized service experience.

[0052] In some embodiments of the present disclosure, the determining of the storage space pre-allocated for the position index and the data to be extracted includes: determining the storage space according to the position index; determining the current instance based on the opened instance switch; extracting the data to be extracted from the current instance.

[0053] In this embodiment, the opened instance switch refers to the instance switch for which the instance data extraction function is turned on. The instance switch can be one or more. Since each instance switch corresponds to an instance of the gateway, when an opened instance switch is detected, the instance corresponding to the opened instance switch can be obtained by means such as table lookup, that is, the current instance.

[0054] The method for determining the storage space and the data to be extracted provided by this optional implementation manner determines the storage space according to the position index; determines the current instance based on the opened instance switch; extracts the data to be extracted from the current instance, and determines the data to be extracted through the opened instance switch, effectively locating the data to be extracted and improving the reliability of obtaining the data to be extracted.

[0055] The present disclosure also proposes another data processing method for an operating system. Figure 2 Fig. 200 shows a flow of another embodiment of the data processing method for an operating system according to the present disclosure. The data processing method for an operating system includes the following steps:

[0056] Step 201, obtaining a position index from a pre-configured consumption queue.

[0057] In this embodiment, the location index is a data space access object used to improve the access speed of spatial locations. The consumption queue is a queue that stores at least one location index in sequence and has a function similar to the first-in, first-out of a message queue. When there are multiple location indexes stored in the consumption queue, the multiple location indexes are sorted in the order of entry. The location index in the consumption queue is the index of the corresponding storage space containing information.

[0058] In this embodiment, there can be one or multiple consumption queues. When there are multiple consumption queues, the consumption queues can be the queues corresponding to multiple traffic loading processing modules. Through the current data extraction request, the corresponding traffic loading processing module can be distinguished, and the consumption queue corresponding to the traffic loading processing module can be determined.

[0059] Optionally, in order to distinguish different traffic loading processing modules for loading data, the location index can carry the identifier of the traffic loading processing module. Through this identifier, it can be determined which traffic loading processing module each location index belongs to.

[0060] Step 202: Determine the storage space pre-allocated for the location index.

[0061] In this embodiment, each location index in the consumption queue corresponds to a storage space. This corresponding relationship can be stored in a correspondence table. After obtaining the location index, by querying the correspondence table, the storage space pre-allocated for the location index can be obtained.

[0062] Step 203: Extract the data to be extracted from the storage space and store the data to be extracted in an extraction file.

[0063] In this embodiment, the data to be extracted in the storage space can be directly taken out, and the storage space can be assigned zero, or the data to be extracted in the storage space can be copied, and the storage space can be cleared.

[0064] Step 204: Put the location index into a pre-configured production queue.

[0065] In this embodiment, the execution entity on which the data processing method for the operating system runs is the traffic extraction processing module. The traffic extraction processing module can be an information extraction processing thread. The information extraction processing thread takes out a location index from the consumption queue, extracts the data to be extracted corresponding to the location index, and releases the location index after processing and returns it to the production queue. This lock-free operation mechanism significantly reduces performance loss and ensures the efficiency of data processing.

[0066] In this embodiment, there can be one or multiple production queues. When there are multiple production queues, they can be the queues corresponding to multiple traffic loading and processing modules. Through the position index representation form, such as having the identifier of the corresponding traffic loading and processing module in the position index, or by querying the queue module correspondence table, the corresponding traffic loading and processing module can be distinguished, and the production queue corresponding to the traffic loading and processing module can be determined. Among them, the queue module correspondence table is used to pre-set the correspondence between each production queue and the traffic loading and processing module. Generally, one traffic loading and processing module corresponds to one production queue.

[0067] The data processing method for the operating system provided in this embodiment performs gateway traffic tracking in an efficient manner, collects detailed information statistics, and at the same time ensures that it will not have any negative impact on the core forwarding function of the gateway. This means that deep data can be obtained without sacrificing performance.

[0068] For the data processing method for the operating system provided in the embodiments of the present disclosure, first, obtain the position index from the pre-configured production queue; secondly, determine the storage space pre-allocated for the position index and the data to be extracted; then, copy the data to be extracted to the storage space; finally, mount the position index to the pre-configured consumption queue, so that the traffic extraction processing module can obtain the position index from the consumption queue and extract the data to be extracted from the storage space based on the position index. Thus, by alternately providing the position index through the production queue and the consumption queue, the loading and extraction of the data to be extracted are realized, ensuring that the extraction of the data to be extracted will not interfere with the core forwarding function of the gateway, reducing the complexity of the data extraction operation, and improving the data processing performance.

[0069] In some embodiments of the present disclosure, the above method further includes: monitoring in real time the disk space occupied by the extracted file; and in response to the disk space being greater than or equal to the space threshold, performing data cleaning on the extracted file.

[0070] In this embodiment, by setting a scheduled task in Crontab, the system will automatically monitor the disk space occupied by the data stream information file. Crontab is a tool for scheduling tasks on the operating system. It allows users to create and manage scheduled tasks so that commands or scripts can be automatically run at specific time intervals or time points.

[0071] In this embodiment, the extraction file includes multiple information files. The above data cleaning of the extraction file includes clearing the information files in the extraction file. By means of a preset space threshold of disk occupancy, once it is detected that the overall disk space occupied by the files exceeds this space threshold, the execution entity on which the data processing method for the operating system runs will automatically perform data cleaning and preferentially delete the information file with the longest storage time, thus effectively avoiding the problem of insufficient disk space.

[0072] The data processing method for the operating system provided in this embodiment monitors the disk space occupied by the extraction file in real time; in response to the disk space being greater than or equal to the space threshold, it performs data cleaning on the extraction file, effectively avoiding insufficient disk space.

[0073] In some embodiments of the present disclosure, the above method further includes: reading file data from the extraction file; obtaining retrieval data based on the file data; processing the retrieval data according to predefined requirement information to obtain recombinant data; and sending the recombinant data to the aggregation message queue of the data display platform.

[0074] In this embodiment, the retrieval data is the data obtained by retrieving the file data; the predefined requirement information is the data recombination requirement set by the developer or user.

[0075] As Figure 3 shown, the above data retrieval and recombination are implemented by the data stream information push service in the gateway server. The data stream information push service in the gateway server plays a core role of the middleware. This data stream information push service is responsible for reading file data from the stored extraction file and performing precise retrieval on the file data through the built-in query mechanism. The gateway service in the gateway server is responsible for data forwarding of each instance and the implementation of the data processing method for the operating system as shown above. Figure 2 shown.

[0076] In this embodiment, once the relevant data is retrieved, the data stream information push service will reassemble the message content and order according to the predefined requirement information to obtain recombinant data. Through this recombinant data, the customization and personalization of information can be ensured. Then, these screened and recombinant data are sent to the aggregation message queue dedicated to data stream information aggregation. This process is not only efficient but also can reduce unnecessary data transmission and optimize the overall performance.

[0077] In this embodiment, the client can subscribe to the data stream fields it needs on this basis. Once the subscription is successful, the client will be able to extract relevant data from the aggregated message queue in real time. Then, this data is pushed to the display module of the data display platform to provide the customer with an intuitive data stream analysis and monitoring interface. This solution realizes the real-time observation of gateway traffic. This immediate monitoring enables users to quickly respond to and handle potential traffic problems, thereby improving the stability of the system and user satisfaction.

[0078] The data processing method for the operating system provided in this embodiment reads file data from the extraction file; based on the file data, retrieves data is obtained; according to the predefined requirement information, the retrieved data is processed to obtain restructured data; the restructured data is sent to the aggregated message queue of the data display platform, ensuring the immediacy and reliability of the information. At the same time, by sending the data to the aggregated message queue, a highly customized data display service can be provided for the data display platform to meet the specific needs of different customers.

[0079] In some alternative implementation manners of the present disclosure, the obtaining the retrieved data based on the file data includes: retrieving the file data based on a preset keyword to obtain the retrieved data.

[0080] In this alternative implementation manner, the preset keyword can be an instance identifier, an instance name, etc. Retrieving the file data using the keyword can enable the retrieved data to be retrieved precisely according to the requirements of the keyword to obtain the retrieved data.

[0081] The method for obtaining the retrieved data provided in this alternative implementation manner retrieves the file data based on a preset keyword to obtain the retrieved data, improving the reliability of obtaining the retrieved data.

[0082] In the traditional technology, the management and monitoring of the traffic of the gateway server, especially the traffic management and statistics of a single service, are very cumbersome. It is necessary to collect and store the traffic on the gateway, which requires a large amount of disk space for storage, and the processing of the traffic occupies a large amount of resources, which is very likely to affect the normal business.

[0083] In some alternative implementation manners of the present disclosure, the storing the data to be extracted into the extraction file includes: storing the data to be extracted in binary format into the extraction file.

[0084] In this alternative implementation, the binary format storage not only makes the central processing unit more efficient and takes up less space, but is also more friendly to data transmission to the remote server, helping to save valuable network bandwidth. This refined information storage and transmission strategy ensures that the gateway service is both efficient and resource-saving when processing high-speed data streams. Through the binary storage format, key information can be highlighted; when storing and converting data, the storage space can be maximally saved and the occupation of other resources can be reduced. The processing of data streams is more refined, with a faster processing speed and less resource occupation, improving the performance of the gateway server.

[0085] In this embodiment, since the data stream information file is stored in the binary format, in order to facilitate the viewing and analysis by local operation and maintenance personnel, the execution entity on which the data processing method for the operating system runs can provide a dedicated conversion tool. Through this conversion tool, the binary file can be converted into a text file, enabling the operation and maintenance personnel to intuitively view and understand the real information in the file. This not only improves the efficiency of local maintenance, but also ensures the readability and accessibility of the data. Through such a file management and conversion mechanism, the efficient management of data stream information and the usability for operation and maintenance personnel are ensured.

[0086] The method for storing data to be extracted provided by this alternative implementation stores the data to be extracted in the binary format into the extraction file, improving the efficient management of the data to be extracted.

[0087] Further referring to Figure 4 , as an implementation of the methods shown in the above figures, the present disclosure provides an embodiment of a data processing device for an operating system. This device embodiment corresponds to the method embodiment shown in Figure 1 , and this device can be specifically used in various electronic devices.

[0088] As shown in Figure 4 , the data processing device 400 for an operating system provided in this embodiment includes: an acquisition unit 401, a determination unit 402, a copying unit 403, and a mounting unit 404. Among them, the above acquisition unit 401 can be configured to acquire a location index from a pre-configured production queue. The above determination unit 402 can be configured to determine the storage space and the data to be extracted pre-allocated for the location index. The above copying unit 403 can be configured to copy the data to be extracted into the storage space. The above mounting unit 404 can be configured to mount the location index to a pre-configured consumption queue, so that the traffic extraction processing module can acquire the location index from the consumption queue and extract the data to be extracted from the storage space based on the location index.

[0089] In this embodiment, in the data processing device 400 for an operating system: For the specific processing of the acquisition unit 401, the determination unit 402, the copying unit 403, and the mounting unit 404 and the technical effects brought by them, reference can be respectively made to Figure 1 the relevant descriptions of steps 101, 102, 103, and 104 in the corresponding embodiment, which will not be elaborated here.

[0090] In some embodiments of this embodiment, the above-mentioned acquisition unit 401 is configured to: obtain the global switch signal of the pre-set global switch in real time; detect whether the global switch signal indicates that the global data stream extraction function is turned on; in response to detecting that the global switch signal indicates that the global data stream extraction function is turned on, obtain the position index from the pre-configured production queue.

[0091] In some embodiments of this embodiment, the above-mentioned acquisition unit 401 is further configured to: obtain the global switch signal of the pre-set global switch in real time; detect whether the global switch signal indicates that the global data stream extraction function is turned on; in response to detecting that the global switch signal indicates that the global data stream extraction function is turned on, obtain the instance switch signal of the instance switch set for each instance in real time; detect whether the instance switch signal indicates that the current instance data extraction function is turned on; in response to detecting that the instance switch signal indicates that the current instance data extraction function is turned on, obtain the position index from the pre-configured production queue.

[0092] In some embodiments of this embodiment, the above-mentioned determination unit 402 is configured to: determine the storage space according to the position index; determine the current instance based on the turned-on instance switch; extract the data to be extracted from the current instance.

[0093] The data processing device for an operating system provided by the embodiments of the present disclosure, first, the acquisition unit 401 obtains the position index from the pre-configured production queue; second, the determination unit 402 determines the storage space and the data to be extracted pre-allocated for the position index; then, the copying unit 403 copies the data to be extracted to the storage space; finally, the mounting unit 404 mounts the position index to the pre-configured consumption queue, so that the traffic extraction processing module obtains the position index from the consumption queue and extracts the data to be extracted from the storage space based on the position index. Thus, by alternately providing the position index by the production queue and the consumption queue, the loading and extraction of the data to be extracted are realized, ensuring that the extraction of the data to be extracted does not interfere with the core forwarding function of the gateway, reducing the complexity of the data extraction operation, and improving the data processing performance.

[0094] The data processing device for an operating system provided by the embodiments of the present disclosure realizes the extraction, copying, and distribution of data through a series of ordered operations, facilitating the subsequent processing of data. Especially in scenarios where a large amount of data needs to be processed or the data needs to be shared and processed by multiple modules, it helps improve the efficiency and flexibility of data processing.

[0095] Further referring to Figure 5 , as an implementation of the methods shown in the above figures, the present disclosure provides an embodiment of a data processing device for an operating system. This device embodiment corresponds to Figure 2 the method embodiment shown, and this device can be specifically used in various electronic devices.

[0096] As shown in Figure 5 , the data processing device 500 for an operating system provided in this embodiment includes: a consumption unit 501, an allocation unit 502, an extraction unit 503, and a putting unit 504. Among them, the above consumption unit 501 can be configured to obtain a position index from a pre-configured consumption queue. The above allocation unit 502 can be configured to determine the storage space pre-allocated for the position index. The above extraction unit 503 can be configured to extract the data to be extracted from the storage space and store the data to be extracted in an extraction file. The above putting unit 504 can be configured to put the position index into a pre-configured production queue.

[0097] In this embodiment, in the data processing device 500 for an operating system: the specific processing of the consumption unit 501, the allocation unit 502, the extraction unit 503, and the putting unit 504 and the technical effects brought by them can respectively refer to Figure 2 the relevant descriptions of step 201, step 202, step 203, and step 204 in the corresponding embodiments, which will not be elaborated here.

[0098] In some alternative implementation manners of this embodiment, the above device 500 further includes: a file processing unit (not shown in the figure), and the above file processing unit is configured to monitor in real time the disk space occupied by the extraction file; in response to the disk space being greater than or equal to a space threshold, perform data cleaning on the extraction file.

[0099] In some alternative implementation manners of this embodiment, the above device 500 further includes: a recombination unit (not shown in the figure), and the above recombination unit is configured to read file data from the extraction file; based on the file data, obtain retrieval data; process the retrieval data according to pre-defined requirement information to obtain recombination data; and send the recombination data to the aggregation message queue of the data display platform.

[0100] In some alternative implementation manners of this embodiment, the above recombination unit is further configured to: retrieve file data based on a preset keyword to obtain retrieved data.

[0101] In some alternative implementation manners of this embodiment, the above extraction unit is configured to: store the data to be extracted in a binary format into an extraction file.

[0102] For the data processing apparatus for an operating system provided by the embodiments of the present disclosure, first, the consumption unit 501 obtains a position index from a pre-configured consumption queue; second, the allocation unit 502 determines a storage space pre-allocated for the position index; then, the extraction unit 503 extracts the data to be extracted from the storage space and stores the data to be extracted in an extraction file; finally, the putting unit 504 puts the position index into a pre-configured production queue. Thus, by alternately providing the position index through the production queue and the consumption queue, the loading and extraction of the data to be extracted are realized, ensuring that the extraction of the data to be extracted does not interfere with the core forwarding function of the gateway, reducing the complexity of the data extraction operation, and improving the data processing performance.

[0103] Further referring to Figure 6 , as an implementation of the methods shown in the above figures, the present disclosure provides an embodiment of a data processing system, and this system embodiment corresponds to Figure 1 the method embodiment shown, and this system can be specifically implemented by a variety of electronic devices.

[0104] As Figure 6 shown, the data processing system 600 provided in this embodiment includes: at least one traffic loading processing module 601 and a traffic extraction processing module 602. Each traffic loading processing module 601 is used to obtain a position index from a pre-configured production queue; determine a storage space pre-allocated for the position index and the data to be extracted; copy the data to be extracted into the storage space; and mount the position index to a pre-configured consumption queue. The traffic extraction processing module 602 is used to obtain a position index from the consumption queue; determine a storage space pre-allocated for this position index; based on this position index, extract the data to be extracted from the storage space and store the data to be extracted in an extraction file; and put the position index into the production queue.

[0105] In this embodiment, there may be one or more traffic loading processing modules 601, and there is only one traffic extraction processing module. As Figure 6 shown, each traffic loading processing module has a corresponding production queue and consumption queue, and all traffic loading processing modules correspond to the traffic extraction processing module. The traffic extraction processing module 602 can obtain data from the consumption queues of each traffic loading processing module 601 and store the position index into the production queue of this traffic loading processing module 601.

[0106] Optionally, when there are multiple traffic loading processing modules, the multiple traffic loading processing modules can share the same production queue and a consumption queue. At this time, the position indexes in the production queue and the consumption queue also have identifiers corresponding to each traffic loading processing module, so as to facilitate each traffic loading processing module to extract its own position index from the production queue and store data in the storage space based on the position index; the traffic extraction processing module can retrieve data from the consumption queue and store the position index in the production queue

[0107] In this embodiment, for the specific processing of the traffic loading processing module 601 and the technical effects brought thereby, reference can be made respectively to Figure 1 the relevant descriptions of steps 101, step 102, step 103, and step 104 in the corresponding embodiments, which will not be elaborated herein

[0108] In this embodiment, for the specific processing of the traffic extraction processing module 602 and the technical effects brought thereby, reference can be made respectively to Figure 2 the relevant descriptions of steps 201, step 202, step 203, and step 204 in the corresponding embodiments, which will not be elaborated herein

[0109] In some embodiments of the present disclosure, the above system 600 further includes: a data display platform (not shown in the figure). The traffic extraction processing module 602 reads file data from the extraction file, obtains retrieval data based on the file data, processes the retrieval data according to predefined requirement information to obtain restructured data, and sends the restructured data to the aggregation message queue of the data display platform

[0110] In the data processing system provided in this embodiment, when the traffic loading processing module 601 needs to load data to be extracted, it obtains the position index of the production queue. After the data to be extracted is stored in the storage space, it loads the position index into the consumption queue. When the traffic extraction processing module 602 extracts the data to be extracted, it stores the data to be extracted in the extraction file, supporting data backtracking, retry, or further analysis. Putting the position index back into the production queue forms a closed loop for the entire workflow, having no impact on other connection services of the gateway server, improving the efficiency and flexibility of data processing, and enabling the data processing system to dynamically adjust the data processing data and scale according to requirements

[0111] In the technical solution of the present disclosure, the collection, storage, use, processing, transmission, provision, and disclosure of the user's personal information involved all comply with the provisions of relevant laws and regulations and do not violate public order and good customs

[0112] According to the embodiments of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium, and a computer program product

[0113] Figure 7 FIG. shows a schematic block diagram of an exemplary electronic device 700 that can be used to implement embodiments of the present disclosure. The electronic device is intended to represent various forms of digital computers, such as, for example, laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as, for example, personal digital processors, cellular telephones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely exemplary and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0114] As Figure 7 shown, the device 700 includes a computing unit 701 that can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 702 or a computer program loaded from a storage unit 708 into a random access memory (RAM) 703. In the RAM 703, various programs and data required for the operation of the device 700 can also be stored. The computing unit 701, the ROM 702, and the RAM 703 are connected to each other via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0115] A plurality of components in the device 700 are connected to the I / O interface 705, including: an input unit 706, such as, for example, a keyboard, a mouse, etc.; an output unit 707, such as, for example, various types of displays, speakers, etc.; a storage unit 708, such as, for example, a magnetic disk, an optical disk, etc.; and a communication unit 709, such as, for example, a network card, a modem, a wireless communication transceiver, etc. The communication unit 709 allows the device 700 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks.

[0116] The computing unit 701 can be various general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 701 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 701 executes the various methods and processes described above, such as the data processing method for the operating system. For example, in some embodiments, the data processing method for the operating system can be implemented as a computer software program that is tangibly contained in a machine-readable medium, such as the storage unit 708. In some embodiments, part or all of the computer program can be loaded and / or installed onto the device 700 via the ROM 702 and / or the communication unit 709. When the computer program is loaded into the RAM 703 and executed by the computing unit 701, one or more steps of the data processing method for the operating system described above can be executed. Alternatively, in other embodiments, the computing unit 701 can be configured to execute the data processing method for the operating system by any other suitable means (e.g., by means of firmware).

[0117] The various embodiments of the systems and techniques described above in this document can be implemented in digital electronic circuitry, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a dedicated or general-purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit the data and instructions to the storage system, the at least one input device, and the at least one output device.

[0118] The program code for implementing the methods of the present disclosure can be written in any combination of one or more programming languages. This program code can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device for the operating system, such that when the program code is executed by the processor or controller, the functions / operations specified in the flowcharts and / or block diagrams are implemented. The program code can be executed entirely on the machine, partially on the machine, as a stand-alone software package partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0119] In the context of this disclosure, a machine-readable medium can be a tangible medium that can contain or store a program for use by or in connection with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of a machine-readable storage medium would include an electrical connection based on one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0120] To provide for interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or a trackball) by which the user can provide input to the computer. Other kinds of devices can also be used to provide for interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic, speech, or tactile input).

[0121] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an information server), or a computing system that includes front-end components (e.g., a user computer having a graphical user interface or a web browser through which the user can interact with an implementation of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of a communication network include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0122] A computer system can include a client and a server. The client and the server are generally remote from each other and typically interact through a communication network. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship to each other. The server can be a cloud server, a server of a distributed system, or a server incorporating a blockchain.

[0123] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in a different order, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and no limitations are imposed herein.

[0124] The above specific embodiments do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub - combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure shall be included within the scope of protection of this disclosure.

Claims

1. A data processing method for an operating system, the method comprising: Get a position index from a pre-configured production queue; Determining the storage space pre-allocated for the location index and the data to be extracted; Copying the data to be extracted to the storage space; The location index is mounted to a pre-configured consumption queue, so that the traffic extraction processing module obtains the location index from the consumption queue, and extracts the data to be extracted from the storage space based on the location index.

2. According to the method of claim 1, obtaining a location index from a preconfigured production queue comprises: Acquire the global switch signal of the pre-set global switch in real time; Detecting whether the global switch signal indicates that the global data stream extraction function has been turned on; In response to detecting that the global switch signal indicates that the global data stream extraction function has been turned on, a position index is obtained from a pre-configured production queue.

3. The method according to claim 1, wherein obtaining a location index from a preconfigured production queue further comprises: Acquire the global switch signal of the pre-set global switch in real time; Detecting whether the global switch signal indicates that the global data stream extraction function has been turned on; In response to detecting that the global switch signal indicates that the global data stream extraction function has been turned on, acquiring in real time an instance switch signal of an instance switch set for each instance; Detecting whether the instance switch signal indicates that the current instance data extraction function has been turned on; In response to detecting that the instance switch signal indicates that the data extraction function of the current instance has been turned on, a location index is obtained from a pre-configured production queue.

4. The method according to claim 1, wherein: The determining of the storage space pre-allocated for the location index and the data to be extracted comprises: Determine the storage space according to the location index; Determine the current instance based on the instance switch that is turned on; Extract the data to be extracted from the current instance.

5. A data processing method for an operating system, the method comprising: Get the position index from the pre-configured consumer queue; Determining a storage space pre-allocated for the location index; Extracting the data to be extracted from the storage space, and storing the data to be extracted in an extraction file; The location index is placed into a pre-configured production queue.

6. The method according to claim 5, further comprising: Real-time monitoring of the disk space occupied by the extracted files; In response to the disk space being greater than or equal to a space threshold, data cleaning is performed on the extracted file.

7. The method according to claim 5, further comprising: Reading file data from the extraction file; Based on the file data, obtaining search data; According to predefined requirement information, the retrieved data is processed to obtain reorganized data; The reorganized data is sent to the aggregation message queue of the data display platform.

8. The method according to claim 7, wherein: The obtaining of the retrieval data based on the file data comprises: Based on the preset keywords, the file data is searched to obtain search data.

9. The method according to any one of claims 5 to 8, wherein: The storing the to-be-extracted data into an extraction file comprises: The data to be extracted is stored in an extraction file in binary format.

10. A data processing device for an operating system, the device comprising: an acquisition unit configured to acquire a position index from a preconfigured production queue; a determining unit, configured to determine the storage space pre-allocated for the location index and the data to be extracted; a copying unit, configured to copy the to-be-extracted data to the storage space; The mounting unit is configured to mount the location index to a preconfigured consumption queue, so that the traffic extraction processing module obtains the location index from the consumption queue and extracts the data to be extracted from the storage space based on the location index.

11. A data processing device for an operating system, the device comprising: The consumer unit is configured to obtain the position index from the pre-configured consumer queue; an allocation unit, configured to determine a storage space pre-allocated for the location index; An extraction unit is configured to extract the data to be extracted from the storage space and store the data to be extracted into an extraction file; The placing unit is configured to place the location index into a pre-configured production queue.

12. A data processing system, the system comprising: at least one traffic loading processing module and one traffic extraction processing module; Each traffic loading processing module is used to obtain a position index from a pre-configured production queue; Determine the storage space and the data to be extracted pre-allocated for the location index; copy the data to be extracted to the storage space; mount the location index to the pre-configured consumption queue; the traffic extraction processing module is used to obtain the location index from the consumption queue of each traffic loading processing module; determine the storage space pre-allocated for the location index; Based on the location index, the data to be extracted is extracted from the storage space, and the data to be extracted is stored in an extraction file; and the location index is placed in the production queue of each traffic loading processing module.

13. An electronic device, characterized in that: include: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 9.

14. A non-transitory computer-readable storage medium storing computer instructions, characterized in that: The computer instructions are used to cause the computer to execute the method according to any one of claims 1 to 9.

15. A computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the computer program implements the method according to any one of claims 1 to 9.

Images