A data processing method, a collection device, a collection system and a medium

Abstract

The present application discloses a data processing method, an acquisition module, an acquisition system and a medium, the method comprising: determining first address information; wherein the first address information comprises address information of a first storage space; the first storage space comprises a storage space in a shared storage space where the first acquisition module can perform data storage operations on data to be processed; control information is stored in the first storage space; based on the first address information, the control information is determined; based on the control information, the data to be processed acquired by the first acquisition module is processed to obtain target data; based on the first address information and the target data, the data storage operation is performed so that the second acquisition module can perform data processing operations on the target data.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a data processing method, an acquisition module, an acquisition system and a medium. Background Art

[0002] In practical applications, data acquisition technology based on shared storage space has been widely used. However, data acquisition technology based on shared storage space usually uses shared storage space as a data transmission channel from the data acquisition end to the data processing end, and the control information used to control data acquisition and data transmission processing needs to be transmitted through a dedicated control information transmission channel. Such a transmission scheme has a large data processing performance overhead. Summary of the invention

[0003] Based on the above problems, the embodiments of the present application provide a data processing method, an acquisition module, an acquisition system and a medium.

[0004] The technical solution provided by the embodiment of the present application is as follows:

[0005] A data processing method is applied to a first acquisition module; the method comprises:

[0006] Determine first address information; wherein the first address information includes address information of a first storage space; the first storage space includes a storage space in a shared storage space where the first acquisition module can perform a data storage operation on the data to be processed; the first storage space stores control information;

[0007] Determining the control information based on the first address information;

[0008] Processing the data to be processed collected by the first collection module based on the control information to obtain target data;

[0009] Based on the first address information and the target data, the data storage operation is performed so that the second acquisition module can perform a data processing operation on the target data.

[0010] The present application also provides a data processing method applied to the second acquisition module, the method comprising:

[0011] Determine first address information; wherein the first address information includes address information of a first storage space; the first storage space includes a storage space in a shared storage space where the second acquisition module can perform data processing operations on target type data; the first storage space stores control information and target data; the target data is obtained by the first acquisition module processing the to-be-processed data acquired by the first acquisition module based on the control information; the control information is determined by the first acquisition module based on the first address information; the data type of the target data is the target type;

[0012] Based on the first address information, the data processing operation is performed on the target data.

[0013] The embodiment of the present application further provides a first acquisition module, which includes a first determination unit, a first acquisition unit, and a first processing unit; wherein:

[0014] The first determining unit is used to determine the first address information; wherein the first address information includes address information of the first storage space; the first storage space includes a storage space in the shared storage space where the first acquisition module can perform a data storage operation on the data to be processed; the first storage space stores control information;

[0015] The first acquiring unit is used to determine the control information based on the first address information;

[0016] The first processing unit is used to process the data to be processed collected by the first collection module based on the control information to obtain target data;

[0017] The first processing unit is used to perform the data storage operation based on the first address information and the target data, so that the second acquisition module can perform a data processing operation on the target data.

[0018] An embodiment of the present application also provides another first acquisition module, which includes a first processor and a first memory; wherein: the first memory stores a computer program that can be run on the first processor, and when the first processor executes the computer program, it implements the data processing method applied to the first acquisition module as described above.

[0019] The embodiment of the present application further provides a second acquisition module, the second acquisition module comprising a second determination unit, a second acquisition unit and a second processing unit; wherein:

[0020] The second determination unit is used to determine the first address information; wherein the first address information includes the address information of the first storage space; the first storage space includes a storage space in the shared storage space where the second acquisition module can perform data processing operations on target type data; the first storage space stores control information and target data; the target data is obtained by the first acquisition module processing the to-be-processed data acquired by the first acquisition module based on the control information; the control information is determined by the first acquisition module based on the first address information; the type of the target data is the target type;

[0021] The second acquisition unit is used to acquire the target data based on the first address information;

[0022] The second processing unit is used to perform the data processing operation on the target data based on the first address information.

[0023] An embodiment of the present application also provides another second acquisition module, which includes a second processor and a second memory; wherein: the second memory stores a computer program that can be run on the second processor, and when the second processor executes the computer program, it implements the data processing method applied to the second acquisition module as described above.

[0024] An embodiment of the present application further provides a data processing system, which includes a shared storage space, the first acquisition module as described above, and the second acquisition module as described above.

[0025] An embodiment of the present application also provides a computer-readable storage medium, in which a computer program is stored. When the computer program is executed by a processor, it can implement the data processing method applied to the first acquisition module as described above or the data processing method applied to the second acquisition module as described above.

[0026] In the data processing method applied to the first acquisition module provided in the embodiment of the present application, control information and target data are stored based on the first address information, thereby realizing the bundling of target data and control information. In this way, in the process of storing and processing target data based on a shared storage space, through a single read and write operation on the storage space or storage unit corresponding to the first address information, the acquisition or storage transmission of control information and target data can be realized simultaneously, thereby realizing the merged establishment of a data transmission channel and a control information transmission channel, thereby reducing the performance overhead generated by the transmission of control information during the target data transmission process, and realizing low-overhead and high-performance target data transmission operations based on a shared storage space. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 A schematic diagram of a flow chart of a data processing method applied to a first acquisition module provided in an embodiment of the present application;

[0028] Figure 2 A schematic diagram of a process for determining first address information by a first acquisition module provided in an embodiment of the present application;

[0029] Figure 3 A schematic diagram of a process for processing data to be processed based on a control identifier provided in an embodiment of the present application;

[0030] Figure 4 A schematic diagram of a flow chart of a data processing method applied to a second acquisition module provided in an embodiment of the present application;

[0031] Figure 5 A schematic diagram of a process for obtaining target data by a second acquisition module provided in an embodiment of the present application;

[0032] Figure 6 A schematic diagram of the architecture of data processing of the first acquisition module and the second acquisition module provided in an embodiment of the present application;

[0033] Figure 7 A schematic diagram of the overall flow of the data processing method provided in the embodiment of the present application;

[0034] Figure 8 A schematic diagram of the architecture of the data processing method provided in the embodiment of the present application in an indicator data acquisition and transmission frame;

[0035] Figure 9a A schematic diagram of the structure of a first acquisition module provided in an embodiment of the present application;

[0036] Figure 9b Another structural schematic diagram of the first acquisition module provided in an embodiment of the present application;

[0037] Fig.10a A schematic diagram of the structure of a second acquisition module provided in an embodiment of the present application;

[0038] Fig.10b Another structural schematic diagram of the second acquisition module provided in an embodiment of the present application. DETAILED DESCRIPTION

[0039] The technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application.

[0040] It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0041] The present application relates to the field of information technology, and in particular to a data processing method, a collection module, a collection system and a medium.

[0042] In practical applications, data collection technology based on shared storage space has been widely used. For example, in indicator data collection and analysis, at least one indicator data can be collected through data collection technology based on shared storage space, and the indicator data can be stored in the shared storage space. When the indicator data needs to be analyzed, the indicator data can be obtained from the shared storage space.

[0043] However, taking indicator data collection and analysis as an example, the shared storage space becomes the data path between one end for collecting indicator data and one end for acquiring and analyzing indicator data, which realizes the direct transmission of data between the above two ends; at the same time, the control information between the one end for collecting indicator data and the shared storage space, and between the shared storage space and the one end for acquiring and analyzing indicator data, still needs to be transmitted through dedicated channels, which results in serious data transmission performance overhead.

[0044] Based on the above problems, an embodiment of the present application provides a data processing method, in which control information and target data are stored in the same storage space. In this way, when the modules at both ends of the shared storage space perform data processing operations in the storage space, they can directly obtain the control information stored in the storage space, thereby realizing bundled storage and transmission processing of control information and target data, reducing the data processing overhead caused by the transmission and processing of control information, and improving the efficiency of data processing in the shared storage space.

[0045] The embodiment of the present application first provides a data processing method applied to a first acquisition module.

[0046] It should be noted that the data processing method applied to the first acquisition module provided in the embodiment of the present application can be implemented by the first processor of the first acquisition module. The above-mentioned first processor can be at least one of an application-specific integrated circuit (ASIC), a digital signal processor (DSP), a digital signal processing device (DSPD), a programmable logic device (PLD), a field programmable gate array (FPGA), a central processing unit (CPU), a controller, a microcontroller, and a microprocessor.

[0047] In one implementation, the first acquisition module may be a module that can implement the data acquisition function and is carried by a physical machine; the first acquisition module may also be a virtual module that can implement the data acquisition function, or may be an executable code that can implement the data acquisition function.

[0048] Figure 1 Schematic diagram of the flow of the data processing method applied to the first acquisition module provided in the embodiment of the present application. Figure 1 As shown, the method may include steps 101 to 104:

[0049] Step 101: Determine first address information.

[0050] Among them, the first address information includes address information of the first storage space; the first storage space includes a storage space in the shared storage space where the first acquisition module can perform storage operations on the data to be processed; and control information is stored in the first storage space.

[0051] In one implementation, the shared storage space may be a physical storage space; illustratively, the shared storage space may be a storage space on a hard disk; or it may be a storage space in a memory, in which case the shared storage space may be a shared memory.

[0052] In one implementation, the shared storage space may include at least one sub-storage space, and the first storage space may be a sub-storage space of the shared storage space.

[0053] In one implementation, the first storage space may include at least one storage unit; illustratively, the at least one storage unit may be continuously distributed in the shared storage space.

[0054] In one embodiment, the data storage operation may include storing at least one data; and may also include storing at least one type of data. Exemplarily, when storing at least one type of data, different types of data may be stored in parallel.

[0055] In one implementation, the data storage operation on the data to be processed may include a data storage operation only on the data to be processed; or a data storage operation performed on part of the data to be processed.

[0056] In one implementation, the first address information may be address information of a starting storage unit of the first storage space; illustratively, the first address information may be a physical address or a virtual address.

[0057] In one embodiment, the control information may include at least one type of information for controlling data processing operations, such as controlling whether to perform a data storage operation, how to perform a data storage operation, and when to perform a data storage operation.

[0058] In one implementation, the control information may be embodied in any form of a character string, a single bit, or a bit combination.

[0059] In one embodiment, the shared storage space may include at least one sub-storage space, some of which already store valid data. In this case, if data storage operations are performed on these sub-storage spaces without processing these valid data, the valid data will be overwritten, thereby causing waste of valid data. Therefore, in an embodiment of the present application, the first address information is the address information of the first storage space in which the first acquisition module can perform data acquisition operations.

[0060] Step 102: Acquire control information based on the first address information.

[0061] In one embodiment, the control information can be stored in a storage unit corresponding to the first address information in the first storage space. Exemplarily, the control information can be stored in the starting storage unit of the first storage space. The control information can also be stored in at least one storage unit that maintains a certain address offset with the first address information. Exemplarily, the above-mentioned address offset can be fixed or adjustable.

[0062] In one implementation, only one copy of the control information may be stored in the first storage space; illustratively, one copy of the control information may be stored in each storage unit of the first storage space, and the control information stored in each storage unit may be different.

[0063] Step 103: Process the data to be processed collected by the first collection module based on the control information to obtain target data.

[0064] In one embodiment, the data to be processed may be raw data collected by the first collection module. Exemplarily, the raw data may be data related to network traffic, and may include at least one of virtual machine (VM) traffic data, network card traffic data, and other traffic data.

[0065] In one implementation, processing the data to be processed based on the control information may be implemented in any of the following ways:

[0066] Based on the control information, additional information is added to the data to be processed; illustratively, the additional information may be at least one of the acquisition time and type information of the data to be processed.

[0067] Based on the control information, a negation operation is performed on part of the data to be processed; the part of the data may be part of the bits in the data to be processed.

[0068] When the control information indicates that the data to be processed should be kept unchanged, the data to be processed should be kept unchanged.

[0069] Step 104: Perform a data storage operation based on the first address information and the target data, so that the second acquisition module can perform a data processing operation on the target data.

[0070] In one implementation, performing the data storage operation based on the first address information and the target data may be implemented in any of the following ways:

[0071] The target data is stored in a first storage space corresponding to the first address information.

[0072] The target data is stored in a partial storage unit of the first storage space corresponding to the first address information; illustratively, the partial storage unit may be at least one storage unit adjacent to the starting storage unit in the first storage space; wherein the number of the at least one storage unit may correspond to the data volume of the target data.

[0073] The target data is stored in a storage unit whose address and first address information maintain a specified address offset; illustratively, the specified address offset may be associated with the data to be processed and / or the target data; the specified address offset may be adjustable and controllable.

[0074] In one implementation, the second acquisition module may include a module capable of acquiring data from a shared storage space; illustratively, the second acquisition module may be a physical machine module or an executable code module.

[0075] In one implementation, the data processing operation performed by the second acquisition module may include at least one of the following operations:

[0076] Get the target data.

[0077] The target data is read, and the storage unit corresponding to the first address information is set to an initial value, that is, the storage space corresponding to the first address information is released so that the first acquisition module can perform a data storage operation on the storage unit next time.

[0078] The target data is processed based on the first address information, and the processing result is updated to the storage space corresponding to the first address information.

[0079] The target data is acquired from the first storage unit corresponding to the first address information, and the target data is sent to other modules so that the other modules can further analyze and process the target data.

[0080] From the above, it can be seen that in the data processing method applied to the first acquisition module provided in the embodiment of the present application, after the first acquisition module acquires the data to be processed and determines the first address information of the first storage space on which it can perform data storage operations, it can obtain control information for processing the data to be processed based on the first address information, and process the data to be processed based on the control information to obtain target data, and then perform data storage operations on the target data based on the first address information, so that the second acquisition module can perform data processing operations on the target data.

[0081] Thus, in the data processing method applied to the first acquisition module provided in the embodiment of the present application, the control information and the target data are stored based on the first address information, thereby realizing the bundling of the target data and the control information. Thus, in the process of storing and processing the target data based on the shared storage space, the acquisition or storage transmission of the control information and the target data can be realized simultaneously through a single read and write operation on the storage space or storage unit corresponding to the first address information, thereby realizing the combined establishment of the data transmission channel and the control information transmission channel, thereby reducing the performance overhead caused by the transmission of the control information during the target data transmission process, and realizing low-overhead and high-performance target data transmission operations based on the shared storage space.

[0082] Based on the above embodiments, in the data processing method applied to the first acquisition module provided in the embodiments of the present application, step 101 can be implemented through steps A1 to A2:

[0083] Step A1: Obtain first information.

[0084] The first information includes storage status information of the storage unit in the second storage space; the second storage space includes storage space in the shared storage space that is associated with type information of the data to be processed.

[0085] In one embodiment, the shared storage space may include at least one sub-storage space associated with the type information of the data to be processed; illustratively, the shared storage space may include a sub-storage space associated with the type information of the data to be processed and a sub-storage space not associated with the type information of the data to be processed. In other words, the second storage space may be a sub-storage space of the shared storage space.

[0086] In one embodiment, the first information may include storage status information of each storage unit in the second storage unit; exemplarily, the storage status information may include at least one information such as whether the storage unit stores data, whether all bits in the storage unit are filled with valid data, and the continuous storage time of the data stored in the storage unit.

[0087] In one implementation, the first information may be stored in the second storage space; illustratively, the first information may be stored in a designated location in the shared storage space, thereby facilitating acquisition by the first acquisition module and reducing addressing operations of the first acquisition module.

[0088] Step A2: Determine first address information based on the first information.

[0089] In one implementation, determining the first address information based on the first information may be implemented in any of the following ways:

[0090] Based on the storage status information of each storage unit in the second storage space included in the first information, a storage unit with the smallest address among the storage units that do not store valid data is determined, and the address information of the storage unit is determined as the first address information.

[0091] Based on the storage status information of each storage unit in the second storage space included in the first information, a storage unit with the smallest address among the storage units that are not fully written with valid data is determined, and the address information of the storage unit is determined as the first address information.

[0092] From the above, it can be seen that in the data processing method provided in the embodiment of the present application, the shared storage unit is pre-divided according to the type information of the data. Therefore, when obtaining any data to be processed, the first address information can be quickly determined according to the type information of the data to be processed, thereby narrowing the addressing range in the process of determining the first address information and reducing the amount of addressing calculation in the process of determining the first address information; and, the first address information can also be determined according to the storage status information of the sub-storage space, thereby realizing precise storage status control, which can further improve the efficiency of determining the first address information, reduce the probability of target data covering valid data, and improve the efficiency of the first acquisition module in performing data storage operations.

[0093] Based on the above embodiments, in the data processing method applied to the first acquisition module provided in the embodiments of the present application, step A1 can also be implemented through steps B1 to B3:

[0094] Step B1, obtaining type space association information.

[0095] The type space association information includes information on the association relationship between the data type information and the address information of the shared storage space.

[0096] In one embodiment, the type space association relationship information may be determined before the data processing method is executed; illustratively, the type space association relationship may be stored in a shared storage space, or may be stored in a storage unit of the first acquisition module as configuration information for implementing the data processing process, or may be set in the executable code of the first acquisition module.

[0097] In one implementation, the type space association relationship information may also be reflected through data information stored in at least one data structure such as a two-dimensional array, a set, and a structure.

[0098] In one implementation, the type space association relationship information, in addition to including information about the association relationship between the data type information and the address information, may also include the size of the sub-storage space corresponding to each data type information.

[0099] In one implementation, the type space association relationship information may be reflected by shared storage space metadata. Exemplarily, the data structure of the shared storage space metadata may be as shown in Table 1:

[0100] Table 1

[0101]

[0102] In Table 1, the shared storage space metadata includes first type metadata to Nth type metadata, which correspond to first data type information to Nth data type information respectively; the first data type information may include first data type 1 to first data type p; the Nth data type information may include Nth data type 1 to Nth data type m; wherein N, p and m may all be integers greater than or equal to 2.

[0103] In Table 1, the shared storage space metadata may include the size of the sub-storage space, the number of the sub-storage spaces, and the address information of the sub-storage space corresponding to at least one data type information in the shared storage space.

[0104] In Table 1, each type of metadata may include the size of the sub-storage space corresponding to the type information, the storage status information of the sub-storage space corresponding to the type information, etc.; wherein, the storage status information may include the total number of storage units in the sub-storage space corresponding to the type information, the amount of data in the storage units that have stored data, the number of storage units that can be written with target data, etc.

[0105] As shown in Table 1, the first type metadata may include the storage status of the first storage unit to the pth storage unit of the first type sub-storage space, that is, whether the storage status of the first storage unit to the pth storage unit of the first type sub-storage space corresponds to the storage of the first type data 1 to the first type data p; the Nth type metadata may include the storage status of the first storage unit to the mth storage unit of the Nth type sub-storage space, that is, whether the storage status of the first storage unit to the mth storage unit of the Nth type sub-storage space corresponds to the storage of the Nth type data 1 to the Nth type data m. Exemplarily, in any type of sub-storage space, only one target data of this type may be stored, or no target data of this type may be stored.

[0106] It should be noted that before the first acquisition module acquires the data to be processed, a pointer pointing to the shared storage space can be added to the executable code of the first acquisition module; the shared storage space metadata can be obtained by adding a certain offset to the above pointer. Exemplarily, the above pointer can be implemented in the existing code for implementing data storage operations, such as by adding or rewriting an interface structure, adding or extending an interface class; the above pointer can also be a global pointer, which can reduce the amount of data operations generated by maintaining the pointer in the functional implementation of the data storage operation.

[0107] Step B2: Determine the second address information based on the type information of the data to be processed and the type space association information.

[0108] The second address information includes address information of the second storage space.

[0109] In one implementation, the second address information may include address information of a storage unit in the second storage space that stores the first information.

[0110] In one implementation, the second address information may be address information of a starting storage unit of the second storage space.

[0111] In one implementation, determining the second address information based on the type information of the data to be processed and the type space association information may be implemented in any of the following ways:

[0112] When any data type information in the type space association information matches the type information of the data to be processed, the address information of the storage space corresponding to the type information of the data to be processed is determined to be the second address information, and the second address information is a valid address.

[0113] When any data type information in the type space association information does not match the type information of the data to be processed, the second address information is determined to be an invalid address.

[0114] Step B3: Based on the second address information, obtain the first information.

[0115] In one implementation, the first information may be obtained in any of the following ways:

[0116] The first information is directly obtained from the storage unit corresponding to the second address information.

[0117] Based on the second address information, a storage unit storing the first information is addressed, and the first information is obtained from the storage unit.

[0118] From the above, it can be seen that the data processing method applied to the first acquisition module provided in the embodiment of the present application divides the shared storage space according to the data type information. After obtaining the data to be processed, the storage status information of the corresponding sub-storage space can be determined according to the type information of the data to be processed. This improves the efficiency of determining the first address information and lays the foundation for efficient acquisition of control information and efficient data storage operations of target data.

[0119] Based on the foregoing embodiment, in the data processing method applied to the first acquisition module provided by the embodiment of the present application, the first information includes bitmap information associated with the second storage space; after the data storage operation is completed, the first information is updated.

[0120] In one implementation, the bitmap information may be used to indicate whether data is stored in a storage unit in the shared storage space.

[0121] In one embodiment, the bitmap information may include a storage status identifier corresponding to each storage unit of the shared storage space. Exemplarily, the storage status identifier may be binary data. For example, when the storage status identifier is 1, it may indicate that valid target data is stored in the storage unit; when the storage status identifier is 0, it may indicate that the target data is not stored in the storage unit, or the stored data is invalid data; exemplarily, the storage status identifier in the bitmap information may be recorded as bit information.

[0122] In one implementation, the bitmap information may be stored in metadata associated with the data type information.

[0123] In one implementation, the first information may also be embodied in the form of an array.

[0124] In the embodiment of the present application, flexible addressing of the first address information can be achieved through the bit map information, thereby improving the efficiency of determining the first address information.

[0125] Accordingly, updating the first information may be achieved by any of the following methods:

[0126] A result of the data storage operation is judged, and the first information is updated according to the judgment result.

[0127] When the judgment result indicates that the target data is successfully written into the storage unit corresponding to the first address information, the storage status flag associated with the storage unit in the first information that is used to write the target data is set; when the judgment result indicates that the target data fails to be written into the storage unit corresponding to the first address information, the storage status flag associated with the storage unit in the first information that is used to write the target data is reset.

[0128] From the above, it can be seen that in the data processing method provided in the embodiment of the present application, since the information representation method in the bit map information is intuitive, vivid and simple, therefore, when the bit map information is used to represent the storage status information of the sub-storage space of the shared storage space, the first address information can be quickly and accurately determined according to the bit map information, thereby reducing the pointer traversal operation of the shared storage space generated by addressing the first address information, and improving the efficiency of determining the first address information; and after the data storage operation is completed, the first information can also be updated in time, thereby providing an accurate addressing basis for the subsequent data storage operation of the first acquisition module and the execution of the data processing operation of the second acquisition module.

[0129] Figure 2 A schematic diagram of a process for determining first address information by a first acquisition module provided in an embodiment of the present application. Figure 1 As shown, the process may include steps 201 to 207:

[0130] Step 201: Collect data to be processed.

[0131] Exemplarily, before collecting the data to be processed, collection requirement configuration information may be obtained first, and then the data to be processed may be collected according to the collection requirement configuration information.

[0132] Step 202: Obtain bitmap information in the shared storage space metadata.

[0133] Exemplarily, the bitmap information in the shared storage space metadata may be the bitmap information in the sub-storage space corresponding to the type information of the data to be processed, that is, the storage status information represented by the bit information in the bitmap information is the storage status information of the storage unit in the sub-storage space corresponding to the type information of the data to be processed.

[0134] Step 203: Determine whether all bit information in the bitmap information has been detected.

[0135] Exemplarily, when the bit information in the bitmap information is represented in binary and the bitmap information includes at least two bits of information, it is possible to quickly determine whether all the bit information in the bitmap information has been detected by performing an XOR operation or a value comparison on the at least two bits of information.

[0136] Exemplarily, if the information in the bit map information has not been fully detected, it may indicate that the second acquisition module has not performed data processing operations on the data stored in all storage units in the second storage space, and step 204 may be executed at this time; if the bit information in the bit map information has been fully detected, it may indicate that the second acquisition module has performed data processing operations on the data stored in each storage unit in the second storage space, and step 205 may be executed at this time.

[0137] Step 204: Obtain the next bit information in the bitmap information.

[0138] Exemplarily, the next bit information may be selected according to the subscript of the bit information in the bitmap information, and then step 206 is performed.

[0139] Step 205: Return the first pointer.

[0140] Exemplarily, the first pointer may represent the address of a starting storage unit in the second storage space.

[0141] Step 206: Determine whether the next bit information in the bitmap information is detected.

[0142] Exemplarily, if the next bit of information is detected, the next bit of information can be set to 1 by the second acquisition module, indicating that the second acquisition module has performed a data processing operation on the data stored in the storage unit corresponding to the next bit of information. At this time, it means that the first acquisition module can perform a data storage operation on the storage unit, and step 207 can be executed; exemplarily, the storage unit can be recorded as a storage unit available to the first acquisition module.

[0143] Exemplarily, if the next bit of information is not detected, the value of the next bit of information may be 0, indicating that the second acquisition module has not performed a data processing operation on the data stored in the storage unit corresponding to the next bit of information, that is, the first acquisition module cannot perform a data storage operation on the storage unit. At this time, steps 203 to 205 may be executed to continue to determine the state of the next bit of information.

[0144] Step 207: Return the second pointer.

[0145] Exemplarily, the second pointer points to a storage unit where the first acquisition module can perform data storage operations.

[0146] exist Figure 2After the process shown is finished, the first pointer or the second pointer can be determined as the first address information.

[0147] From the above, it can be seen that in the process of determining the first address information, the judgment of the bit map information is involved. Based on the particularity that the bit information in the bit map information can be binary data, a simple XOR or comparison operation can be used to quickly determine whether all storage units in the second storage space are unavailable, thereby improving the efficiency of determining the first address information.

[0148] Based on the above embodiments, in the data processing method applied to the first acquisition module provided in the embodiments of the present application, step 102 can be determined through steps C1 to C2:

[0149] Step C1: Based on the first address information, obtain the second information.

[0150] The second information includes data processing configuration information associated with the storage unit of the first storage space; the data processing configuration information includes at least one type of configuration information.

[0151] In one implementation, the second information may be directly obtained from the starting storage unit corresponding to the first address information.

[0152] In one implementation, in order to facilitate data reading and writing analysis, the configuration information may be embodied in digital form; illustratively, the configuration information may be binary digits; illustratively, the configuration information may include pre-configured information on the data to be processed.

[0153] In one embodiment, when the data processing configuration information includes at least two types of configuration information, the data processing configuration information can be embodied in the form of at least one data structure in an array and a structure; when the configuration information is a binary number, the data processing configuration information can include at least two binary numbers.

[0154] Step C1: when the configuration information in the second information is updated by the second acquisition module, determine the control information based on each configuration information in the second information.

[0155] In one embodiment, the second acquisition module can perform a data processing operation on the target data based on the first address information, and can update the second information after the data processing operation is completed; exemplarily, when the number of configuration information in the second information is at least two, after the data processing operation is completed, the second acquisition module can update at least one configuration information.

[0156] In one embodiment, if the second acquisition module updates part of the configuration information in the second information, it may indicate that the data processing operation of the second acquisition module for the target data stored in the first address information has not yet been completed. At this time, the data storage operation of the first acquisition module based on the first address information may be prohibited, or the first acquisition module may be enabled to perform data storage operations based on the updated part of the configuration information.

[0157] In one embodiment, if the second acquisition module updates each configuration information in the second information, it can indicate that the second acquisition module has completed the processing operation of the target data stored in the first address information, and the data storage operation of the first acquisition module for the first address information is safe and effective.

[0158] In one implementation, determining the control information based on the configuration information in the second information may be implemented in any of the following ways:

[0159] Obtain each configuration information in the second information, process each configuration information, and determine the control information; illustratively, the processing of each configuration information in the second information can be determined according to the data processing requirements between the first acquisition module and the second acquisition module.

[0160] At least one configuration information in the second information is selected, and control information is determined by the selected at least one configuration information.

[0161] Each piece of configuration information in the second information is determined as control information.

[0162] Based on the foregoing embodiment, the data processing method applied to the first acquisition module provided in the embodiment of the present application may further include the following operations:

[0163] After the data storage operation is completed, the second information is updated.

[0164] In one implementation, updating the second information may be updating all configuration information in the second information; or updating at least one configuration information based on which the first acquisition module processes the target data.

[0165] When the data processing configuration information is a control identifier and the configuration information in the data processing configuration information is a control flag, the embodiment of the present application also provides a process for processing the data to be processed based on the control identifier. Figure 3 A flow chart of processing data to be processed based on a control identifier provided in an embodiment of the present application, such as Figure 3 As shown, the process may include steps 301 to 308:

[0166] Step 301: Collect data to be processed.

[0167] Step 302: Get the control identifier.

[0168] Exemplarily, the acquisition of the control identifier may be achieved by the method for acquiring the second information provided in the aforementioned embodiment, which will not be described in detail here.

[0169] Step 303: Determine whether all control identification bits in the control identification have been detected.

[0170] Exemplarily, the control identification bits in the control identification may be binary data, and therefore, it is possible to quickly determine whether all control indications in the control identification are detected by performing an XOR operation or comparison on all control indication bits as described in the aforementioned embodiments.

[0171] Exemplarily, if all control identification bits in the control identification are detected, it may indicate that the second acquisition module has detected and set all control identification bits in the control identification, that is, the second acquisition module has completed the update of the control identification. In this case, step 304 may be executed.

[0172] Exemplarily, if the control identification bits in the control identification are not all detected, it may mean that the second acquisition module has detected and set some of the control identification bits in the control identification, that is, the second acquisition module has completed the update of some of the control identification bits in the control identification. In this case, step 305 may be executed.

[0173] Step 304: determine target data and store the target data.

[0174] Exemplarily, the first acquisition module may process the data to be processed according to the control identification bit, thereby determining the target data, and store the target data in a storage unit corresponding to the first address information.

[0175] Step 305: Select the next control flag in sequence.

[0176] Exemplarily, according to the numbers of the control flags, the control flag with the smallest number may be selected from the control flags that have not been determined as the next control flag.

[0177] Step 306: Determine whether the next control flag is set.

[0178] Exemplarily, determining whether the next control flag is set may be achieved by determining whether the next control flag is equal to a preset flag; exemplary, the preset flag may be 1.

[0179] Exemplarily, the next control flag position may include that the second acquisition module has set and updated the control flag, and step 307 can be executed at this time; if the next control flag is not set, it means that the second acquisition module has not updated the control flag, and step 303 can be executed at this time.

[0180] Step 307: Execute corresponding control operations on the data to be processed.

[0181] Exemplarily, the corresponding control operation may be determined by the next control flag set by the second acquisition module in step 306; exemplary, the target data may be obtained by performing the corresponding control operation on the data to be processed.

[0182] Step 308: reset the next control flag.

[0183] After step 308 is completed, step 303 may be further executed to implement various operations on the data to be processed, thereby obtaining at least one target data, and storing at least one target data in the shared storage space.

[0184] From the above, it can be seen that in the embodiment of the present application, the first acquisition module determines the control information based on each configuration information in the second information only when it determines that the configuration information in the second information is updated by the second acquisition module. In this way, the control information determined by the above process can coordinate the data operations of the first acquisition module and the second acquisition module based on the shared storage space, reduce the probability of failure of the target data operation, and improve the efficiency of data processing based on the shared storage space.

[0185] Based on the foregoing embodiments, the embodiments of the present application further provide a data processing method applied to a second acquisition module. It should be noted that the method can be implemented by a second processor of the second acquisition module. Exemplarily, the second processor can be at least one of an ASIC, a DSP, a DSPD, a PLD, an FPGA, a CPU, a controller, a microcontroller, and a microprocessor.

[0186] Figure 4 Schematic diagram of the flow of the data processing method applied to the second acquisition module provided in the embodiment of the present application. Figure 4 As shown, the method may include steps 401 to 402:

[0187] Step 401: Determine first address information.

[0188] Among them, the first address information includes the address information of the first storage space; the first storage space includes a storage space in the shared storage space where the second acquisition module can perform data processing operations on the target type data; the first storage space stores control information and target data; the target data is obtained by the first acquisition module processing the data to be processed collected by the first acquisition module based on the control information; the control information is determined by the first acquisition module based on the first address information; the type of the target data is the target type.

[0189] In one embodiment, the target type may be determined by the second acquisition module before the data processing method is started; illustratively, the second acquisition module may determine the target type based on actual data acquisition requirements, and may also determine the target type based on a data acquisition instruction issued by a host computer.

[0190] In one implementation, the data processing operation may include an operation of acquiring target type data.

[0191] In one implementation, the data processing operation may include obtaining data of a target type, processing the data of the target type, and then storing the processed data in the first storage space.

[0192] Step 402: Perform a data processing operation on the target data based on the first address information.

[0193] Exemplarily, performing a data processing operation on the target data based on the first address information may be implemented in the following manner:

[0194] The first address information is obtained, and the target data stored in the storage unit in the specified range with the first address information as the starting address is processed.

[0195] The target data stored in the storage unit corresponding to the first address information is processed.

[0196] From the above, it can be seen that the target data of the data processing operation performed by the second acquisition module is obtained by the first acquisition module processing the data to be processed collected by it based on the control information, and the control information and the target data are stored at the same time in the first storage space corresponding to the first address information. Therefore, whether it is the first acquisition module or the second acquisition module, through a read and write operation on the first storage space, they can realize their dual reading and writing of the control information and the target data, thereby reducing their data processing overhead based on the shared storage space and improving the data processing efficiency based on the shared storage space.

[0197] Based on the above embodiments, in the data processing method applied to the second acquisition module provided in the embodiments of the present application, step 401 can be implemented through steps D1 to D2:

[0198] Step D1, obtaining first information.

[0199] The first information includes storage status information of the storage unit in the second storage space; the second storage space includes storage space associated with the type information of the target data in the shared storage space;

[0200] Step D2: Determine first address information based on the first information.

[0201] From the above, it can be seen that in the embodiment of the present application, the second acquisition module can obtain the storage status information of each storage unit in the second storage space, and then determine the first address information based on the storage status information. When the storage status information is represented in a simple and intuitive manner, the first address information can be efficiently, quickly and accurately determined.

[0202] Based on the above embodiments, in the data processing method applied to the second acquisition module provided in the embodiments of the present application, step D1 can be implemented by steps E1 to E3:

[0203] Step E1, obtaining type space association information.

[0204] The type space association information includes information on the association relationship between the data type information and the address information in the shared storage space.

[0205] Step E2: Determine the second address information based on the type information of the target type and the type space association information.

[0206] The second address information includes address information of the second storage space.

[0207] Step E3: Obtain the first information based on the second address information.

[0208] From the above, it can be seen that the data processing method applied to the second acquisition module provided in the embodiment of the present application, after the second acquisition module obtains the type space association information, can determine the second address information based on the type information of the target data and the type space association information, and then obtain the first information based on the second address information. In this way, on the one hand, a reasonable division of sub-storage space in the shared storage space is achieved, and on the other hand, rapid acquisition of the first information is also achieved.

[0209] Based on the foregoing embodiment, in the data processing method applied to the second acquisition module provided in the embodiment of the present application, the first information includes bitmap information associated with the second storage space; the method further includes:

[0210] After the data processing operation is completed, the first information is updated.

[0211] From the above, it can be seen that in the embodiment of the present application, the second acquisition module can timely update the first information after the data processing operation is completed. Combined with the description of the previous embodiment, it can be seen that after the first information is updated, the first acquisition module can quickly and accurately determine the first address information based on the first information, thereby laying the foundation for the efficient data storage operation of the first acquisition module.

[0212] Based on the above embodiments, in the data processing method applied to the second acquisition module provided in the embodiments of the present application, step 401 can be implemented by steps F1 to F3:

[0213] Step F1, obtaining the execution times of the data processing operation.

[0214] In one implementation, the number of executions of the data processing operation may include the total number of data processing operations executed by the second acquisition module; and may also include the number of data processing operations executed by the second acquisition module on data of specified type information.

[0215] In one implementation, the number of executions of the data processing operation may include the number of data processing operations executed from the time when the second acquisition module last started the data processing operation to the current time.

[0216] In one implementation, the number of executions of the data processing operation may include the number of data processing operations executed by the second acquisition module for data of a specified type of information from the time when the last set of data processing operations ended to the current moment.

[0217] In one embodiment, the number of executions of the data processing operation may include the number of data processing operations executed by the second acquisition module within a collection cycle, and after the end of a collection cycle, the number of data processing operations may be cleared. Exemplarily, the collection cycle may include the duration of the second acquisition module executing the data processing operation, and may also include the number of data processing operations executed by the second acquisition module.

[0218] Step F2: Based on the first address information, obtain second information.

[0219] The second information includes data processing configuration information associated with the storage unit of the first storage space; the data processing configuration information includes at least one type of configuration information.

[0220] Step F3: When the specified conditions are met, a data processing operation is performed on the target data.

[0221] The specified conditions include that the number of executions is less than or equal to a specified threshold and the configuration information in the second information is updated by the first acquisition module after the data storage operation is completed.

[0222] In one embodiment, the specified threshold value can be determined based on the data processing capability of the second acquisition module; exemplarily, the specified threshold value can be determined based on the centralized processing capability or centralized processing requirement of the second acquisition module for at least one target data; exemplarily, the specified threshold value can also be determined based on the command issued by the host computer of the second acquisition module; exemplarily, the specified threshold value can be determined based on the acquisition period of the second acquisition module.

[0223] In one embodiment, the second acquisition module may package the acquired at least one target data to obtain a data packet, and at this time, the specified threshold may be the maximum number of target data that can be included in the data packet. Exemplarily, the second acquisition module may report the data packet to a host computer for further analysis and processing of the data packet by the host computer.

[0224] In one implementation, the process of updating the configuration information in the second information by the first acquisition module is the same as the process of updating the configuration information in the second information by the second acquisition module in the aforementioned embodiment, and will not be described again.

[0225] In the embodiment of the present application, the second information may be further updated after the data processing operation is completed.

[0226] From the above, it can be seen that the second acquisition module performs data processing operations on the target data only when the specified conditions are met, thereby improving the pertinence of the second acquisition module in performing data processing operations, realizing the filtering of the target data, and improving the efficiency of the second acquisition module in performing data processing operations; and, the second acquisition module updates the second information in a timely manner after the data processing operation is completed, which can also improve the processing efficiency of the first acquisition module on the data to be processed.

[0227] Under the condition that the data processing operation performed by the second acquisition module includes data acquisition and data reporting operations, the embodiment of the present application also provides a method for the second acquisition module to acquire and report target data. Figure 5 A schematic diagram of a process for obtaining target data by a second acquisition module provided in an embodiment of the present application. Figure 5 As shown, the method may include steps 501 to 510:

[0228] Step 501: Obtain shared storage space metadata.

[0229] Step 502: Obtain bitmap information corresponding to the specified data type.

[0230] Exemplarily, the specified data type may be set by the second acquisition module, or may be determined by the second acquisition module based on an instruction issued by the host computer; the specified data type may be consistent with the type information of the target data.

[0231] Step 503: Determine whether the number of executions is greater than or equal to a specified threshold.

[0232] Exemplarily, the number of executions here may include the number of times the second acquisition module executes to obtain target data.

[0233] Exemplarily, if the number of executions is greater than or equal to a specified threshold, steps 504 to 505 are executed; if the number of executions is less than the specified threshold, step 506 is executed.

[0234] Step 504: Obtain target data.

[0235] Step 505: process the target data, obtain a data packet, and report the data packet.

[0236] Exemplarily, the second acquisition module may process the target data based on preset packaging configuration information.

[0237] Step 506: Select the next set bit information according to the bitmap information.

[0238] Exemplarily, the storage unit corresponding to the next set bit information may be a storage unit in which the second acquisition module can perform data acquisition operations.

[0239] Step 507: Determine whether the control identification bit in the control identification has been reset.

[0240] Exemplarily, the control identification bit in the control identification is reset, which may be executed by the first acquisition module after performing the data processing operation, and step 508 can be executed at this time; if the control identification bit in the control identification is not reset, it means that the first acquisition module has not yet written the target data, and step 503 can be executed at this time.

[0241] Step 508: Determine whether the target data is invalid data.

[0242] Exemplarily, invalid data may be all 0s or all Fs.

[0243] Exemplarily, if the target data is not invalid data, step 509 is executed; if the target data is invalid data, step 503 is executed.

[0244] Step 509: Obtain target data.

[0245] Step 510: Update the control flag.

[0246] From the above, it can be seen that in the process of acquiring target data, the second acquisition module judges and processes multiple conditions such as bit map information, acquisition operation execution times, control identification bits, and whether the target data is invalid times. It not only realizes the accurate acquisition of target data, but also realizes the screening and filtering of target data. Data that is not updated or invalid within the acquisition cycle of the second acquisition module is not acquired, thereby reducing the computational complexity of target data transmission; after acquiring the target data, it can not only package and report the target data, but also update the control identification bits in real time, thereby laying the foundation for the efficient collaborative data processing operations of the first acquisition module and the second acquisition module based on shared storage space.

[0247] Based on the foregoing embodiments, the embodiments of the present application also provide a data processing architecture diagram of a first acquisition module and a second acquisition module. Figure 6 The schematic diagram of the architecture of the data processing of the first acquisition module and the second acquisition module provided in the embodiment of the present application is as follows: Figure 6 As shown, the architecture includes:

[0248] The first acquisition module 601, the second acquisition module 602 and the shared storage unit 603; wherein the shared storage unit 603 may include a first type storage space 6031 and a second type storage space 6032; the first type storage space 6031 includes a first storage area 60311 and a second storage area 60312; the second type storage space 6032 may include a third storage area 60321 and a fourth storage area 60322. The first storage area 60311 and the third storage area 60321 are used to store metadata of the corresponding space, each metadata may include bitmap information and control identification, etc.; the second storage area 60312 and the fourth storage area 60322 are used to store target data of the first data type and target data of the second data type.

[0249] Exemplarily, the first acquisition module 601 can obtain metadata of the first storage area 60311, and exemplary, determine a storage unit that can be used to store target data through the bitmap information in the metadata, and can also obtain a control identifier to process the processed data based on the control identifier to obtain the target data; after the first acquisition module 601 completes the data storage operation, the data identifier and the bitmap information can be updated.

[0250] Exemplarily, the second acquisition module can also obtain the metadata of the first storage area 60311, and determine the storage unit that can read the target data based on the bit map information and the control identifier. After obtaining the target data, the target data can also be packaged and processed, and the bit map information and the control identifier can be updated for the first acquisition module to use when performing data storage operations next time.

[0251] Figure 7 The overall flow chart of the data processing method provided in the embodiment of the present application is as follows. Figure 7 As shown, the process may include steps 701 to 704:

[0252] Step 701: Divide the shared storage space according to data types.

[0253] Exemplarily, before dividing the shared storage space, the data type may be first obtained, and then the shared storage space may be divided according to the number of data types and the collection requirements for various types of data. Each type of data may occupy a continuous storage space in the shared storage space, and the sub-storage spaces divided for different types of data may be the same, and sufficient storage capacity may be reserved for each data in the sub-storage space.

[0254] Step 702: Transmit target data based on the bitmap information and the control flag.

[0255] Exemplarily, as described in the aforementioned embodiments, the first acquisition module 601 and the second acquisition module 602 can obtain bitmap information and control identifiers from each metadata of the shared storage space, and perform corresponding data processing operations based on the bitmap information and control identifiers. After the data processing operation is completed, the bitmap information and control identifiers are updated as a basis for the other module to perform the data processing operation.

[0256] Step 703: Acquire valid target data and report the valid target data.

[0257] Step 704: Release storage space.

[0258] Exemplarily, when a certain type of data does not need to be collected, the storage space corresponding to the specified type of data can be completely released; after the second collection module obtains the target data, the storage unit storing the target data can also be initialized, thereby achieving the purpose of releasing the storage space. Exemplarily, after the storage space is released, the bitmap information associated with the storage space can also be updated.

[0259] From the above, it can be seen that, based on various metadata, the first acquisition module 601, the second acquisition module 602 and the shared storage space 603 can realize the transmission of target data from the first acquisition module 601 to the second acquisition module 602, and the first acquisition module 601 and the second acquisition module 602 can manage their respective data processing operations based on the bitmap information and control identifiers in the metadata, thereby realizing efficient data transmission based on the shared storage space and reducing the resource consumption of control information transmission during data transmission.

[0260] The data processing method provided in the embodiment of the present application can also be applied to the collection and transmission of indicator data in a network system. Figure 8 A schematic diagram of the architecture of the data processing method provided in an embodiment of the present application in an indicator data acquisition and transmission frame.

[0261] exist Figure 8 In the data plane development kit (DPDK), the data plane 801 is used to collect the flow data of VM802, network card 803 and other network interface devices, and obtain the indicator data such as the first flow, the second flow and other flows respectively. For example, the first flow can be the VM flow, and the second flow can be the network card flow.

[0262] Exemplarily, after obtaining the above-mentioned various data, the DPDK data plane 801 can determine the sub-storage space in the shared storage space 804 for storing each type of information according to the type information of the various data and its corresponding metadata, and write the above-mentioned various traffic data into the sub-storage space of the shared storage space 804 based on the bitmap information and control identifier in the metadata, and then update the bitmap information and control identifier.

[0263] Exemplarily, the collector 805 can determine the target data type it needs to obtain based on the data collection requirement information, and determine the metadata corresponding to the target data type in the shared storage space 804 based on the target data type, and then obtain the target data based on the bitmap information and control identifier in the metadata, and then update the bitmap information and control identifier.

[0264] From the above, it can be seen that when the data processing method provided in the embodiment of the present application is applied to traffic data collection, whether it is the DPDK data plane 801 or the collector 805, the control identifier and bitmap information in the metadata can be obtained through a single operation on the metadata, thereby improving the efficiency of control identifier acquisition, realizing fast and accurate execution of data processing operations, and thus realizing efficient collection of traffic indicator data.

[0265] In practical applications, the collection of indicator data usually includes methods such as indicator file parsing, socket communication and shared memory.

[0266] In the method based on the index file analysis, after the DPDK data plane 801 collects various index data, it writes it into a file in a certain format, and the collector 805 reads the data in the file for data analysis. This method can reduce the coupling between the DPDK data plane 801 and the collector 805, and the file is more readable and analyzable, can directly reflect the index results, and is easy to view and analyze.

[0267] In the socket communication-based method, the transmission and interaction of various indicator data can be realized through standard socket interfaces between the DPDK data plane 801 and the shared storage space 804, and between the shared storage space 804 and the collector 805. The standardized socket interface is easy to connect with different collectors 805, and the indicator data can also be directly converted according to the indicator storage format. Therefore, this method has low development difficulty and performance overhead;

[0268] In the shared memory-based method, a shared memory space 804 is introduced between the DPDK data plane 801 and the collector 805. The DPDK data plane 801 collects the indicator data and stores it in the shared memory space 804 for the collector 805 to read. In the process of implementing this method, the indicator data usually needs to be transmitted through a data queue or other means. In this method, the existence of the shared memory space 804 reduces unnecessary indicator data copying and transmission, so the overhead is low and the collection performance is good.

[0269] However, for the method based on indicator file parsing, this method needs to operate the file to realize file content reading and parsing, and the operation of the file usually needs to rely on file input and output operations and data string operations, etc. Therefore, in the scenario of massive indicator data collection, the performance of file parsing is significantly reduced, and under the premise of collector 805 resource constraints, the performance of this method is low and cannot meet the indicator data collection requirements.

[0270] For methods based on socket communication, indicator data collection through sockets can directly parse the indicator data stream to obtain the collection indicators. However, the socket transmission process involves operations such as message encapsulation, serialization, and protocol stack processing. These operations will inevitably cause transmission and processing delays and consume bandwidth resources, thereby limiting the collection performance.

[0271] For the shared memory-based method, most of the acquisition solutions in practical applications use shared memory technology to acquire indicator data. Among them, one type of solution is to directly write the indicator data into the shared memory for reading by the collector 805. This type of solution uses the shared memory as a data path to directly transmit data, and the control path is processed through other channels, which has performance overhead; another type of solution puts the indicator data into the shared memory queue and is processed and analyzed by the collector 805 in a timely manner to obtain the indicator data that needs to be collected. This type of solution has high performance resource requirements for the collector 805 and is not easy to deploy in actual scenarios.

[0272] The data processing method provided in the embodiment of the present application divides the shared storage space 804 in advance according to the data type information to obtain at least one sub-storage space. Metadata corresponding to the data type information is also set in each sub-storage space. The metadata and the actually transmitted data are stored together in the sub-storage space. After the metadata-based addressing is completed, the DPDK data plane 801 and the collector 805 can realize the dual establishment of the data transmission channel and the control signal channel through a single addressing operation; and the zero-copy advantage of the shared storage space 804 can also greatly reduce the collection overhead and improve the collection efficiency; the bidirectional control of the control identifier and the bitmap information by the DPDK data plane 801 and the collector 805 can not only reduce the control overhead, but also realize the coordination of the indicator data processing by the DPDK data plane 801 and the collector 805.

[0273] Based on the above embodiments, the present application also provides a first acquisition module 601. Figure 9a A first structural diagram of the first acquisition module 601 provided in an embodiment of the present application is shown in FIG. Figure 9a As shown, the first acquisition module 601 may include: a first determination unit 901, a first acquisition unit 902 and a first processing unit 903; wherein:

[0274] The first determining unit 901 is used to determine the first address information; wherein the first address information includes the address information of the first storage space; the first storage space includes a storage space in the shared storage space where the first acquisition module can perform a data storage operation on the data to be processed; the first storage space stores control information;

[0275] A first acquiring unit 902, configured to determine control information based on the first address information;

[0276] The first processing unit 903 is used to process the data to be processed collected by the first collection module based on the control information to obtain target data;

[0277] The first processing unit 903 is used to perform a data storage operation based on the first address information and the target data, so that the second acquisition module can perform a data processing operation on the target data.

[0278] In one implementation, the first acquisition unit 902 is used to acquire first information; wherein the first information includes storage status information of the storage unit in the second storage space; the second storage space includes a storage space associated with type information of the data to be processed in the shared storage space;

[0279] The first determining unit 901 is configured to determine first address information based on first information.

[0280] In one implementation, the first acquisition unit 902 is configured to acquire type space association information; wherein the type space association information includes information on an association relationship between data type information and address information of a shared storage space;

[0281] The first determining unit 901 is used to determine the second address information based on the type information of the data to be processed and the type space association information; wherein the second address information includes the address information of the second storage space;

[0282] The first acquiring unit 902 is configured to acquire first information based on the second address information.

[0283] In one implementation, the first information includes bitmap information associated with the second storage space; the first processing unit 903 is used to update the first information after the data storage operation is completed.

[0284] In one implementation, the first acquisition unit 902 is configured to acquire second information based on the first address information; wherein the second information includes data processing configuration information associated with the storage unit of the first storage space; the data processing configuration information includes at least one configuration information;

[0285] The first determining unit 901 is configured to determine control information based on each piece of configuration information in the second information when the configuration information in the second information is updated by the second acquisition module.

[0286] In one implementation, the first processing unit 903 is configured to update the second information after the data storage operation is completed.

[0287] It should be noted that the first determining unit 901 , the first acquiring unit 902 and the first processing unit 903 may be implemented by the first processor in the first acquisition module 601 .

[0288] Figure 9b Another structural diagram of the first acquisition module 601 provided in the embodiment of the present application is as follows: Figure 9b As shown, the first acquisition module 601 may include a first processor 904 and a first memory 905, wherein: the first memory 905 stores a computer program that can be run on the first processor 904, and when the first processor 904 executes the computer program, it implements any of the previous data processing methods applied to the first acquisition module.

[0289] The first memory 905 may be a volatile memory, such as RAM; or a non-volatile memory, such as ROM, flash memory, HDD or SSD; or a combination of the above types of memories, and provide instructions and data to the processor.

[0290] Fig.10a A first structural diagram of the second acquisition module 602 provided in an embodiment of the present application is shown in FIG. Fig.10a As shown, the second acquisition module 602 may include: a second determination unit 1001, a second acquisition unit 1002 and a second processing unit 1003; wherein:

[0291] The second determination unit 1001 is used to determine the first address information; wherein the first address information includes the address information of the first storage space; the first storage space includes a storage space in the shared storage space where the second acquisition module can perform data processing operations on the target type data; the first storage space stores control information and target data; the target data is obtained by the first acquisition module processing the to-be-processed data collected by the first acquisition module based on the control information; the control information is determined by the first acquisition module based on the first address information; the data type of the target data is the target type.

[0292] The second acquisition unit 1002 is used to acquire target data based on the first address information.

[0293] The second processing unit 1003 is used to perform a data processing operation on the target data based on the first address information.

[0294] In one implementation, the second acquisition unit 1002 is used to acquire first information; wherein the first information includes storage status information of the storage unit in the second storage space; the second storage space includes storage space associated with type information of the target data in the shared storage space;

[0295] The second determining unit 1001 is configured to determine first address information based on the first information.

[0296] In one embodiment, the first information includes bitmap information associated with the second storage space;

[0297] The second processing unit 1003 is used to update the first information after the data processing operation is completed.

[0298] In one implementation, the second acquisition unit 1002 is used to acquire type space association information; wherein the type space association information includes information on the association relationship between the data type information and the address information in the shared storage space;

[0299] The second determination unit 1001 is used to determine the second address information based on the type information of the target type and the type space association information; and obtain the first information based on the second address information, wherein the second address information includes the address information of the second storage space.

[0300] In one embodiment, the second acquisition unit 1002 is used to acquire the number of executions of the data processing operation; based on the first address information, acquire the second information; wherein the second information includes data processing configuration information associated with the storage unit of the first storage space; the data processing configuration information includes at least one configuration information;

[0301] The second processing unit 1003 is used to perform 90 data processing operations on the target data when the specified conditions are met; wherein the specified conditions include that the number of executions is less than or equal to a specified threshold and the configuration information in the second information is updated by the first acquisition module after the data storage operation is completed.

[0302] In one implementation, the second processing unit 1003 is configured to update the second information after the data processing operation is completed.

[0303] It should be noted that the second determining unit 1001 , the second acquiring unit 1002 and the second processing unit 1003 may be implemented by the second processor in the second acquisition module 602 .

[0304] Fig.10b Another structural diagram of the second acquisition module 602 provided in the embodiment of the present application. Fig.10b As shown, the second acquisition module 602 may include a second processor 1004 and a second memory 1005, wherein: the second memory 1005 stores a computer program that can be run on the second processor 1004, and when the second processor 1004 executes the computer program, it implements any of the previous data processing methods applied to the second acquisition module 602.

[0305] The second memory 1005 may be a volatile memory, such as RAM; or a non-volatile memory, such as ROM, flash memory, HDD or SSD; or a combination of the above types of memories, and provide instructions and data to the processor.

[0306] The present application also provides a data processing system, which includes: Figure 9a Or the first acquisition module 601 shown in 9b, and Fig.10aOr the second acquisition module 602 shown in 10b.

[0307] An embodiment of the present application further provides a computer-readable storage medium, in which a computer program is stored. When the computer program is executed by a processor, the data processing method applied to the first acquisition module 601 or the second acquisition module 602 as provided in the above-mentioned embodiment can be implemented.

[0308] The above description of various embodiments tends to emphasize the differences between the various embodiments. The same or similar aspects can be referenced to each other, and for the sake of brevity, they will not be repeated herein.

[0309] The methods disclosed in the various method embodiments provided in this application can be arbitrarily combined without conflict to obtain new method embodiments.

[0310] The features disclosed in the various product embodiments provided in this application can be arbitrarily combined without conflict to obtain new product embodiments.

[0311] The features disclosed in the various method or device embodiments provided in this application can be arbitrarily combined without conflict to obtain new method embodiments or device embodiments.

[0312] It should be noted that the above-mentioned computer-readable storage medium can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a magnetic random access memory (FRAM), a flash memory (Flash Memory), a magnetic surface memory, an optical disc, or a compact disc read-only memory (CD-ROM) and other memories; it can also be various electronic devices including one or any combination of the above-mentioned memories, such as mobile phones, computers, tablet devices, personal digital assistants, etc.

[0313] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.

[0314] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0315] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus necessary general hardware nodes, and of course can also be implemented by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in each embodiment of the present application.

[0316] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0317] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0318] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0319] The above are only preferred embodiments of the present application, and are not intended to limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A data processing method, characterized in that: The method is applied to a first acquisition device; the method comprises: Determine first address information; wherein the first address information includes address information of a first storage space; the first storage space includes a storage space in a shared storage space where the first acquisition device can perform a data storage operation on the data to be processed; the first storage space stores control information; Determining the control information based on the first address information; Processing the data to be processed collected by the first collection device based on the control information to obtain target data; wherein the target data includes the data to be processed and additional information; the additional information at least includes the collection time of the data to be processed; Based on the first address information and the target data, the data storage operation is performed so that the second acquisition device can perform a data processing operation on the target data.

2. The method according to claim 1, characterized in that: The determining of the first address information includes: Acquire first information; wherein the first information includes storage status information of a storage unit in a second storage space; the second storage space includes a storage space in the shared storage space associated with the type information of the data to be processed; Based on the first information, the first address information is determined.

3. The method according to claim 2, characterized in that The obtaining of the first information includes: Acquire type space association information; wherein the type space association information includes information on the association relationship between the data type information and the address information of the shared storage space; Determine second address information based on the type information of the data to be processed and the type space association information; wherein the second address information includes address information of the second storage space; Based on the second address information, the first information is acquired.

4. The method according to claim 2, characterized in that: The first information includes bitmap information associated with the second storage space; and the method further includes: After the data storage operation is completed, the first information is updated.

5. The method according to claim 1, characterized in that The determining the control information based on the first address information includes: Based on the first address information, obtaining second information; wherein the second information includes data processing configuration information associated with the storage unit of the first storage space; the data processing configuration information includes at least one configuration information; In the case where the configuration information in the second information is updated by the second collecting device, the control information is determined based on each of the configuration information in the second information.

6. A data processing method, characterized in that: The method is applied to a second acquisition device, and the method comprises: Determine first address information; wherein the first address information includes address information of a first storage space; the first storage space includes a storage space in a shared storage space where the second acquisition device can perform data processing operations on target type data; control information and target data are stored in the first storage space; the target data is obtained by the first acquisition device processing the data to be processed collected by the first acquisition device based on the control information; the control information is determined by the first acquisition device based on the first address information; the data type of the target data is the target type; the target data includes the data to be processed and additional information; the additional information includes at least the acquisition time of the data to be processed; Based on the first address information, the data processing operation is performed on the target data.

7. The method according to claim 6, characterized in that The determining of the first address information includes: Acquire first information; wherein the first information includes storage status information of a storage unit in a second storage space; the second storage space includes a storage space in the shared storage space associated with the type information of the target data; Based on the first information, the first address information is determined.

8. The method according to claim 7, characterized in that The obtaining of the first information includes: Acquire type space association information; wherein the type space association information includes information on the association relationship between the data type information and the address information in the shared storage space; Determine second address information based on the type information of the target type and the type space association information; wherein the second address information includes address information of the second storage space; Based on the second address information, the first information is acquired.

9. The method according to claim 6, characterized in that The performing a data processing operation on the target data based on the first address information includes: Obtaining the number of executions of the data processing operation; Based on the first address information, obtaining second information; wherein the second information includes data processing configuration information associated with the storage unit of the first storage space; the data processing configuration information includes at least one configuration information; When specified conditions are met, the data processing operation is performed on the target data; wherein the specified conditions include that the number of executions is less than or equal to a specified threshold, and the configuration information in the second information is updated by the first acquisition device after the data storage operation is completed.

10. A first collection device, characterized in that: The first acquisition device includes a first determination unit, a first acquisition unit and a first processing unit; wherein: The first determining unit is used to determine the first address information; wherein the first address information includes address information of the first storage space; the first storage space includes a storage space in the shared storage space where the first acquisition device can perform a data storage operation on the data to be processed; the first storage space stores control information; The first acquiring unit is used to determine the control information based on the first address information; The first processing unit is used to process the data to be processed collected by the first collection device based on the control information to obtain target data; wherein the target data includes the data to be processed and additional information; the additional information at least includes the collection time of the data to be processed; The first processing unit is used to perform the data storage operation based on the first address information and the target data, so that the second acquisition device can perform a data processing operation on the target data.

11. A first collection device, characterized in that: The first acquisition device includes a first processor and a first memory; wherein: the first memory stores a computer program that can be run on the first processor, and when the first processor executes the computer program, the data processing method according to any one of claims 1 to 5 is implemented.

12. A second collection device, characterized in that: The second acquisition device includes a second determination unit, a second acquisition unit and a second processing unit; wherein: The second determination unit is used to determine the first address information; wherein the first address information includes the address information of the first storage space; the first storage space includes a storage space in the shared storage space where the second acquisition device can perform data processing operations on target type data; the first storage space stores control information and target data; the target data is obtained by the first acquisition device processing the data to be processed collected by the first acquisition device based on the control information; the control information is determined by the first acquisition device based on the first address information; the data type of the target data is the target type; the target data includes the data to be processed and additional information; the additional information at least includes the acquisition time of the data to be processed; The second acquisition unit is used to acquire the target data based on the first address information; The second processing unit is used to perform the data processing operation on the target data based on the first address information.

13. A second collection device, characterized in that: The second acquisition device includes a second processor and a second memory; wherein: the second memory stores a computer program that can be run on the second processor, and when the second processor executes the computer program, the data processing method according to any one of claims 6 to 9 is implemented.

14. A data processing system, characterized in that: The data processing system comprises a shared storage space, a first acquisition device as claimed in claim 10 or 11, and a second acquisition device as claimed in claim 12 or 13.

15. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the data processing method according to any one of claims 1 to 5 or claims 6 to 9 can be implemented.

Images