Data processing method and device

Abstract

The invention provides a data processing method and device. The method comprises the steps of obtaining to-be-processed data continuously output to a cache; analyzing the to-be-processed data, and extracting first data in the to-be-processed data in response to the condition that the to-be-processed data meets a set data compression condition; the first data at least comprises constant information and variable information capable of representing loop characteristics; generating second data based on the first data; the second data at least comprises loop structure data corresponding to constant information and variable information; the loop structure data is used for representing loop feature information of elements appearing repeatedly in the first data; and storing the second data.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a data processing method and apparatus. Background Technology

[0002] In current data processing, systems typically generate large amounts of log data, which often contains duplicate information. This duplicate information not only consumes a significant amount of storage space but also reduces program efficiency. Therefore, current data processing suffers from log data redundancy, leading to low storage efficiency, and the need to retrieve large amounts of data for log data analysis further reduces analysis efficiency. Summary of the Invention

[0003] This application provides a data processing method, apparatus, electronic device, and storage medium.

[0004] This application provides a data processing method, the method comprising: obtaining data to be processed continuously output to a cache; analyzing the data to be processed, and extracting first data from the data to be processed in response to the data to be processed meeting a set data compression condition; the first data including at least constant information and variable information capable of characterizing cyclic features; generating second data based on the first data; the second data including at least cyclic structure data corresponding to the constant information and variable information; the cyclic structure data being used to represent cyclic feature information of recurring elements in the first data; and storing the second data.

[0005] According to one embodiment of this application, the step of extracting first data from the data to be processed in response to the data to be processed meeting the set data compression conditions includes: in response to the existence of the same text structure information in a consecutive first number of data to be processed, extracting the text structure information as constant information in the first data; the same text structure information represents the content and / or the description content with the same position in the loop process.

[0006] According to one embodiment of this application, the step of extracting the first data from the data to be processed in response to the data to be processed meeting the set data compression conditions includes: in response to the frequency of numerical changes at a set field position in a second consecutive number of data to be processed exceeding a preset threshold, extracting the name of the changed data and the corresponding set of values ​​as the variable information; the name of the changed data represents the variable identifier of the changed data during the loop, and the set of values ​​represents the value of the changed data during the loop.

[0007] According to one embodiment of this application, the method further includes: determining the data to be processed as cyclic data in response to the data to be processed meeting a set data compression condition; obtaining a timestamp corresponding to each piece of cyclic data; determining the cycle start time and cycle end time corresponding to the cyclic data based on the timestamp corresponding to each piece of cyclic data in response to the next piece of cyclic data not meeting the set data compression condition; and generating the second data based on the first data in the cyclic data, the cycle start time, and the cycle end time.

[0008] According to one embodiment of this application, generating second data based on the first data includes: in response to at least two pieces of first data including constant information, determining a constant data structure corresponding to the constant information; the constant data structure characterizing the fixed description content of multiple pieces of first data in a loop process; in response to at least two pieces of first data including variable information, determining a variable data structure based on the variable information and the loop rule corresponding to the variable information; the variable data structure characterizing the change content of multiple pieces of first data in a loop process; the change content at least includes the name of the change data and the corresponding set of values; and sorting the constant data structure and the variable data structure based on the chronological order of the data to be processed to obtain the second data.

[0009] According to one embodiment of this application, determining the variable data structure based on the variable information and the corresponding loop rule includes: determining the field position of the variable information in the data to be processed; determining the change pattern of the data at the field position with the timestamp of the data to be processed; determining the variable data structure based on the change pattern of the timestamp and the variable information in response to the change pattern satisfying the loop rule; and determining the number of loops of the variable information in the loop process in response to the change pattern not satisfying the loop rule, and determining the variable data structure based on the variable information and the number of loops.

[0010] According to one embodiment of this application, after storing the second data, the method further includes: deleting pending data associated with the second data from the cache.

[0011] According to one embodiment of this application, the method further includes: in response to the change pattern satisfying the cyclic rule, determining the time period of the data to be processed, wherein the time period is a fixed interval between adjacent timestamps; and recording the time period in the second data to represent the cyclic characteristics of the first data in the cyclic process.

[0012] According to one embodiment of this application, the method further includes: in response to the change pattern not satisfying the cyclic rule, recording the number of cycles in the second data to represent the cyclic characteristics of the first data during the cyclic process.

[0013] This application also provides a data processing apparatus, the apparatus comprising: an acquisition module for acquiring data to be processed continuously output to a cache; a processing module for analyzing the data to be processed and, in response to the data to be processed meeting a set data compression condition, extracting first data from the data to be processed; the first data including at least constant information and variable information capable of characterizing cyclic features; a generation module for generating second data based on the first data; the second data including at least cyclic structure data corresponding to the constant information and variable information; the cyclic structure data representing cyclic feature information of recurring elements in the first data; and a storage module for storing the second data. Attached Figure Description

[0014] The above and other objects, features, and advantages of exemplary embodiments of this application will become readily apparent from the following detailed description taken in conjunction with the accompanying drawings. Several embodiments of this application are illustrated in the drawings by way of example and not limitation, in which: In the accompanying drawings, the same or corresponding reference numerals indicate the same or corresponding parts.

[0015] Figure 1 This application illustrates a schematic diagram of the data processing method provided in an embodiment. Figure 1 ; Figure 2 This application illustrates a schematic diagram of the data processing method provided in an embodiment. Figure 2 ; Figure 3 This illustration shows an application scenario diagram of the data processing method provided in an embodiment of this application; Figure 4 An optional schematic diagram of the data processing apparatus provided in an embodiment of this application is shown; Figure 5 A schematic diagram of the composition structure of the electronic device provided in the embodiments of this application is shown. Detailed Implementation

[0016] To make the objectives, features, and advantages of this application more apparent and understandable, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0017] In the following description, references are made to “some embodiments,” which describe a subset of all possible embodiments. However, it is understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0018] In the following description, the terms "first" and "second" are used merely to distinguish similar objects and do not represent a specific ordering of objects. It is understood that "first" and "second" may be interchanged in a specific order or sequence where permitted, so that the embodiments of this application described herein can be implemented in an order other than that illustrated or described herein.

[0019] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0020] The processing flow of the data processing method provided in the embodiments of this application will be described. See [link to relevant documentation]. Figure 1 , Figure 1 This is a schematic diagram of the data processing method provided in the embodiments of this application. Figure 1 , will combine Figure 1 Steps S101-S104 are explained below.

[0021] Step S101: Obtain the data to be processed that is continuously output to the cache.

[0022] Step S102: Analyze the data to be processed. In response to the data to be processed meeting the set data compression conditions, extract the first data from the data to be processed. The first data includes at least constant information and variable information that can characterize the cyclic features.

[0023] In some embodiments, the data to be processed may include: log records, transaction records, sensor data, etc. This application embodiment does not limit the specific data to be processed. Data compression conditions may include: data repetition characteristics, thresholds for cyclic characteristics, etc. Data compression conditions can be used to determine whether there is extractable regularity information in the data to be processed, such as the existence of identical text structure information appearing consecutively a certain number of times, or the frequency of numerical changes at a specific field position exceeding a preset threshold. The first data may include constant information and variable information with cyclic characteristics extracted from the data to be processed. Constant information may include: data elements that remain unchanged during the cycle, such as fixed text descriptions, fixed identifiers, etc. Variable information may include: data elements that change during the cycle, such as the name of a variable and its corresponding set of values, etc.

[0024] Step S103: Generate second data based on the first data; the second data includes at least loop structure data corresponding to constant information and variable information; the loop structure data is used to represent the loop feature information of repeated elements in the first data.

[0025] Step S104: Store the second data.

[0026] In some embodiments, the second data may include data generated after compression. Specifically, the second data may include loop structure data corresponding to constant and variable information. The second data can replace the original data to be processed for storage and transmission. The loop structure data can be used to represent the loop characteristic information of recurring elements in the first data, such as the start time, end time, number of loops, time interval, name of changing data, and set of values. This application does not limit the specific loop structure data.

[0027] As an example, taking the system's logging process as an example, logs are continuously output to a cache. First, the system obtains the data to be processed from the cache. Next, the system analyzes the data to be processed and determines that there are duplicate login records. These login records have similar text structures (such as "user login attempt") and changing user IDs and timestamps. The system determines that the duplicate login records in the data to be processed meet the data compression conditions, and extracts constant information (such as the text structure "user login attempt") and variable information (such as user ID) from the data to be processed. Then, the system generates second data based on the extracted first data. In the second data, the cyclic structure data includes cyclic feature information representing login attempts, such as the user ID corresponding to each login during the time from the start of logging to the end of logging. Finally, the system stores the second data.

[0028] The method in this application embodiment effectively reduces the amount of data that needs to be stored and improves storage efficiency by extracting first data from the data to be processed and generating second data containing cyclic structure data based on the first data. This not only reduces storage space usage but also simplifies the subsequent data analysis process without altering the current output log process. The cyclic pattern can be determined using the second data without processing a large amount of repetitive information, thereby improving the efficiency of data analysis.

[0029] In some embodiments, after storing the second data in step S104, the method further includes: deleting the pending data associated with the second data from the cache.

[0030] As an example, the system continuously receives pending data being output to the cache in real time, as follows: 2025-03-14 14:32:00 Loop through the records in the table: A1=1 2025-03-14 14:32:01 Loop through the records in the table: A1=3 2025-03-14 14:32:02 Loop through the records in the table: A1=6 2025-03-14 14:32:03 Loop through the records in the table: A1=10 To analyze the data to be processed, first determine whether there are a certain number (e.g., 2) of data to be processed that have the same text structure information. The text structure information can be a fixed text pattern such as "looping through the records in the table". If this condition is met, the text structure information is extracted as constant information in the first data.

[0031] At the same time, it is determined whether the frequency of value changes in the position of the set field in a certain number of consecutive data (such as 2) exceeds a preset threshold (for example, the value of the position of the query result changes every time). If so, the name of the changed data, namely A1, and the corresponding set of values ​​are extracted as variable information in the first data.

[0032] The second data is generated based on the extracted first data. Specifically, the constant data structure corresponding to the constant information is first determined, that is, the fixed description content such as log text description is determined; then, based on the loop rules corresponding to the variable information (such as the law of sequentially increasing values), the variable data structure is determined, including the name of the changed data and the corresponding set of values.

[0033] Finally, according to the chronological order of the data to be processed, the constant data structure and the variable data structure are integrated to form the second data, for example, "A loop: from 2025-03-14 14:32:00 to 2025-03-14 14:32:03, each second (or 4 times): loop through the table records: {A1}={1, 3, 6, 10}", and this second data is stored to compress the original data to be processed. At the same time, the original data to be processed associated with the second data is deleted from the cache, that is, the first 3 data in the data to be processed are deleted to release the cache space.

[0034] The method in this application embodiment deletes the associated pending data in the cache after storing the second data, which can release the cache space in a timely manner, avoid unnecessary data redundancy, effectively reduce the amount of data that needs to be stored, and improve storage efficiency.

[0035] In some embodiments, step S102, in response to the data to be processed meeting the set data compression conditions, extracting the first data from the data to be processed, includes: in response to the existence of the same text structure information in a consecutive first number of data to be processed, extracting the text structure information as constant information in the first data; the same text structure information represents the content and / or the description content with the same position in the loop process.

[0036] For an example of identical descriptions within a loop, the data acquisition system reads sensor data every minute and outputs a log. The log format is "[timestamp] Read sensor data: Temperature = [temperature value], Humidity = [humidity value]". Specifically, the following log entries are output to the cache: Sensor data read at 14:32:00 on March 14, 2025: Temperature = 25℃, Humidity = 60%. Sensor data read at 14:33:00 on March 14, 2025: Temperature = 26℃, Humidity = 62%. Sensor data read at 14:34:00 on March 14, 2025: Temperature = 25℃, Humidity = 61%. After obtaining the cached logs, the system begins analysis. The first quantity can include a pre-defined number of consecutive data entries, such as 3 or 5 consecutive entries. For example, the first quantity can be twice. The system determines that the same text structure information, "read sensor data," exists in 3 consecutive log entries. This content is completely consistent in each log entry, and there are identical descriptive contents throughout the loop. The system extracts this text structure information as constant information in the first data set.

[0037] For an example of recurring descriptions within a loop, the data acquisition system reads sensor data every minute and outputs a log. The log format is "[timestamp] Read sensor data: Temperature = [temperature value], Record number of reads: Number of reads = [number of reads]". Specifically, the following log entries are output to the cache: 2025-03-14 14:32:00 Read sensor data: Temperature = 25℃, Number of reads recorded: 1 2025-03-14 14:33:00 Sensor data read: Temperature = 26℃, Number of reads recorded: 2 2025-03-14 14:34:00 Read sensor data: Temperature = 25℃, Number of reads recorded: Number = 3 After obtaining the cached logs, the system begins analysis. The first quantity can include a pre-defined number of consecutive data entries, such as 3 or 5 consecutive entries. For example, the first quantity could be 3 times. The system determines that the "number of reads recorded" description is in the same position in each log entry; that is, the log records the number of reads after recording the sensor data read. It also determines that the same text structure information, "number of reads recorded," exists in three consecutive log entries, with this content in the exact same position in each log entry. This identical description appears throughout the loop. This text structure information is extracted as constant information in the first data set.

[0038] The method in this application extracts constant information by determining whether there is the same text structure information in a continuous first number of data to be processed. This can identify fixed descriptive content in the log, effectively reduce the amount of data that needs to be stored, and improve storage efficiency.

[0039] In some embodiments, step S102, in response to the data to be processed meeting the set data compression conditions, extracting the first data from the data to be processed, includes: in response to the frequency of numerical changes at the position of a set field in a consecutive second number of data to be processed exceeding a preset threshold, extracting the name of the changed data and the corresponding set of values ​​as variable information; the name of the changed data represents the variable identifier of the changed data during the loop, and the set of values ​​represents the value of the changed data during the loop.

[0040] As an example, the data acquisition system reads sensor data every minute and outputs a log. The log format is "[timestamp] Read sensor data: Temperature = [temperature value], Humidity = [humidity value]". Specifically, the following log entries are output to the cache: Sensor data read at 14:32:00 on March 14, 2025: Temperature = 25℃, Humidity = 60%. Sensor data read at 14:33:00 on March 14, 2025: Temperature = 26℃, Humidity = 62%. Sensor data read at 14:34:00 on March 14, 2025: Temperature = 25℃, Humidity = 61%. After obtaining the cached logs, the system begins analysis. The second quantity can include a pre-defined number of consecutive data entries, such as 3 or 5 consecutive entries. For example, the second quantity could be 3 times, determining that the frequency of change in the values ​​of "temperature" and "humidity" in 3 consecutive log entries exceeds a preset threshold of once per minute. "Temperature" and "humidity" are extracted as variable names, and the corresponding numerical sets are collected as variable information, specifically {25℃, 26℃, 25℃} and {60%, 62%, 61%}, respectively.

[0041] The method in this application extracts the names and sets of values ​​of changing data as variable information. By monitoring the frequency of value changes at designated field positions, it can determine the dynamic changes in the cyclic process. It records the changes in cyclic data, thereby improving data analysis efficiency, reducing the processing of irrelevant data, identifying variable information in the log, effectively reducing the amount of data that needs to be stored, and improving storage efficiency.

[0042] In some embodiments, the data processing method further includes: determining the data to be processed as cyclic data in response to the data to be processed meeting the set data compression conditions; obtaining a timestamp corresponding to each cyclic data; determining the cycle start time and cycle end time corresponding to the cyclic data based on the timestamp corresponding to each cyclic data in response to the next data in the cyclic data not meeting the set data compression conditions; and generating second data based on the first data in the cyclic data, the cycle start time, and the cycle end time.

[0043] In this embodiment, the cyclic data may include: a data sequence with cyclic characteristics. The timestamp may include: time information corresponding to each data item. The cycle start time may include: the timestamp of the first data item in the cyclic data. The cycle end time may include: the timestamp of the last data item in the cyclic data.

[0044] As an example, the system generated the following logs during operation: 2025-03-14 14:32:00 Sensor A value: 10 2025-03-14 14:32:01 Sensor A value: 12 2025-03-14 14:32:02 Sensor A value: 11 2025-03-14 14:32:03 Sensor B value: 13 After obtaining these cached logs, the system begins analysis. First, it determines whether the logs meet the set data compression conditions. In this example, the data compression condition is that at least two consecutive data entries have the same text structure. The system detects that the text structure of the first three log entries is "Value of Sensor A: [Value]", which meets the data compression condition; therefore, the first three log entries are determined to be cyclic data.

[0045] Next, the timestamps of each loop data entry are obtained: 2025-03-14 14:32:00, 2025-03-14 14:32:01, and 2025-03-14 14:32:02. When it is determined that the next data entry, with timestamp 2025-03-14 14:32:03 and content "Value of Sensor B: 13", does not meet the data compression conditions, based on the previously obtained timestamps, the loop start time is determined to be 2025-03-14 14:32:00, and the loop end time is determined to be 2025-03-14 14:32:02.

[0046] Extract the variable information from the cyclic data, specifically the set of values ​​{10, 12, 11} from sensor A. Finally, based on the first data point, the loop start time, and the loop end time, generate the second data point. For example, the generated second data point is as follows: A loop: from 2025-03-14 14:32:00 to 2025-03-14 14:32:02, each second (3 times): value of sensor A: {10, 12, 11}.

[0047] The method in this application, after determining that the data to be processed is cyclic data, determines the time range within the cyclic data by obtaining the timestamp of each cyclic data item and determining the start and end times of the cycle at the end of the cycle. This can be used to identify the occurrence time of the cycle during subsequent data analysis, improving analysis efficiency.

[0048] In some embodiments, the processing flow of the data processing method is illustrated. Figure 2 ,like Figure 2 As shown, step S103, which generates second data based on the first data, may specifically include: Step S201: In response to the inclusion of constant information in at least two first data sets, determine the constant data structure corresponding to the constant information.

[0049] Step S202: In response to the inclusion of variable information in at least two first data sets, determine the variable data structure based on the variable information and the corresponding loop rule.

[0050] In some embodiments, determining the variable data structure based on variable information and the corresponding loop rules in step S202 may include: determining the field position of the variable information in the data to be processed; determining the change pattern of the data at the field position with the timestamp of the data to be processed; determining the variable data structure based on the change pattern of the timestamp and the variable information in response to the change pattern satisfying the loop rules; determining the number of loops of the variable information in the loop process in response to the change pattern not satisfying the loop rules, and determining the variable data structure based on the variable information and the number of loops.

[0051] As an example, the system generates the following logs during operation and stores them in the cache: 2025-03-14 14:32:00 File scan results: Number of files = 5 2025-03-14 14:32:01 File scan results: Number of files = 7 2025-03-14 14:32:02 File scan results: Number of files = 6 2025-03-14 14:32:03 File scan results: Number of files = 8 The system determines the position of the set of values ​​corresponding to the variable identifier "file count" in the data to be processed. For example, its position in the log string is fixed, immediately following "File scan result: file count=". Analyzing the data at this position with the timestamp reveals that the recording time interval is consistently 1 second (i.e., 14:32:00 to 14:32:01, 14:32:01 to 14:32:02, and 14:32:02 to 14:32:03, each with a 1-second interval). This time interval is then compared with a preset loop rule, which requires data to be recorded once per second. The actual recording time interval is confirmed to satisfy this once-per-second loop rule. Therefore, based on the timestamp variation pattern (once per second) and the variable information "file count" and its corresponding set of values, the system determines the variable data structure. For example, the number of files recorded per second is integrated into: {file count} = {5, 7, 6, 8}.

[0052] As an example, the system generates the following logs during operation and stores them in the cache: 2025-03-14 14:32:00 File scan results: Number of files = 5 2025-03-14 14:32:03 File scan results: Number of files = 7 2025-03-14 14:32:06 File scan results: Number of files = 6 2025-03-14 14:32:08 File scan results: Number of files = 8 The system determines the position of the set of values ​​corresponding to the variable identifier "file count" in the variable information within the field of the data to be processed. For example, its position in the log string is fixed, immediately following "File scan result: file count=". The system then determines the pattern of data variation with timestamps at this field position, finding that the recording time intervals are 3 seconds, 3 seconds, and 2 seconds (i.e., 3 seconds from 14:32:00 to 14:32:03, 3 seconds from 14:32:03 to 14:32:06, and 2 seconds from 14:32:06 to 14:32:08), and the time intervals are not fixed. This pattern is compared with the preset loop rule, which requires data to be recorded once per second. The actual recording time interval does not meet the once-per-second loop rule. Therefore, the system determines the number of loops in the variable information process, which is 4 times. Based on the loop count of 4, the variable identifier "file count" in the variable information, and the corresponding set of values, the system determines the variable data structure, for example, consolidating the file count recorded each time into: {file count} = {5, 7, 6, 8}.

[0053] Step S203: Based on the chronological order of the data to be processed, sort the constant data structure and the variable data structure to obtain the second data.

[0054] In this embodiment, the constant data structure may include a format for storing constant information. The variable data structure may include a format for storing variable information. The loop rule may include a pre-defined pattern for describing the changes in variable information. The temporal order may include the order in which the data to be processed was generated.

[0055] As an example, the system generates the following logs during operation and stores them in the cache: 2025-03-14 14:32:00 Loop through the records in the table: A1=10 2025-03-14 14:32:01 Loop through the records in the table: A1=12 2025-03-14 14:32:02 Loop through the records in the table: A1=11 The system extracts the constant information from the first set of data, namely "the record content of the loop query table:", and determines its corresponding constant data structure to represent the fixed description of the loop. Then, it extracts the variable information, namely the variable identifier A1 and its corresponding set of values ​​{10,12,11}, and determines the variable data structure based on the variable information and the loop rule (querying once per second) to represent the changing content in the loop.

[0056] The system sorts the constant data structure and variable data structure according to the time sequence of the data to be processed (from 2025-03-14 14:32:00 to 2025-03-14 14:32:01 to 2025-03-14 14:32:02), and integrates them into the second data. For example, "A loop: from 2025-03-14 14:32:00 to 2025-03-14 14:32:02, each second: loop to query the record content of the table: {A1}={10, 12, 11}".

[0057] The method in this application, by determining constant data structures and variable data structures and sorting them according to time order to generate second data, can determine fixed and variable information in the loop process. This not only reduces storage space usage but also simplifies the subsequent data analysis process. The loop pattern can be determined through the second data without processing a large amount of repetitive information, thereby improving the efficiency of data analysis.

[0058] In some embodiments, the data processing method further includes: in response to the change pattern satisfying a cyclic rule, determining the time period of the data to be processed, wherein the time period is a fixed interval between adjacent timestamps; and recording the time period in the second data to represent the cyclic characteristics of the first data in the cyclic process.

[0059] As an example, the system generates the following logs during operation and stores them in the cache: 2025-03-14 14:32:00 File scan results: Number of files = 5 2025-03-14 14:32:01 File scan results: Number of files = 7 2025-03-14 14:32:02 File scan results: Number of files = 6 2025-03-14 14:32:03 File scan results: Number of files = 8 The system determines the position of the set of values ​​corresponding to the variable identifier "Number of Files" in the data to be processed. For example, its position in the log string is fixed, immediately following "File Scan Result: Number of Files=". Analyzing the data at this position with the timestamp reveals that the recording time interval is consistently 1 second (i.e., 14:32:00 to 14:32:01, 14:32:01 to 14:32:02, and 14:32:02 to 14:32:03, each with a 1-second interval). This time interval is compared with a preset loop rule, which requires data to be recorded once per second. The actual recording time interval is determined to satisfy this once-per-second loop rule. Therefore, the system determines the time period of the data to be processed to be 1 second. This time period is recorded in the second data set to represent the looping characteristics of the first data set during the looping process. For example, the second data generated by the system is as follows: A loop: from 2025-03-14 14:32:00 to 2025-03-14 14:32:03, eachsecond: file scan result: number of files = {5,7,6,8}. Here, each second represents the time period in the second data.

[0060] In some embodiments, the data processing method further includes: in response to the change pattern not satisfying the cyclic rule, recording the number of cycles in the second data to represent the cyclic characteristics of the first data during the cyclic process.

[0061] As an example, the system generates the following logs during operation and stores them in the cache: 2025-03-14 14:32:00 File scan results: Number of files = 5 2025-03-14 14:32:03 File scan results: Number of files = 7 2025-03-14 14:32:06 File scan results: Number of files = 6 2025-03-14 14:32:08 File scan results: Number of files = 8 The system determines the position of the set of values ​​corresponding to the variable identifier "file count" in the variable information within the field of the data to be processed. For example, its position in the log string is fixed, immediately following "File scan result: file count=". The system then determines the pattern of data variation with timestamps at this field position, finding that the recording time intervals are 3 seconds, 3 seconds, and 2 seconds (i.e., 3 seconds from 14:32:00 to 14:32:03, 3 seconds from 14:32:03 to 14:32:06, and 2 seconds from 14:32:06 to 14:32:08), and the time intervals are not fixed. This pattern is compared with a preset loop rule, which requires data to be recorded once per second. The actual recording time interval does not meet the once-per-second loop rule. Therefore, the system determines the number of loops in the variable information process, which is 4 times. This number of loops is recorded in the second data set to represent the looping characteristics of the first data set during the looping process. For example, the generated second data is as follows: A loop: from 2025-03-14 14:32:00 to 2025-03-14 14:32:08, 4times: File scan results: Number of files = {5,7,6,8}. Here, 4times represents the number of loops in the second data.

[0062] The method in this application, when the change pattern meets a cyclical rule, determines the time period of the data to be processed and records it in second data. When the change pattern does not meet the cyclical rule, the cyclical characteristic is represented by recording the number of cycles. This not only reduces storage space usage but also simplifies the subsequent data analysis process. The cyclical pattern can be determined through the second data without processing a large amount of repetitive information, thereby improving the efficiency of data analysis.

[0063] refer to Figure 3 The application scenario diagram of the data processing method provided in this application embodiment is applied to the data processing of cyclic log.

[0064] The code runs and begins outputting logs, recording the log content to a file cache. The log content is as follows: 2025-03-14 14:32:00 Loop through the records in the table: A1=1 2025-03-14 14:32:01 Loop through the records in the table: A1=3 2025-03-14 14:32:02 Loop through the records in the table: A1=6 2025-03-14 14:32:03 Loop through the records in the table: A1=10 To determine if a log entry is looping, specifically, check if the two adjacent log entries contain the same content (excluding timestamps). If, in addition to the identical content, there is also content that changes at a fixed position, then it's considered a loop. This loop continuously checks the content of the two adjacent log entries until the condition is no longer met, at which point the loop ends. The conditions include: **Fixed text structure:** The log entries contain the same text structure information (e.g., "Looping through the records in the table: A1="). **Changed data:** Data within the fixed text structure changes (e.g., the value of A1 is different each time).

[0065] If the log content satisfies the loop condition, the loop content in the file cache is organized and merged. Specifically, the loop log content in the cache is organized and merged to form a loop-structured data. The specific steps include: Extract constant information (e.g., "Loop through the records in the table:"). Extract variable information (e.g., the value of variable A1). Determine the start and end times of the loop (extracted from the timestamp in the log). Determine the number of loop iterations or the time period (e.g., a total of 4 iterations or once per second). The generated loop structure data is in the following format: A loop: from 2025-03-14 14:32:00 to 2025-03-14 14:32:03, each second (or 4 times): Loop through the records in the table: {A1}={1,3,6,10}.

[0066] The loop continues until the log content no longer meets the loop condition, at which point the loop ends, the cleaned loop content is written to a file, and the file cache is cleared. Log recording continues until the application terminates, at which point log recording ceases.

[0067] The following continues to describe exemplary structures of the data processing apparatus 90 provided in the embodiments of this application as software modules. In some embodiments, such as Figure 4 As shown, the data processing device 90 may include: an acquisition module 901, used to acquire data to be processed that is continuously output to a buffer; a processing module 902, used to analyze the data to be processed and, in response to the data to be processed meeting a set data compression condition, extract first data from the data to be processed; the first data includes at least constant information and variable information that can characterize cyclic features; a generation module 903, used to generate second data based on the first data; the second data includes at least cyclic structure data corresponding to the constant information and variable information; the cyclic structure data is used to represent the cyclic feature information of repeated elements in the first data; and a storage module 904, used to store the second data.

[0068] In some embodiments, the obtaining module 901 is configured to: extract the text structure information as constant information in the first data in response to the existence of the same text structure information in a consecutive first number of data to be processed; the same text structure information represents the content and / or the description content with the same position in the loop process.

[0069] In some embodiments, the obtaining module 901 is configured to: in response to the frequency of numerical changes at a set field position in a second consecutive number of data to be processed exceeding a preset threshold, extract the name of the changed data and the corresponding set of values ​​as variable information; the name of the changed data represents the variable identifier of the changed data during the loop, and the set of values ​​represents the value of the changed data during the loop.

[0070] In some embodiments, the generation module 903 is configured to: determine the data to be processed as cyclic data in response to the data to be processed meeting the set data compression conditions; obtain the timestamp corresponding to each cyclic data; determine the cycle start time and cycle end time corresponding to the cyclic data based on the timestamp corresponding to each cyclic data in response to the next data in the cyclic data not meeting the set data compression conditions; and generate second data based on the first data in the cyclic data, the cycle start time, and the cycle end time.

[0071] In some embodiments, the generation module 903 is configured to: in response to at least two first data including constant information, determine a constant data structure corresponding to the constant information; the constant data structure represents the fixed description content of multiple first data in the loop process; in response to at least two first data including variable information, determine a variable data structure based on the variable information and the loop rule corresponding to the variable information; the variable data structure represents the change content of multiple first data in the loop process; the change content includes at least the name of the changed data and the corresponding set of values; and sort the constant data structure and the variable data structure based on the time sequence of the data to be processed to obtain the second data.

[0072] In some embodiments, the generation module 903 is configured to: determine the field position of variable information in the data to be processed; determine the change pattern of the data at the field position with the timestamp of the data to be processed; in response to the change pattern satisfying the loop rule, determine the variable data structure based on the change pattern of the timestamp and the variable information; in response to the change pattern not satisfying the loop rule, determine the number of loops of the variable information in the loop process, and determine the variable data structure based on the variable information and the number of loops.

[0073] In some embodiments, the data processing apparatus 90 may further include a deletion module, which is configured to: delete unprocessed data associated with the second data in the cache.

[0074] In some embodiments, the generation module 903 is configured to: determine the time period of the data to be processed in response to the change pattern satisfying the cyclic rule, wherein the time period is a fixed interval between adjacent timestamps; and record the time period in the second data to represent the cyclic characteristics of the first data in the cyclic process.

[0075] In some embodiments, the generation module 903 is configured to: in response to a change pattern not satisfying a cyclic rule, record the number of cycles in the second data to represent the cyclic characteristics of the first data during the cyclic process.

[0076] It should be noted that the description of the apparatus in this application embodiment is similar to the description of the method embodiment above, and has similar beneficial effects as the method embodiment; therefore, it will not be repeated. For any technical details not covered in the data processing apparatus provided in this application embodiment, please refer to... Figures 1 to 4 The meaning is understood in accordance with the description of any of the accompanying drawings.

[0077] According to embodiments of this application, this application also provides an electronic device and a non-transitory computer-readable storage medium.

[0078] Figure 5 A schematic block diagram of an example electronic device 800 that can be used to implement embodiments of this application is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the application described and / or claimed herein.

[0079] like Figure 5 As shown, the electronic device 800 includes a computing unit 801, which can perform various appropriate actions and processes based on a computer program stored in a read-only memory (ROM) 802 or a computer program loaded from a storage unit 808 into a random access memory (RAM) 803. The RAM 803 may also store various programs and data required for the operation of the electronic device 800. The computing unit 801, ROM 802, and RAM 803 are interconnected via a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.

[0080] Multiple components in electronic device 800 are connected to I / O interface 805, including: input unit 806, such as keyboard, mouse, etc.; output unit 807, such as various types of displays, speakers, etc.; storage unit 808, such as disk, optical disk, etc.; and communication unit 809, such as network card, modem, wireless transceiver, etc. Communication unit 809 allows electronic device 800 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0081] The computing unit 801 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 801 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 801 performs the various methods and processes described above, such as data processing methods. For example, in some embodiments, the data processing method may be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 808. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 800 via ROM 802 and / or communication unit 809. When the computer program is loaded into RAM 803 and executed by the computing unit 801, one or more steps of the data processing method described above may be performed. Alternatively, in other embodiments, the computing unit 801 may be configured to perform data processing methods by any other suitable means (e.g., by means of firmware).

[0082] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0083] The program code used to implement the methods of this application may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing device, such that when executed by the processor or controller, the functions / operations specified in the flowcharts and / or block diagrams are implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0084] In the context of this application, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. Machine-readable media can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0085] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0086] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0087] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact via communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, servers in distributed systems, or servers incorporating blockchain technology.

[0088] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this application can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this application can be achieved, and this is not limited herein.

[0089] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this application, "a plurality of" means two or more, unless otherwise explicitly specified.

[0090] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A data processing method, the method comprising: Obtain the pending data that is continuously output to the cache; The data to be processed is analyzed, and in response to the data to be processed meeting the set data compression conditions, the first data in the data to be processed is extracted; the first data includes at least constant information and variable information that can characterize the cyclic features; The second data is generated based on the first data; the second data includes at least loop structure data corresponding to constant information and variable information. The cyclic structure data is used to represent the cyclic feature information of repeated elements in the first data; Store the second data.

2. The method according to claim 1, wherein extracting the first data from the data to be processed in response to the data to be processed satisfying the set data compression conditions comprises: In response to the existence of the same text structure information in a first consecutive number of data to be processed, the text structure information is extracted as constant information in the first data. The same text structure information represents the content and / or the description content that is in the same position during the loop process.

3. The method according to claim 1, wherein extracting the first data from the data to be processed in response to the data to be processed satisfying the set data compression conditions comprises: In response to the fact that the frequency of numerical changes in the position of a set field in a second consecutive number of unprocessed data exceeds a preset threshold, the name of the changed data and the corresponding set of numerical values ​​are extracted as the variable information. The name of the changing data represents the variable identifier of the changing data during the loop, and the set of values ​​represents the value of the changing data during the loop.

4. The method according to claim 1, further comprising: In response to the data to be processed meeting the set data compression conditions, the data to be processed is determined to be cyclic data; Obtain the timestamp corresponding to each of the aforementioned cyclic data items; In response to the fact that the next data in the loop does not meet the set data compression conditions, the loop start time and loop end time corresponding to the loop data are determined based on the timestamp corresponding to each data in the loop. The second data is generated based on the first data in the loop data, the loop start time, and the loop end time.

5. The method according to claim 1, wherein generating the second data based on the first data comprises: In response to the inclusion of constant information in at least two first data sets, a constant data structure corresponding to the constant information is determined; A constant data structure represents the fixed description of multiple first data during the loop process; In response to the inclusion of variable information in at least two first data points, a variable data structure is determined based on the variable information and the corresponding loop rule; the variable data structure represents the changes of multiple first data points during the loop process. The changes include at least the name of the changed data and the corresponding set of values; Based on the chronological order of the data to be processed, the constant data structure and the variable data structure are sorted to obtain the second data.

6. The method according to claim 5, wherein determining the variable data structure based on the variable information and the corresponding loop rule includes: Determine the field position of the variable information in the data to be processed; Determine the pattern of how the data at the position of the field changes with the timestamp of the data to be processed; In response to the change pattern satisfying the cyclic rule, the variable data structure is determined based on the change pattern of the timestamp and the variable information; In response to the change pattern not satisfying the loop rule, the number of loops of the variable information in the loop process is determined, and the variable data structure is determined based on the variable information and the number of loops.

7. The method according to claim 1, wherein after storing the second data, the method further comprises: Delete the pending data in the cache that is associated with the second data.

8. The method according to claim 6, further comprising: In response to the change pattern satisfying the cyclic rule, the time period of the data to be processed is determined, and the time period is a fixed interval between adjacent timestamps; The time period is recorded in the second data to represent the cyclical characteristics of the first data during the cyclic process.

9. The method according to claim 6, further comprising: In response to the change pattern not satisfying the cyclic rule, the number of cycles is recorded in the second data to represent the cyclic characteristics of the first data during the cyclic process.

10. A data processing apparatus, the apparatus comprising: The acquisition module is used to acquire the data to be processed that is continuously output to the cache; The processing module is used to analyze the data to be processed and extract the first data from the data to be processed in response to the data to be processed meeting the set data compression conditions. The first data includes at least constant information and variable information that can characterize the cyclic features; The generation module is used to generate second data based on the first data; the second data includes at least loop structure data corresponding to constant information and variable information; The cyclic structure data is used to represent the cyclic feature information of repeated elements in the first data; A storage module for storing the second data.

Images