Data processing method and device, storage medium and electronic equipment

Abstract

The application discloses a data processing method and device, a storage medium and an electronic device. The application obtains at least one scanning task submitted by a user for a target file, wherein the scanning task includes a scanning task regular expression and a to-be-scanned file attribute type, performs pre-aggregation processing on the scanning task according to the to-be-scanned file attribute type, obtains at least one pre-aggregation task, performs aggregation processing on the pre-aggregation task based on the corresponding scanning task regular expression in the pre-aggregation task, obtains an aggregation regular expression corresponding to each pre-aggregation task, generates a target aggregation task based on the aggregation regular expression, and performs data scanning on the target file based on the target aggregation task to obtain target data matched with each scanning task. Therefore, the application can combine the regular expressions in the scanning task into an aggregation regular expression for matching, so that multiple regular expressions can be matched by performing scanning only once, and the scanning efficiency can be significantly improved.

Description

Technical Field

[0001] This application relates to the field of data processing technology, specifically to a data processing method, apparatus, storage medium, and electronic device. Background Technology

[0002] In the field of cybersecurity, logs play an extremely important role, serving as a crucial tool for detecting abnormal behavior and potential attacks. In the event of a cyberattack, security experts can match the characteristics of the attack behavior within massive amounts of user logs to assess the scope of the attack and promptly notify users to upgrade their security measures. One common manifestation of cyberattack behavior is that a user is tricked into triggering remote attack commands or downloading and opening a file with a specific name. By using command names, command parameters, or filenames to fuzzily match user process logs or behavior logs, the attacked user can be located and further action taken.

[0003] However, in related technologies, only one keyword can be matched at a time when matching logs, resulting in low matching efficiency. Summary of the Invention

[0004] This application provides a data processing method, apparatus, storage medium, and electronic device that can improve the efficiency of matching data in logs.

[0005] In a first aspect, embodiments of this application provide a data processing method, including:

[0006] Obtain at least one scan task for a target file submitted by a user, wherein the scan task includes a scan task regular expression and the attribute type of the file to be scanned;

[0007] The scanning task is pre-aggregated based on the attribute type of the file to be scanned to obtain at least one pre-aggregated task.

[0008] Based on the regular expression of the scanning task corresponding to the pre-aggregation task, the pre-aggregation task is aggregated to obtain the aggregation regular expression corresponding to each pre-aggregation task, and the target aggregation task is generated based on the aggregation regular expression.

[0009] Based on the target aggregation task, the target file is scanned to obtain target data that matches each of the scanning tasks.

[0010] In some implementations, the step of pre-aggregating the scanning task based on the attribute type of the file to be scanned to obtain at least one pre-aggregated task includes:

[0011] Based on the attribute type of the file to be scanned, the scanning task is divided to obtain at least one pre-aggregation task.

[0012] In some implementations, dividing the scanning task based on the attribute type of the file to be scanned to obtain at least one pre-aggregation task includes:

[0013] Each scan task belonging to the same attribute type of the file to be scanned is considered as a pre-aggregation task, resulting in at least one pre-aggregation task.

[0014] In some implementations, the step of aggregating the pre-aggregation tasks based on the regular expressions of the corresponding scanning tasks in the pre-aggregation tasks to obtain an aggregation regular expression for each pre-aggregation task, and generating a target aggregation task based on the aggregation regular expressions, includes:

[0015] Based on a preset regular expression matching library, the regular expressions of the scanning tasks corresponding to the pre-aggregation tasks are aggregated to obtain the aggregated regular expression corresponding to each pre-aggregation task.

[0016] The target aggregation task is determined based on the aggregation regular expression.

[0017] In some implementations, the step of aggregating the regular expressions of the corresponding scanning tasks in the pre-aggregation tasks to obtain an aggregated regular expression for each pre-aggregation task, and determining the target aggregation task based on the aggregated regular expression, includes:

[0018] The pre-aggregation task is assigned to multiple target computing nodes, so that the target computing nodes perform aggregation processing based on the regular expression of the corresponding scan task in the pre-aggregation task to obtain an aggregation regular expression for the pre-aggregation task, and determine the target aggregation task based on the aggregation regular expression.

[0019] The step of scanning the target file based on the target aggregation task to obtain target data matching each scanning task includes:

[0020] The multiple target computing nodes obtain target data matching each scanning task by scanning the target file based on the target aggregation task.

[0021] In some implementations, obtaining at least one scan task for a target file submitted by a user includes:

[0022] Obtain the user's input regarding the scanning task and generate prompt words for that task;

[0023] The prompt words generated for the scanning task are input into the large model for generating scanning tasks, and the scanning tasks that generate prompt words for the scanning tasks are matched.

[0024] In some implementations, the scanning task generates a large model that is trained according to the following steps:

[0025] Obtain a basic large language model, and create an initial large model for generating prompt words for scanning tasks based on the basic large language model;

[0026] Obtain sample scanning task prompts and label the sample scanning task prompts with scanning task tags.

[0027] The prompt words generated by the sample scanning task are used to train the large model generated by the initial scanning task for at least one round.

[0028] During the forward propagation training of the model, a large model is generated based on the initial scanning task to generate prompt words to determine the predicted scanning task.

[0029] During the backpropagation training of the model, the scanning task generation loss is determined based on the predicted scanning task and the scanning task label. The scanning task generation loss is used to adjust the model parameters of the initial scanning task generation model until the initial scanning task generation model ends the model training, thus obtaining the scanning task generation model.

[0030] Secondly, embodiments of this application also provide a data processing apparatus, including:

[0031] The acquisition module is used to acquire at least one scanning task for a target file submitted by the user, wherein the scanning task includes a scanning task regular expression and the attribute type of the file to be scanned.

[0032] A pre-aggregation processing module is used to perform pre-aggregation processing on the scanning task according to the attribute type of the file to be scanned, so as to obtain at least one pre-aggregation task.

[0033] The aggregation processing module is used to aggregate the pre-aggregation tasks based on the regular expression of the scanning task corresponding to the pre-aggregation task, to obtain the aggregation regular expression corresponding to each pre-aggregation task, and to generate the target aggregation task based on the aggregation regular expression.

[0034] The matching module is used to perform data scanning on the target file based on the target aggregation task to obtain target data that matches each of the scanning tasks.

[0035] In some embodiments, the pre-aggregation processing module is configured to:

[0036] Based on the attribute type of the file to be scanned, the scanning task is divided to obtain at least one pre-aggregation task.

[0037] In some embodiments, the pre-aggregation processing module is configured to:

[0038] Each scan task belonging to the same attribute type of the file to be scanned is considered as a pre-aggregation task, resulting in at least one pre-aggregation task.

[0039] In some embodiments, the aggregation processing module is configured to:

[0040] Based on a preset regular expression matching library, the regular expressions of the scanning tasks corresponding to the pre-aggregation tasks are aggregated to obtain the aggregated regular expression corresponding to each pre-aggregation task.

[0041] The target aggregation task is determined based on the aggregation regular expression.

[0042] In some embodiments, the aggregation processing module is configured to:

[0043] The pre-aggregation task is assigned to multiple target computing nodes, so that the target computing nodes perform aggregation processing based on the regular expression of the corresponding scan task in the pre-aggregation task to obtain an aggregation regular expression for the pre-aggregation task, and determine the target aggregation task based on the aggregation regular expression.

[0044] In some embodiments, the matching module is configured to:

[0045] The multiple target computing nodes obtain target data matching each scanning task by scanning the target file based on the target aggregation task.

[0046] In some embodiments, the acquisition module is configured to:

[0047] Obtain the user's input regarding the scanning task and generate prompt words for that task;

[0048] The prompt words generated for the scanning task are input into the large model for generating scanning tasks, and the scanning tasks that generate prompt words for the scanning tasks are matched.

[0049] In some embodiments, the scanning task generates a large model that is trained according to the following steps:

[0050] Obtain a basic large language model, and create an initial large model for generating prompt words for scanning tasks based on the basic large language model;

[0051] Obtain sample scanning task prompts and label the sample scanning task prompts with scanning task tags.

[0052] The prompt words generated by the sample scanning task are used to train the large model generated by the initial scanning task for at least one round.

[0053] During the forward propagation training of the model, a large model is generated based on the initial scanning task to generate prompt words to determine the predicted scanning task.

[0054] During the backpropagation training of the model, the scanning task generation loss is determined based on the predicted scanning task and the scanning task label. The scanning task generation loss is used to adjust the model parameters of the initial scanning task generation model until the initial scanning task generation model ends the model training, thus obtaining the scanning task generation model.

[0055] Thirdly, embodiments of this application also provide a computer-readable storage medium having a computer program stored thereon, which, when run on a computer, causes the computer to perform a data processing method as provided in any embodiment of this application.

[0056] Fourthly, embodiments of this application also provide an electronic device, including a processor and a memory, wherein the memory has a computer program, and the processor executes a data processing method as provided in any embodiment of this application by calling the computer program.

[0057] The technical solution provided in this application involves obtaining at least one scanning task for a target file submitted by a user. Each scanning task includes a scanning task regular expression and the attribute type of the file to be scanned. The scanning tasks are pre-aggregated based on the attribute type of the file to be scanned, resulting in at least one pre-aggregated task. Based on the corresponding scanning task regular expressions in the pre-aggregated tasks, the pre-aggregated tasks are aggregated to obtain an aggregated regular expression corresponding to each pre-aggregated task. A target aggregated task is generated based on the aggregated regular expression. Data scanning of the target file is then performed based on the target aggregated task to obtain target data matching each scanning task. Therefore, this application can combine regular expressions in scanning tasks into a single aggregated regular expression for matching, allowing multiple regular expressions or keywords to be matched with only one scan, significantly improving scanning efficiency. Attached Figure Description

[0058] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0059] Figure 1 This is a schematic diagram of a first type of data processing method provided in an embodiment of this application.

[0060] Figure 2This is a second flowchart illustrating the data processing method provided in an embodiment of this application.

[0061] Figure 3 A flowchart illustrating the application of the data processing method provided in this application to a specific scenario.

[0062] Figure 4 This is a schematic diagram of the structure of the data processing apparatus provided in the embodiments of this application.

[0063] Figure 5 This is a schematic diagram of a first structure of an electronic device provided in an embodiment of this application.

[0064] Figure 6 This is a schematic diagram of a second structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0065] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the protection scope of this application.

[0066] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0067] In the field of cybersecurity, logs play an extremely important role, serving as a crucial tool for detecting abnormal behavior and potential attacks. In the event of a cyberattack, security experts can match the characteristics of the attack behavior within massive amounts of user logs to assess the scope of the attack and promptly notify users to upgrade their security measures. One common manifestation of cyberattack behavior is that a user is tricked into triggering remote attack commands or downloading and opening a file with a specific name. By using command names, command parameters, or filenames to fuzzily match user process logs or behavior logs, the attacked user can be located and further action taken.

[0068] However, in related technologies, only one keyword can be matched at a time when matching logs, resulting in low matching efficiency.

[0069] To address the aforementioned issues, embodiments of this specification provide a data processing method. The entity executing this data processing method may be the data processing device provided in the embodiments of this application, or an electronic device integrating the data processing device. The data processing device may be implemented in hardware or software.

[0070] Among them, electronic devices can be any device equipped with a processor and having processing capabilities, such as mobile electronic devices with processors such as smartphones, tablets, handheld computers, and laptops, or fixed electronic devices with processors such as desktop computers, televisions, and servers.

[0071] It should be noted that this application is not only applicable to scanning and processing the log files mentioned above, such as system logs (e.g., Windows event logs, Linux syslog), application logs, database logs, firewall logs, etc., but also applicable to scanning and processing other files, such as configuration files, source code and script files, binary files, database files, backup files, email files, document and report files, temporary and cache files, system files, etc.

[0072] Please see Figure 1 , Figure 1 This is a schematic diagram of a first flowchart of a data processing method provided in an embodiment of this application. The specific flow of the data processing method provided in an embodiment of this application can be as follows:

[0073] S110. Obtain at least one scan task submitted by the user for the target file, wherein the scan task includes a scan task regular expression and the attribute type of the file to be scanned.

[0074] In this context, target files refer to specific files or sets of files that the user wants the system to scan. These files can be stored on the local computer, a remote server, or cloud storage. The types of target files can include those exemplified above.

[0075] In this embodiment, the user can initiate one or more scan requests on the scanning system or platform. These requests can be submitted through a graphical user interface (GUI), a command-line interface (CLI), or an application programming interface (API), without specific limitations. The scan request submitted by the user includes the scan task, and the scan request should explicitly specify which files are target files. This can be achieved through filenames, file paths, file types, or other identifiers.

[0076] Regular expressions (Regex) are powerful text processing tools used to match specific patterns in strings. In scanning tasks, regular expressions can be used to specify the text patterns to search for. For example, if a user wants to scan all filenames containing a specific date format, they can use a regular expression to define that date format. Regular expressions can also be used to search for specific strings or patterns within file content. The regular expression for the scanning task in this embodiment is the same as the regular expression provided by the user for the scanning task.

[0077] Among them, the file attribute type to be scanned refers to the specific characteristics or metadata of the file. Common file attribute types include file name, file extension, file size, file creation date, file modification date, file permissions (such as read, write, and execute permissions), etc.

[0078] For example, users can specify the file attribute types to be scanned in the scan task. For example, users can specify the file extensions to be scanned to ensure that only specific types of files (such as .txt, .doc, etc.) are scanned.

[0079] S120. Perform pre-aggregation processing on the scanning task according to the attribute type of the file to be scanned, and obtain at least one pre-aggregation task.

[0080] In this application, when performing pre-aggregation processing on the scanning task according to the attribute type of the file to be scanned to obtain at least one pre-aggregation task, the scanning task can be divided based on the attribute type of the file to be scanned to obtain at least one pre-aggregation task.

[0081] For example, if the file attributes to be scanned include file extensions such as .txt and .doc, then scan tasks with the same extension will be treated as a pre-aggregation task. That is, scan tasks with the .txt extension will be treated as a pre-aggregation task, and scan tasks with the .doc extension will be treated as a pre-aggregation task.

[0082] For example, if the attribute type of the file to be scanned is the file's creation date, including date range 1, date range 2, and date range 3, then the scan tasks within the same date range will be treated as a pre-aggregation task. That is, the scan task corresponding to date range 1 will be treated as a pre-aggregation task, the scan task corresponding to date range 2 will be treated as a pre-aggregation task, and the scan task corresponding to date range 3 will be treated as a pre-aggregation task.

[0083] It is understandable that the attribute type of the files to be scanned is consistent for each pre-aggregation task.

[0084] S130. Based on the regular expression of the scanning task corresponding to the pre-aggregation task, the pre-aggregation task is aggregated to obtain the aggregation regular expression corresponding to each pre-aggregation task, and the target aggregation task is generated based on the aggregation regular expression.

[0085] In this embodiment, it can be understood that a pre-aggregation task may include one or more scanning tasks. The regular expressions of the scanning tasks included in each pre-aggregation task are aggregated to obtain an aggregated regular expression for each pre-aggregation task. The aggregated regular expression for each pre-aggregation task and the attribute type of the file to be scanned constitute the target aggregation task.

[0086] S140. Based on the target aggregation task, perform data scanning on the target file to obtain target data that matches each scanning task.

[0087] In this embodiment, after the target aggregation task is formed based on the aggregation regular expression corresponding to each pre-aggregation task and the attribute type of the file to be scanned, the target aggregation task can be executed to scan the target file and obtain the target data matching each scanning task.

[0088] In practice, this application is not limited by the execution order of the described steps. Without causing conflicts, some steps may be performed in other orders or simultaneously.

[0089] As can be seen from the above, the data processing method provided in this application obtains at least one scanning task for a target file submitted by a user. Each scanning task includes a scanning task regular expression and the attribute type of the file to be scanned. The scanning task is pre-aggregated based on the attribute type of the file to be scanned to obtain at least one pre-aggregated task. Based on the corresponding scanning task regular expression in the pre-aggregated task, the pre-aggregated task is aggregated to obtain a target aggregated task. The target file is then scanned based on the target aggregated task to obtain target data matching each scanning task. Therefore, this application can combine the regular expressions in the scanning tasks into a single aggregated regular expression for matching, enabling the matching of multiple regular expressions or keywords with only one scan, significantly improving scanning efficiency.

[0090] Based on the methods described in the preceding embodiments, the following examples will provide further detailed explanations.

[0091] Please see Figure 2 , Figure 2 This is a second flowchart illustrating the data processing method provided in an embodiment of this application. The method includes:

[0092] S210. Obtain at least one scan task submitted by the user for the target file, wherein the scan task includes a scan task regular expression and the attribute type of the file to be scanned.

[0093] In one implementation, the format of the scan task can be defined to facilitate subsequent processing of the scan task. For example, the format of the scan task can be defined as <scan task regular expression, file attribute type to be scanned>.

[0094] In one example, assuming the file to be scanned has the attribute type "file creation time", the format of the scan task submitted by the user can be <scan task regular expression, file creation time>, for example, scan task:<regxp1,dayrange1> ,<regxp2,dayrange2> ,<regxp3,dayrange2> ,<regxp4,dayrange1> ,<regxp5,dayrange3> Each of these five scan tasks includes a regular expression for the scan task and a time range for the scanned data (i.e., the time when the file was created).

[0095] In this embodiment, the user can directly provide the scanning task, that is, provide the regular expression for the scanning task and the attribute type of the file to be scanned, so that the scanning task can be processed later.

[0096] In addition, users can provide scan task generation prompts and generate corresponding scan tasks using these prompts. For example, in one implementation, step S210, "obtaining at least one scan task for the target file submitted by the user," may include the following steps S2101 to S2102:

[0097] S2101. Obtain the user-inputted prompt word for the scanning task.

[0098] The prompt message generated by the scanning task is a description of the scanning task. It may include the purpose of the scanning task, the object to be scanned, the scanning requirements, the number of scans, and so on.

[0099] In some implementations, when obtaining user input and generating prompts for a scanning task, the user can be provided with guidance statements about the scanning task to help them clarify the prompts that need to be generated. For example, the guidance statements could be: What is the purpose of your scan?, What type of file attributes do you want to scan?, etc.

[0100] S2102. Input the prompt words generated for the scanning task into the large model for generating scanning tasks, and match the scanning tasks that generate prompt words for the scanning tasks.

[0101] Specifically, the large model for generating scanning tasks is configured to match the prompt words generated by the scanning tasks with the prompt words generated for those scanning tasks.

[0102] In one implementation, the large model generated by the scanning task is trained according to the following steps S2103-S2107:

[0103] S2103. Obtain the basic large language model, and create an initial large model for generating prompt words for scanning tasks based on the basic large language model.

[0104] Among them, basic large language models include, but are not limited to, large models in the field of natural language processing (NLP), such as GPT-3, GPT-4, ChatGPT, BERT, and RoBERTa.

[0105] S2104. Obtain the prompt words generated for the sample scanning task and label the prompt words generated for the sample scanning task with scanning task tags;

[0106] S2105. Use the prompt words generated by the sample scanning task to train the large model generated by the initial scanning task at least once.

[0107] S2106. During the model forward propagation training process, a large model is generated through the initial scanning task, and prompt words are generated based on the sample scanning task to determine the prediction scanning task.

[0108] S2107. During the backpropagation training of the model, the scanning task generation loss is determined based on the predicted scanning task and the scanning task label. The scanning task generation loss is used to adjust the model parameters of the initial scanning task generation large model until the initial scanning task generation large model ends the model training, and the scanning task generation large model is obtained.

[0109] Specifically, the scanning task generation loss is calculated using a preset model loss formula based on the predicted scanning task and the scanning task label. This preset model loss formula can be one or more of the following fitting techniques: contrastive loss, cross-entropy loss, hinge loss, etc.

[0110] Optionally, the training termination conditions for the large model generated by the initial scanning task may include, for example, the value of the loss function being less than or equal to a preset loss function threshold, or the number of iterations reaching a preset threshold. Specific training termination conditions can be determined based on actual circumstances and are not specifically limited here.

[0111] The above method generates corresponding scanning tasks based on user-input scanning task prompts, allowing users who are not skilled in editing regular expressions to use the method provided in this application for data matching, thus facilitating user use and improving the universality of this application.

[0112] S220. Take scanning tasks that belong to the same attribute type of the file to be scanned as a pre-aggregation task to obtain at least one pre-aggregation task.

[0113] For example, if the file attributes to be scanned include file extensions such as .txt and .doc, then scan tasks with the same extension will be treated as a pre-aggregation task. That is, scan tasks with the .txt extension will be treated as a pre-aggregation task, and scan tasks with the .doc extension will be treated as a pre-aggregation task.

[0114] For example, if the attribute type of the file to be scanned is the file's creation date, including date range 1, date range 2, and date range 3, then the scan tasks within the same date range will be treated as a pre-aggregation task. That is, the scan task corresponding to date range 1 will be treated as a pre-aggregation task, the scan task corresponding to date range 2 will be treated as a pre-aggregation task, and the scan task corresponding to date range 3 will be treated as a pre-aggregation task.

[0115] For example, the five scanning tasks exemplified in step S210:<regxp1,dayrange1> ,<regxp2,dayrange2> ,<regxp3,dayrange2> ,<regxp4,dayrange1> ,<regxp5,dayrange3> For these five scanning tasks, those belonging to the same scanning data time range are treated as a single pre-aggregation task, resulting in the following three pre-aggregation tasks:<regxp1®xp4,dayrange1> ,<regxp2®xp3,dayrange2> ,<regxp5,dayrange3> .

[0116] It is understandable that the attribute type of the files to be scanned is consistent for each pre-aggregation task.

[0117] S230. Based on the preset regular expression matching library, the regular expressions of the corresponding scanning tasks in the pre-aggregation tasks are aggregated to obtain the aggregated regular expression corresponding to each pre-aggregation task.

[0118] The default regular expression matching library can be Hyperscan, which is a high-performance regular expression engine launched by Intel.

[0119] In this embodiment, a preset regular expression matching library can be used to aggregate the regular expressions of the corresponding scanning tasks in the pre-aggregation tasks to obtain the aggregated regular expression corresponding to each pre-aggregation task.

[0120] For example, the three pre-aggregation tasks exemplified in S220<regxp1®xp4,dayrange1> ,<regxp2®xp3,dayrange2> ,<regxp5,dayrange3> The regular expressions corresponding to each pre-aggregation task are aggregated, that is, the regular expressions within a scan data time range are merged into a DFA (Deterministic Finite Automaton) using Hyperscan.

[0121] S240. Determine the target aggregation task based on the aggregation regular expression.

[0122] It is understandable that the aggregation regular expression and the attribute type of the file to be scanned constitute the target aggregation task for each pre-aggregation task.

[0123] In some implementations, the regular expressions of the corresponding scan tasks in the pre-aggregation tasks are aggregated to obtain an aggregated regular expression for each pre-aggregation task. The target aggregation task is then determined based on the aggregated regular expression, including:

[0124] The pre-aggregation task is assigned to multiple target computing nodes, so that the target computing nodes can perform aggregation processing based on the regular expression of the corresponding scan task in the pre-aggregation task to obtain the aggregation regular expression for the pre-aggregation task, and determine the target aggregation task based on the aggregation regular expression.

[0125] In the context of computer science and technology, a computing node refers to a device or resource unit used to perform computing tasks, particularly in scenarios such as distributed computing, cloud computing, or big data processing. These devices or resource units can be physical computers, virtual machines, cloud service instances, or other entities with computing capabilities. Computing nodes typically possess independent processing capabilities, enabling them to independently receive tasks, perform computations, and return results to the requester or the master node.

[0126] In this embodiment, since there can be one or more pre-aggregation tasks in this application, when there are multiple pre-aggregation tasks, these multiple pre-aggregation tasks can be assigned to multiple target computing nodes for processing, so that multiple target computing nodes can process these multiple pre-aggregation tasks in parallel, which can significantly reduce the processing time of scanning tasks and improve processing efficiency.

[0127] In one example, Spark's distributed task scaling capabilities can be used to process multiple pre-aggregated tasks in this application in a distributed manner.

[0128] S250. Based on the target aggregation task, perform data scanning on the target file to obtain target data that matches each scanning task.

[0129] In this embodiment, after obtaining the target aggregation task, the target file can be scanned based on the target aggregation task to obtain the target data that matches each scanning task.

[0130] For example, referring to the example in step S230, regular expressions within a scan data time range are merged into a DFA (Deterministic Finite Automaton) using Hyperscan, which is to obtain the target aggregation task. Then, the DFA corresponding to the scan task is used to perform data matching, thereby obtaining the target data that matches each scan task. This allows the application to achieve the goal of matching multiple regular expressions in a directory with only one data scan.

[0131] In some implementations, data scanning of the target file based on the target aggregation task yields target data matching each scanning task, including:

[0132] Multiple target computing nodes are obtained by performing data scanning on target files based on target aggregation tasks to obtain target data that matches each scanning task.

[0133] In this embodiment, since the aggregation regular expression and the attribute type of the file to be scanned constitute the target aggregation task for each pre-aggregation task, the target aggregation task can be split and distributed to multiple target computing nodes for processing.

[0134] It is understandable that if a pre-aggregation task has been assigned to multiple target computing nodes for distributed processing, then those multiple target computing nodes can continue to perform distributed processing on the target aggregation task according to the assignment.

[0135] It is understood that, in this application, when processing the pre-aggregation task, it can be processed locally or distributed to multiple target computing nodes for distributed processing. Similarly, in this application, when processing the target aggregation task, it can be processed locally or distributed to multiple target computing nodes for distributed processing.

[0136] As can be seen from the above, the data processing method proposed in this application obtains at least one scanning task for a target file submitted by a user. Each scanning task includes a scanning task regular expression and the attribute type of the file to be scanned. Scanning tasks belonging to the same attribute type of the file to be scanned are treated as a pre-aggregation task, resulting in at least one pre-aggregation task. Based on a preset regular expression matching library, the corresponding scanning task regular expressions in the pre-aggregation tasks are aggregated to obtain an aggregated regular expression corresponding to each pre-aggregation task. A target aggregation task is determined based on the aggregated regular expression, and the target file is scanned based on the target aggregation task to obtain target data matching each scanning task. Therefore, this application can combine the regular expressions in the scanning tasks into an aggregated regular expression for matching, enabling multiple regular expressions or keywords to be matched with only one scan, significantly improving scanning efficiency.

[0137] To more clearly illustrate the data processing method provided in this application, the following is a flowchart illustrating the application of this data processing method in a specific scenario. Please refer to it. Figure 3 , Figure 3 A flowchart illustrating the application of the data processing method provided in this application to a specific scenario.

[0138] In this example, starting from "Start", the user first submits a scan task for the target file:<regxp1,dayrange1> ,<regxp2,dayrange2> ,<regxp3,dayrange2> ,<regxp4,dayrange1> ,<regxp5,dayrange3> Here, the scan task format is defined as <scan task regular expression, scan data time range>. Then, the task parameters are validated for validity. This primarily checks the correctness of the user-input regular expression. If the user-submitted scan task parameters are invalid, the process returns to the step where the user submitted the scan task, allowing the user to resubmit a scan task with correct parameters. If the user-submitted scan task parameters are valid, a corresponding task ID is generated for each scan task.<taskid1,regxp1> ,<taskid2,regxp2> ,<taskid3,regxp3> ,<taskid4,regxp4> ,<taskid5,regxp5> The task ID must be unique to prevent task disorder.

[0139] Next, the user-input scan task and its corresponding task ID are stored. The tasks are then pre-aggregated based on the query time range; that is, the pre-aggregation program pre-aggregates the tasks according to the time range of the scan data, resulting in:<regxp1®xp4,dayrange1> ,<regxp2®xp3,dayrange2> ,<regxp5,dayrange3> Three pre-aggregation tasks are performed and stored. The time range refers to the time range corresponding to the file creation time. For example, file paths can be stored in directories on HDFS according to file creation time. In this example, the scan task consists of two parts: regular expression path and time range. By aggregating tasks with the same time range together for execution, the number of accesses to file paths on the same date can be reduced.

[0140] Next, Spark is launched, reading one pre-aggregation task at a time. Hyperscan is used to recompile multiple regular expressions into a pattern database. In other words, Hyperscan merges multiple regular expressions to obtain the target aggregation task, facilitating concurrent matching during the subsequent scanning phase. The pattern database stores the multiple regular expressions to be merged (the pattern is the regular expression). Each time a regular expression is added to the pattern database, Hyperscan parses it into an NFA (Non-Deterministic Finite Automaton), ultimately converting it into a DFA. This DFA is then used to perform data matching, achieving the goal of matching multiple regular expressions in a single scan. Concurrent matching means that a single scan can match multiple regular expressions.

[0141] Next, execute the target aggregation task to perform the scan, and output the matched hit data, i.e. the target data, according to the task ID. For example, the hit data can be output to HDFS. Then stop Spark and end the data scanning process.

[0142] In the embodiments described above, Hyperscan and Spark are combined. Hyperscan is responsible for optimizing the problem of repeated scanning of data with multiple regular expressions, while Spark is responsible for extending single-machine tasks into distributed tasks. Spark mainly provides distributed scanning capabilities, which divides file paths into time slices, and then different Spark compute nodes are responsible for scanning tasks of different time slices, thereby improving scanning efficiency.

[0143] The data processing method provided in this application is based on Spark and Hyperscan to implement an efficient regular expression scanning engine, which breaks through the limitation of the data scale that a single regular expression engine can handle. After testing, the program in the data processing method provided in this application can run stably on data of hundreds of TB scale. Furthermore, by combining pre-aggregation logic and Hyperscan's multi-regular expression compilation feature, multiple tasks are merged and executed. When a user submits n query tasks at the same time, the queuing latency is reduced from (n-1)*T / 2 to (x-1)*T / 2 (where T is the execution cycle of a single scan task, and x = unique_time).

[0144] In one embodiment, a data processing apparatus is also provided. See also... Figure 4 , Figure 4 This is a schematic diagram of the structure of a data processing device 300 provided in an embodiment of this application. The data processing device 300 is applied to an electronic device and includes an acquisition module 301, a pre-aggregation processing module 302, an aggregation processing module 303, and a matching module 304, as follows:

[0145] The acquisition module 301 is used to acquire at least one scanning task for a target file submitted by the user, wherein the scanning task includes a scanning task regular expression and the attribute type of the file to be scanned.

[0146] The pre-aggregation processing module 302 is used to perform pre-aggregation processing on the scanning task according to the attribute type of the file to be scanned, so as to obtain at least one pre-aggregation task.

[0147] The aggregation processing module 303 is used to aggregate the pre-aggregation tasks based on the regular expression of the scanning task corresponding to the pre-aggregation task, to obtain the aggregation regular expression corresponding to each pre-aggregation task, and to generate the target aggregation task based on the aggregation regular expression.

[0148] The matching module 304 is used to perform data scanning on the target file based on the target aggregation task to obtain target data that matches each of the scanning tasks.

[0149] In some embodiments, the pre-aggregation processing module 302 is configured to:

[0150] Based on the attribute type of the file to be scanned, the scanning task is divided to obtain at least one pre-aggregation task.

[0151] In some embodiments, the pre-aggregation processing module 302 is configured to:

[0152] Each scan task belonging to the same attribute type of the file to be scanned is considered as a pre-aggregation task, resulting in at least one pre-aggregation task.

[0153] In some embodiments, the aggregation processing module 303 is configured to:

[0154] Based on a preset regular expression matching library, the regular expressions of the scanning tasks corresponding to the pre-aggregation tasks are aggregated to obtain the aggregated regular expression corresponding to each pre-aggregation task.

[0155] The target aggregation task is determined based on the aggregation regular expression.

[0156] In some embodiments, the aggregation processing module 303 is configured to:

[0157] The pre-aggregation task is assigned to multiple target computing nodes, so that the target computing nodes perform aggregation processing based on the regular expression of the corresponding scan task in the pre-aggregation task to obtain an aggregation regular expression for the pre-aggregation task, and determine the target aggregation task based on the aggregation regular expression.

[0158] In some embodiments, the matching module 304 is configured to:

[0159] The multiple target computing nodes obtain target data matching each scanning task by scanning the target file based on the target aggregation task.

[0160] In some embodiments, the acquisition module 301 is used for:

[0161] Obtain the user's input regarding the scanning task and generate prompt words for that task;

[0162] The prompt words generated for the scanning task are input into the large model for generating scanning tasks, and the scanning tasks that generate prompt words for the scanning tasks are matched.

[0163] In some embodiments, the scanning task generates a large model that is trained according to the following steps:

[0164] Obtain a basic large language model, and create an initial large model for generating prompt words for scanning tasks based on the basic large language model;

[0165] Obtain sample scanning task prompts and label the sample scanning task prompts with scanning task tags.

[0166] The prompt words generated by the sample scanning task are used to train the large model generated by the initial scanning task for at least one round.

[0167] During the forward propagation training of the model, a large model is generated based on the initial scanning task to generate prompt words to determine the predicted scanning task.

[0168] During the backpropagation training of the model, the scanning task generation loss is determined based on the predicted scanning task and the scanning task label. The scanning task generation loss is used to adjust the model parameters of the initial scanning task generation model until the initial scanning task generation model ends the model training, thus obtaining the scanning task generation model.

[0169] It should be noted that the data processing device provided in this application embodiment and the data processing method in the above embodiment belong to the same concept. The data processing device can implement any of the methods provided in the data processing method embodiment. For details of its implementation process, please refer to the data processing method embodiment, which will not be repeated here.

[0170] Furthermore, to better implement the data processing method in the embodiments of this application, based on the data processing method, this application also provides an electronic device, which may be a smartphone, server, or other such device. Please refer to [link / reference]. Figure 5 , Figure 5 This is a schematic diagram of a first structure of an electronic device provided in an embodiment of this application. The electronic device 400 includes a processor 401 and a memory 402. The processor 401 and the memory 402 are electrically connected.

[0171] The processor 401 is the control center of the electronic device 400. It connects various parts of the electronic device via various interfaces and lines. By running or calling computer programs stored in the memory 402, and by calling data stored in the memory 402, it performs various functions and processes data, thereby providing overall monitoring of the electronic device. The processor 401 can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor or any conventional processor.

[0172] The memory 402 can be used to store computer programs and data. The computer programs stored in the memory 402 contain instructions that can be executed in the processor. The computer programs can be composed of various functional modules. The processor 401 executes various functional applications and data processing by calling the computer programs stored in the memory 402. The memory 402 may mainly include a program storage area and a data storage area. The program storage area may store the operating system, application programs required for at least one function (such as sound playback function, image playback function, etc.), etc.; the data storage area may store data created according to the use of the electronic device 400 (such as audio data, video data, etc.). In addition, the memory may include high-speed random access memory, and may also include non-volatile memory, such as hard disk, RAM, plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, at least one disk storage device, flash memory device, or other volatile solid-state storage device.

[0173] In this embodiment, the processor 401 in the electronic device 400 loads the instructions corresponding to the processes of one or more computer programs into the memory 402 according to the following steps, and the processor 401 runs the computer programs stored in the memory 402 to realize various functions:

[0174] Obtain at least one scan task for a target file submitted by a user, wherein the scan task includes a scan task regular expression and the attribute type of the file to be scanned;

[0175] The scanning task is pre-aggregated based on the attribute type of the file to be scanned to obtain at least one pre-aggregated task.

[0176] Based on the regular expression of the scanning task corresponding to the pre-aggregation task, the pre-aggregation task is aggregated to obtain the aggregation regular expression corresponding to each pre-aggregation task, and the target aggregation task is generated based on the aggregation regular expression.

[0177] Based on the target aggregation task, the target file is scanned to obtain target data that matches each of the scanning tasks.

[0178] In some embodiments, please refer to Figure 6 , Figure 6This is a second structural schematic diagram of the electronic device provided in an embodiment of this application. The electronic device 400 further includes: a radio frequency circuit 403, a display screen 404, a control circuit 405, an input unit 406, an audio circuit 407, a sensor 408, and a power supply 409. The processor 401 is electrically connected to the radio frequency circuit 403, the display screen 404, the control circuit 405, the input unit 406, the audio circuit 407, the sensor 408, and the power supply 409.

[0179] The radio frequency circuit 403 is used to transmit and receive radio frequency signals to communicate with network devices or other electronic devices via wireless communication.

[0180] The display screen 404 can be used to display information input by the user or information provided to the user, as well as various graphical user interfaces of electronic devices, which can be composed of images, text, icons, videos, and any combination thereof.

[0181] The control circuit 405 is electrically connected to the display screen 404 and is used to control the display screen 404 to display information.

[0182] The input unit 406 can be used to receive input numeric or character information or user characteristic information (such as fingerprints), and to generate keyboard, mouse, joystick, optical, or trackball signal inputs related to user settings and function control. The input unit 406 may include a fingerprint recognition module.

[0183] The audio circuit 407 provides an audio interface between the user and the electronic device via a speaker and a microphone. The audio circuit 407 includes a microphone, which is electrically connected to the processor 401. The microphone is used to receive voice information input by the user.

[0184] Sensor 408 is used to collect information about the external environment. Sensor 408 may include one or more sensors such as an ambient light sensor, an accelerometer, and a gyroscope.

[0185] The power supply 409 is used to supply power to the various components of the electronic device 400. In some embodiments, the power supply 409 can be logically connected to the processor 401 through a power management system, thereby enabling functions such as managing charging, discharging, and power consumption through the power management system.

[0186] Although not shown in the figure, electronic device 400 may also include a camera, Bluetooth module, etc., which will not be described in detail here.

[0187] In this embodiment, the processor 401 in the electronic device 400 loads the instructions corresponding to the processes of one or more computer programs into the memory 402 according to the following steps, and the processor 401 runs the computer programs stored in the memory 402 to realize various functions:

[0188] Obtain at least one scan task for a target file submitted by a user, wherein the scan task includes a scan task regular expression and the attribute type of the file to be scanned;

[0189] The scanning task is pre-aggregated based on the attribute type of the file to be scanned to obtain at least one pre-aggregated task.

[0190] Based on the regular expression of the scanning task corresponding to the pre-aggregation task, the pre-aggregation task is aggregated to obtain the aggregation regular expression corresponding to each pre-aggregation task, and the target aggregation task is generated based on the aggregation regular expression.

[0191] Based on the target aggregation task, the target file is scanned to obtain target data that matches each of the scanning tasks.

[0192] In some implementations, when processor 401 performs the pre-aggregation processing on the scanning task based on the attribute type of the file to be scanned to obtain at least one pre-aggregation task, it may perform the following:

[0193] Based on the attribute type of the file to be scanned, the scanning task is divided to obtain at least one pre-aggregation task.

[0194] In some implementations, when processor 401 performs the step of dividing the scanning task based on the attribute type of the file to be scanned to obtain at least one pre-aggregation task, it may perform the following:

[0195] Each scan task belonging to the same attribute type of the file to be scanned is considered as a pre-aggregation task, resulting in at least one pre-aggregation task.

[0196] In some implementations, the processor 401 executes regular expressions based on the scan tasks corresponding to the pre-aggregation tasks to aggregate the pre-aggregation tasks, obtaining an aggregation regular expression corresponding to each pre-aggregation task. When generating a target aggregation task based on the aggregation regular expressions, the processor 401 may execute the following:

[0197] Based on a preset regular expression matching library, the regular expressions of the scanning tasks corresponding to the pre-aggregation tasks are aggregated to obtain the aggregated regular expression corresponding to each pre-aggregation task.

[0198] The target aggregation task is determined based on the aggregation regular expression.

[0199] In some implementations, the processor 401 performs the aggregation processing of the regular expressions corresponding to the scan tasks in the pre-aggregation tasks to obtain an aggregation regular expression corresponding to each pre-aggregation task. When determining the target aggregation task based on the aggregation regular expression, the processor may perform the following:

[0200] The pre-aggregation task is assigned to multiple target computing nodes, so that the target computing nodes perform aggregation processing based on the regular expression of the corresponding scan task in the pre-aggregation task to obtain an aggregation regular expression for the pre-aggregation task, and determine the target aggregation task based on the aggregation regular expression;

[0201] The step of scanning the target file based on the target aggregation task to obtain target data matching each scanning task includes:

[0202] The multiple target computing nodes obtain target data matching each scanning task by scanning the target file based on the target aggregation task.

[0203] In some implementations, when processor 401 executes at least one scan task submitted by the user for a target file, it may perform the following:

[0204] Obtain the user's input regarding the scanning task and generate prompt words for that task;

[0205] The prompt words generated for the scanning task are input into the large model for generating scanning tasks, and the scanning tasks that generate prompt words for the scanning tasks are matched.

[0206] In some implementations, processor 401 may perform the following training steps to generate a large model from the scanning task:

[0207] Obtain a basic large language model, and create an initial large model for generating prompt words for scanning tasks based on the basic large language model;

[0208] Obtain sample scanning task prompts and label the sample scanning task prompts with scanning task tags.

[0209] The prompt words generated by the sample scanning task are used to train the large model generated by the initial scanning task for at least one round.

[0210] During the forward propagation training of the model, a large model is generated based on the initial scanning task to generate prompt words to determine the predicted scanning task.

[0211] During the backpropagation training of the model, the scanning task generation loss is determined based on the predicted scanning task and the scanning task label. The scanning task generation loss is used to adjust the model parameters of the initial scanning task generation model until the initial scanning task generation model ends the model training, thus obtaining the scanning task generation model.

[0212] This application also provides a computer-readable storage medium storing a computer program. When the computer program is run on a computer, the computer executes the data processing method described in any of the above embodiments.

[0213] It should be noted that those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, which may include, but is not limited to, read-only memory (ROM), random access memory (RAM), magnetic disk or optical disk, etc.

[0214] Furthermore, the terms "first," "second," and "third," etc., used in this application are used to distinguish different objects, not to describe a specific order. Additionally, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or modules is not limited to the listed steps or modules, but some embodiments may also include steps or modules not listed, or some embodiments may include other steps or modules inherent to these processes, methods, products, or devices.

[0215] The data processing methods, apparatus, storage media, and electronic devices provided in the embodiments of this application have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the embodiments above are only for the purpose of helping to understand the methods and core ideas of this application; at the same time, those skilled in the art will recognize that, based on the ideas of this application, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A data processing method, characterized in that, include: Obtain at least one scan task for a target file submitted by a user, wherein the scan task includes a scan task regular expression and the attribute type of the file to be scanned; The scanning task is pre-aggregated based on the attribute type of the file to be scanned to obtain at least one pre-aggregated task. Based on the regular expression of the scanning task corresponding to the pre-aggregation task, the pre-aggregation task is aggregated to obtain the aggregation regular expression corresponding to each pre-aggregation task, and the target aggregation task is generated based on the aggregation regular expression. Based on the target aggregation task, the target file is scanned to obtain target data that matches each of the scanning tasks.

2. The method according to claim 1, characterized in that, The step of pre-aggregating the scanning task according to the attribute type of the file to be scanned to obtain at least one pre-aggregated task includes: Based on the attribute type of the file to be scanned, the scanning task is divided to obtain at least one pre-aggregation task.

3. The method according to claim 2, characterized in that, The step of dividing the scanning task based on the attribute type of the file to be scanned to obtain at least one pre-aggregation task includes: Each scan task belonging to the same attribute type of the file to be scanned is considered as a pre-aggregation task, resulting in at least one pre-aggregation task.

4. The method according to claim 2, characterized in that, The step of aggregating the pre-aggregation tasks based on the regular expressions of the corresponding scanning tasks in the pre-aggregation tasks to obtain an aggregation regular expression for each pre-aggregation task, and generating a target aggregation task based on the aggregation regular expressions, includes: Based on a preset regular expression matching library, the regular expressions of the scanning tasks corresponding to the pre-aggregation tasks are aggregated to obtain the aggregated regular expression corresponding to each pre-aggregation task. The target aggregation task is determined based on the aggregation regular expression.

5. The method according to claim 4, characterized in that, The step of aggregating the regular expressions of the corresponding scanning tasks in the pre-aggregation tasks to obtain the aggregated regular expression for each pre-aggregation task, and determining the target aggregation task based on the aggregated regular expression, includes: The pre-aggregation task is assigned to multiple target computing nodes, so that the target computing nodes perform aggregation processing based on the regular expression of the corresponding scan task in the pre-aggregation task to obtain an aggregation regular expression for the pre-aggregation task, and determine the target aggregation task based on the aggregation regular expression; The step of scanning the target file based on the target aggregation task to obtain target data matching each scanning task includes: The multiple target computing nodes obtain target data matching each scanning task by scanning the target file based on the target aggregation task.

6. The method according to claim 1, characterized in that, The process of obtaining at least one scan task for a target file submitted by the user includes: Obtain the user's input regarding the scanning task and generate prompt words for that task; The prompt words generated for the scanning task are input into the large model for generating scanning tasks, and the scanning tasks that generate prompt words for the scanning tasks are matched.

7. The method according to claim 6, characterized in that, The large model generated by the scanning task is trained according to the following steps: Obtain a basic large language model, and create an initial large model for generating prompt words for scanning tasks based on the basic large language model; Obtain sample scanning task prompts and label the sample scanning task prompts with scanning task tags. The prompt words generated by the sample scanning task are used to train the large model generated by the initial scanning task for at least one round. During the forward propagation training of the model, a large model is generated based on the initial scanning task to generate prompt words to determine the predicted scanning task. During the backpropagation training of the model, the scanning task generation loss is determined based on the predicted scanning task and the scanning task label. The scanning task generation loss is used to adjust the model parameters of the initial scanning task generation model until the initial scanning task generation model ends the model training, thus obtaining the scanning task generation model.

8. A data processing apparatus, characterized in that, include: The acquisition module is used to acquire at least one scanning task for a target file submitted by the user, wherein the scanning task includes a scanning task regular expression and the attribute type of the file to be scanned. A pre-aggregation processing module is used to perform pre-aggregation processing on the scanning task according to the attribute type of the file to be scanned, so as to obtain at least one pre-aggregation task. The aggregation processing module is used to aggregate the pre-aggregation tasks based on the regular expression of the scanning task corresponding to the pre-aggregation task, to obtain the aggregation regular expression corresponding to each pre-aggregation task, and to generate the target aggregation task based on the aggregation regular expression. The matching module is used to perform data scanning on the target file based on the target aggregation task to obtain target data that matches each of the scanning tasks.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is run on a computer, it causes the computer to perform the data processing method as described in any one of claims 1 to 7.

10. An electronic device comprising a processor and a memory, the memory storing a computer program, characterized in that, The processor executes the data processing method as described in any one of claims 1 to 7 by invoking the computer program.

Images