Security score generation method and device, equipment, medium and program product

Abstract

The invention provides a safety score generation method. The method can be applied to the technical field of artificial intelligence, the technical field of big data and application of large models in the field of financial science and technology. The method comprises the steps that a original structured query statements are obtained, local duplicate removal is conducted on the a original structured query statements, and b first structured query statements are generated; and performing distributed cache deduplication on the b first structured query statements to generate c second structured query statements. And performing event stream deduplication on the c second structured query statements to generate d target structured query statements. And obtaining d execution plans corresponding to the target structured query statement. And outputting security scores of the d target structured query statements through a pre-trained security score large model. The invention further provides a safety score generation device and equipment, a storage medium and a program product.

Description

Technical Field

[0001] This application relates to the fields of big data technology and artificial intelligence technology, specifically to the application of large models in the field of financial technology, and particularly to a security scoring generation method, apparatus, device, medium and program product. Background Technology

[0002] With the advancement of technology, enterprises, including banks, are increasingly demanding higher security for SQL statements on distributed servers. Current technologies typically involve using rule bases or manual analysis to process massive amounts of SQL statements for security assessment, or directly inputting these statements into a large security scoring model to generate security scores. However, this process requires evaluating a large number of repetitive SQL statements, which leads to interference and reduces the accuracy and reliability of the generated security scores. Furthermore, it results in low computer processing efficiency and wasted computer resources. Summary of the Invention

[0003] In view of the above problems, this application provides a method, apparatus, device, medium and program product for generating security scores.

[0004] According to a first aspect of this application, a security score generation method is provided, the method comprising: obtaining *a* original structured query statements; performing local deduplication, distributed cache deduplication, and event stream deduplication on the *a* original structured query statements to generate *d* target structured query statements; storing the *d* target structured query statements in a data lake, wherein *a* is an integer greater than 1, and *d* is an integer greater than 0 and less than *a*; obtaining the *d* target structured query statements from the data lake; obtaining *d* execution plans corresponding to the *d* target structured query statements; and inputting the *d* target structured query statements and the *d* execution plans corresponding to the *d* target structured query statements into a pre-trained security score large model to output a security score for the *d* target structured query statements.

[0005] According to an embodiment of this application, obtaining *a* original structured query statements, performing local deduplication, distributed cache deduplication, and event stream deduplication on the *a* original structured query statements to generate *d* target structured query statements, includes: obtaining *a* original structured query statements, loading the *a* original structured query statements into a local cache, performing local deduplication on the *a* original structured query statements to generate *b* first structured query statements, and storing the *b* first structured query statements in the data lake, wherein... b is an integer, greater than 0 and less than a; b first structured query statements are obtained from the data lake, loaded into a distributed cache, and deduplicated using the distributed cache to generate c second structured query statements, which are then stored in the data lake, where c is an integer, greater than 0 and less than b; c second structured query statements are obtained from the data lake, and deduplicated using an event stream to generate d target structured query statements, where d is an integer, greater than 0 and less than c.

[0006] According to an embodiment of this application, local deduplication is performed on the a original structured query statements to generate b first structured query statements, including: obtaining e duplicate third structured query statements from the a original structured query statements within a first preset time period using a cache eviction algorithm, where e is an integer, e is greater than 0 and less than a; and merging and deduplicating the e duplicate third structured query statements from the a original structured query statements within the first preset time period to generate the b first structured query statements.

[0007] According to an embodiment of this application, the b first structured query statements are deduplicated using distributed caching to generate c second structured query statements. This includes: obtaining f fourth structured query statements whose time-to-live has expired from the b first structured query statements using a distributed Bloom filter, where f is an integer, f is greater than 0 and less than b, and the sum of f and c equals b; and deleting the f fourth structured query statements from the b first structured query statements to generate the c second structured query statements.

[0008] According to an embodiment of this application, event flow deduplication is performed on the c second structured query statements to generate d target structured query statements, including: obtaining g fifth structured query statements that have completed checkpoint fault recovery from the c second structured query statements through event flow detection, where g is an integer, g is greater than 0 and less than c, and the sum of g and d is equal to c; obtaining g sixth structured query statements that have experienced faults corresponding to the g fifth structured query statements that have completed checkpoint fault recovery; and deleting the g sixth structured query statements from the c second structured query statements to generate the d target structured query statements.

[0009] According to an embodiment of this application, obtaining the d target structured query statements from the data lake includes: obtaining the d target structured query statements from the data lake in a data stream manner through a remote service interface.

[0010] According to an embodiment of this application, training a large-scale security scoring model includes: pre-setting expert rules; constructing a retrieval enhancement generation knowledge base based on the expert rules; obtaining h historical structured query statements and their corresponding security scores; performing retrieval enhancement generation on the h historical structured query statements and their corresponding security scores according to the retrieval enhancement generation knowledge base to generate i historical structured query statements and their corresponding security scores, where h is an integer and h is greater than 1, and i is an integer and i is greater than h; and training the initial large-scale model based on the i historical structured query statements and their corresponding security scores to generate the large-scale security scoring model.

[0011] According to an embodiment of this application, the method further includes: obtaining n third structured query statements among the d target structured query statements whose security scores are lower than a preset threshold, and sending the n third structured query statements, where n is an integer, n is greater than 1 and n is less than d.

[0012] According to a second aspect of this application, a security score generation apparatus is provided, comprising: a first deduplication module, configured to acquire a original structured query statements, perform local deduplication, distributed cache deduplication, and event stream deduplication on the a original structured query statements to generate d target structured query statements, and store the d target structured query statements in a data lake, wherein a is an integer and a is greater than 1, d is an integer and d is greater than 0 and less than a; a fourth acquisition module, configured to acquire the d target structured query statements from the data lake, and acquire d execution plans corresponding to the target structured query statements based on the d target structured query statements; and a first generation module, configured to input the d target structured query statements and the d execution plans corresponding to the target structured query statements into a pre-trained security score large model, and output a security score for the d target structured query statements.

[0013] According to a third aspect of this application, an electronic device is provided, comprising: one or more processors; and a memory for storing one or more computer programs, wherein the one or more processors execute the one or more computer programs to implement the steps of the method described above.

[0014] A fourth aspect of this application also provides a computer-readable storage medium having a computer program or instructions stored thereon, which, when executed by a processor, implement the steps of the above-described method.

[0015] The fifth aspect of this application also provides a computer program product, including a computer program or instructions that, when executed by a processor, implement the steps of the above-described method.

[0016] This application employs a multi-layered, progressive deduplication and refinement process—including local deduplication, distributed caching deduplication, and event stream deduplication—to effectively reduce the amount of data processed for inputting pre-trained security scoring models. This significantly improves the processing efficiency of massive amounts of structured queries, saves computational resources, and enhances the reliability and accuracy of the input data for the pre-trained security scoring models. Furthermore, by refining the input data of the pre-trained security scoring models, the reliability and accuracy of the security scores generated for the target structured queries are significantly improved. This solves the problems of low computer processing efficiency, wasted computer resources, and poor accuracy and reliability of the generated security scores caused by relying on rule bases or manual analysis to judge the security of massive amounts of structured queries, or by directly inputting massive amounts of structured queries into security scoring models to generate security scores in existing technologies. Attached Figure Description

[0017] The above-mentioned contents, other objects, features and advantages of this application will become clearer from the following description of embodiments with reference to the accompanying drawings, in which:

[0018] Figure 1 The illustrations depict application scenarios of the security score generation method, apparatus, device, medium, and program products according to embodiments of this application.

[0019] Figure 2 A flowchart illustrating a security score generation method according to an embodiment of this application is shown schematically.

[0020] Figure 3 This illustration schematically shows a flowchart of generating b first structured statements in a security score generation method according to an embodiment of this application;

[0021] Figure 4 This illustration schematically shows a flowchart of generating c second structured statements in a security score generation method according to an embodiment of this application;

[0022] Figure 5 This illustration schematically shows a flowchart of generating d target structured statements in a security score generation method according to an embodiment of this application;

[0023] Figure 6 The flowchart illustrating the training of a large security score model in the security score generation method according to an embodiment of this application is shown in the schematic diagram.

[0024] Figure 7 This schematic diagram illustrates the structure of a security score generation apparatus according to an embodiment of the present application;

[0025] Figure 8 A block diagram schematically illustrates an electronic device suitable for implementing a security score generation method according to an embodiment of this application. Detailed Implementation

[0026] The embodiments of this application will now be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of this application. In the following detailed description, numerous specific details are set forth to provide a thorough understanding of the embodiments of this application for ease of explanation. However, it will be apparent that one or more embodiments may be implemented without these specific details. Furthermore, descriptions of well-known structures and technologies are omitted in the following description to avoid unnecessarily obscuring the concepts of this application.

[0027] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the scope of this application. The terms “comprising,” “including,” etc., as used herein indicate the presence of the stated features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0028] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein are to be interpreted in a manner consistent with the context of this specification, and not in an idealized or overly rigid way.

[0029] When using expressions such as "at least one of A, B and C", they should generally be interpreted in accordance with the meaning that is commonly understood by those skilled in the art (e.g., "a system having at least one of A, B and C" should include, but is not limited to, a system having A alone, a system having B alone, a system having C alone, a system having A and B, a system having A and C, a system having B and C, and / or a system having A, B and C, etc.).

[0030] The accompanying drawings show some block diagrams and / or flowcharts. It should be understood that some blocks or combinations thereof in the block diagrams and / or flowcharts can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable control device, so that when executed by the processor, these instructions can create means for implementing the functions / operations described in these block diagrams and / or flowcharts.

[0031] First, the technical terms used in this article are explained as follows:

[0032] SQL (Structured Query Language) is a computer language specifically designed for communicating and managing databases.

[0033] Bytecode access tools are libraries or tools that allow direct manipulation or modification of Java class files (i.e., bytecode) after a Java program is compiled but before (or during) execution. They allow changes to the program's behavior without modifying the source code.

[0034] A Bloom filter is a highly space-efficient probabilistic data structure used to determine whether an element exists in a set. Adding a Bloom filter before caching allows for a quick determination of whether requested data might exist, avoiding database queries for non-existent data.

[0035] TTL (Time To Live) is a timer used to set a countdown for a data packet or set of data. When the countdown reaches zero, the packet is discarded or deleted. It can prevent data from looping indefinitely in a network and ensure data freshness in a cache.

[0036] A checkpoint is used to save the current state of a system or application at a specific point in time so that it can be quickly restored from this saved state in the event of a failure, rather than starting from scratch.

[0037] A data lake is a centralized repository that can store massive amounts of raw data of various types. It allows all structured and unstructured data to be stored at any scale and can store data as is without prior structuring. Data lakes are typically built on top of scalable distributed storage systems.

[0038] A data stream is a continuous flow of data, consisting of thousands of data records generated sequentially, though the total data volume may be small. These data records arrive in chronological order and are characterized by high speed, interconnectedness, unordered nature, and infinity.

[0039] RAG (Retrieval Augmented Generation) is a technical framework that combines information retrieval systems with the generative capabilities of large language models.

[0040] This application provides a method for generating security scores. The method includes: acquiring *a* original structured query statements; performing local deduplication, distributed cache deduplication, and event stream deduplication on the *a* original structured query statements to generate *d* target structured query statements; storing the *d* target structured query statements in a data lake, where *a* is an integer greater than 1, and *d* is an integer greater than 0 and less than *a*; acquiring the *d* target structured query statements from the data lake; acquiring *d* execution plans corresponding to the *d* target structured query statements; and inputting the *d* target structured query statements and the *d* execution plans corresponding to the target structured query statements into a pre-trained security scoring model to output a security score for the *d* target structured query statements.

[0041] According to the embodiments of this application, by employing a multi-level, progressive deduplication and refinement process involving local deduplication, distributed cache deduplication, and event stream deduplication on the original structured query statements, the data processing volume of the input data to the pre-trained security scoring model is effectively reduced. This significantly improves the processing efficiency of massive structured query statements, saves computing resources, and simultaneously enhances the reliability and accuracy of the input data to the pre-trained security scoring model. Furthermore, by refining the input data of the pre-trained security scoring model, the reliability and accuracy of the security scores for the generated target structured query statements are significantly improved. This solves the technical problems of low computer processing efficiency, wasted computer resources, and poor accuracy and reliability of the generated security scores caused by judging the security of massive structured query statements through rule bases or manual analysis in existing technologies, or by directly inputting massive structured query statements into the security scoring model to generate security scores.

[0042] Figure 1 The illustration schematically depicts application scenarios of the security score generation method, apparatus, device, medium, and program product according to embodiments of this application. It should be noted that... Figure 1 The examples shown are merely examples of scenarios in which the embodiments of this application can be applied, to help those skilled in the art understand the technical content of this application, but do not mean that the embodiments of this application cannot be used in other devices, systems, environments or scenarios.

[0043] like Figure 1 As shown, application scenario 100 according to this embodiment may include an application scenario for generating security scores. Network 104 serves as a medium for providing a communication link between the first terminal device 101, the second terminal device 102, the third terminal device 103, and the server 105. Network 104 may include various connection types, such as wired or wireless communication links or fiber optic cables, etc.

[0044] Users can use the first terminal device 101, the second terminal device 102, and the third terminal device 103 to interact with the server 105 via the network 104 to receive or send messages, etc. Various communication client applications can be installed on the first terminal device 101, the second terminal device 102, and the third terminal device 103, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social media platform software, etc. (for example only).

[0045] The first terminal device 101, the second terminal device 102, and the third terminal device 103 can be various electronic devices with displays and support web browsing, including but not limited to smartphones, tablets, laptops, and desktop computers.

[0046] Server 105 can be a server that provides various services, such as a backend management server that supports websites browsed by users using the first terminal device 101, the second terminal device 102, and the third terminal device 103 (this is just an example). The backend management server can analyze and process data such as received user requests, and feed back the processing results (such as web pages, information, or data obtained or generated according to user requests) to the terminal devices.

[0047] It should be noted that the security score generation method provided in this application embodiment can generally be executed by server 105. Correspondingly, the security score generation device provided in this application embodiment can generally be located in server 105. The security score generation method provided in this application embodiment can also be executed by a server or server cluster that is different from server 105 and capable of communicating with the first terminal device 101, the second terminal device 102, the third terminal device 103, and / or server 105. Correspondingly, the security score generation device provided in this application embodiment can also be located in a server or server cluster that is different from server 105 and capable of communicating with the first terminal device 101, the second terminal device 102, the third terminal device 103, and / or server 105.

[0048] It should be understood that Figure 1 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.

[0049] The following will be based on Figure 1 The described scene, through Figures 2-6 The security score generation method of the disclosed embodiments is described in detail. It should be noted that the above application scenarios are shown only to facilitate understanding of the spirit and principles of this application, and the implementation of this application is not limited in any way. Rather, the implementation of this application can be applied to any applicable scenario.

[0050] Figure 2 A flowchart illustrating a security score generation method according to an embodiment of this application is shown schematically.

[0051] like Figure 2 As shown, the method 200 includes steps S201 to S205.

[0052] According to an embodiment of this application, in steps S201 to S203, *a* original structured query statements can be obtained. These *a* original structured query statements are then subjected to local deduplication, distributed cache deduplication, and event stream deduplication to generate *d* target structured query statements. These *d* target structured query statements are then stored in a data lake, where *a* is an integer greater than 1, and *d* is an integer greater than 0 and less than *a*. By performing multi-level, progressive deduplication on the original structured query statements through local deduplication, distributed cache deduplication, and event stream deduplication, data processing efficiency is effectively improved, computing resource utilization is increased, and network and distributed resource overhead is reduced.

[0053] Step S201: Obtain a original structured query statements, load the a original structured query statements into the local cache, perform local deduplication on the a original structured query statements to generate b first structured query statements, and store the b first structured query statements in the data lake, where a is an integer and a is greater than 1, b is an integer and b is greater than 0 and less than a.

[0054] For example, the application can send a raw structured query statement, which can be obtained through a bytecode access tool. The bytecode access tool will intercept and collect the complete SQL statement when the program executes the raw structured query statement. The raw structured query statement can be obtained through bytecode enhancement technology. It can be dynamically enabled / disabled without modifying the business code, effectively realizing hot-swappable access.

[0055] Figure 3 The flowchart illustrating the generation of b first structured statements in the security score generation method according to an embodiment of this application is shown in the illustration.

[0056] like Figure 3 As shown, the method 300 includes steps S301 to S302.

[0057] Step S301: Obtain e repeated third structured query statements from the a original structured query statements within a first preset time period using a cache eviction algorithm, where e is an integer, greater than 0 and less than a.

[0058] For example, the first preset time can be set to 10 seconds, and the local cache component can use a cache eviction algorithm to retrieve e duplicate SQL statements that are instantaneously repeated within 10 seconds from a SQL statement.

[0059] Step S302: Merge and deduplicate the e repeated third structured query statements in the a original structured query statements within a first preset time period to generate the b first structured query statements.

[0060] For example, we can merge and deduplicate e duplicate SQL statements that are instantaneously repeated within 10 seconds from a SQL statement. This can effectively eliminate duplicate SQL statements, release local cache resources, and improve memory resource utilization.

[0061] By using a cache eviction algorithm to process duplicate structured query statements within a first preset time period, the timeliness of data deduplication can be improved. This allows local deduplication of the original structured query statements to focus on duplicate structured query statements within the first preset time period, effectively improving the efficiency of local deduplication, increasing the utilization of local resources, and reducing the overhead of network and distributed resources.

[0062] Return to reference Figure 2 In step S202, the b first structured query statements are obtained from the data lake, loaded into a distributed cache, and deduplicated by the distributed cache is performed on the b first structured query statements to generate c second structured query statements. The c second structured query statements are then stored in the data lake, where c is an integer, greater than 0 and less than b.

[0063] Figure 4 The flowchart illustrating the generation of c second structured statements in the security score generation method according to an embodiment of this application is shown in the illustration.

[0064] like Figure 4 As shown, the method 400 includes steps S401 to S402.

[0065] Step S401: Obtain f fourth structured query statements whose time-to-live has expired from the b first structured query statements using a distributed Bloom filter, where f is an integer, f is greater than 0 and less than b, and the sum of f and c is equal to b.

[0066] For example, a distributed Bloom filter can be used to dynamically adjust the TTL adaptive lifespan through batch asynchronous updates, and f SQL statements whose lifespan has expired can be obtained from b SQL statements.

[0067] Step S402: Delete the f fourth structured query statements from the b first structured query statements, and generate the c second structured query statements.

[0068] By extending the first structured query statement to the entire distributed environment through a distributed Bloom filter, and combining it with structured queries that have reached the end of their lifecycle to perform data deduplication on distributed nodes of spatial nodes, global data deduplication in the distributed environment can be achieved, significantly improving the efficiency and reliability of data deduplication.

[0069] Return to reference Figure 2 In step S203, the c second structured query statements are obtained from the data lake, and the c second structured query statements are deduplicated by event stream to generate d target structured query statements. The d target structured query statements are then stored in the data lake, where d is an integer, greater than 0 and less than c.

[0070] Figure 5 The flowchart illustrating the generation of d target structured statements in the security score generation method according to an embodiment of this application is shown in the illustration.

[0071] like Figure 5 As shown, the method 500 includes steps S501 to S503.

[0072] Step S501: Obtain g fifth structured query statements that complete checkpoint fault recovery from the c second structured query statements through event flow detection, where g is an integer, g is greater than 0 and less than c, and the sum of g and d is equal to c.

[0073] Step S502: Obtain the g sixth structured query statements that have failed, corresponding to the g fifth structured query statements that have completed the checkpoint fault recovery.

[0074] Step S503: Delete the g sixth structured query statements from the c second structured query statements, and generate the d target structured query statements.

[0075] By utilizing the checkpoint fault recovery mechanism in event stream detection, data accuracy can be improved by eliminating data duplication caused by fault recovery. Simultaneously, event stream detection can be used for deduplication of time series data, effectively improving both the efficiency and accuracy of data deduplication.

[0076] Return to reference Figure 2 In step S204, the d target structured query statements are obtained from the data lake, and the d execution plans corresponding to the target structured query statements are obtained based on the d target structured query statements.

[0077] For example, the d target structured query statements can be obtained from the data lake via a remote service interface in a data streaming manner. Obtaining data from the data lake via a remote service interface in a data streaming manner improves data processing efficiency, shortens overall data processing time, significantly improves data processing efficiency, and effectively reduces data latency.

[0078] Step S205: Input the d target structured query statements and the d execution plans corresponding to the target structured query statements into the pre-trained security scoring model, and output the security scores of the d target structured query statements.

[0079] Figure 6 The flowchart illustrating the process of training a large security score model in the security score generation method according to an embodiment of this application is shown.

[0080] like Figure 6 As shown, the method 600 includes steps S601 to S603.

[0081] Step S601: Preset expert rules and construct a retrieval-enhanced knowledge base based on the expert rules.

[0082] For example, pre-defined expert rules could include: classifying statements according to their safety status based on an expert database, and creating manual ratings based on execution plan characteristics. For instance, these could be categorized into four levels: high-risk, medium-risk, low-risk, and normal. A search-enhanced knowledge base could then be built based on these expert database classifications.

[0083] Step S602: Obtain h historical structured query statements and their corresponding security scores. Based on the retrieval enhancement generation knowledge base, perform retrieval enhancement generation on the h historical structured query statements and their corresponding security scores to generate i historical structured query statements and their corresponding security scores, where h is an integer and h is greater than 1, and i is an integer and i is greater than h.

[0084] Step S603: Train the initial large model based on the i historical structured query statements and the security scores of the corresponding i historical structured query statements to generate the security score large model.

[0085] By constructing a knowledge base using pre-defined expert rules and retrieving enhanced generative training data, the training efficiency and quality of the model are effectively improved, the interpretability of the model is enhanced, and the reliability and credibility of the generated model are significantly improved.

[0086] Furthermore, before inputting the d target structured queries and their corresponding d execution plans into the pre-trained security scoring model, the d execution plans can be analyzed to generate an execution plan data summary. This summary can then be used to fine-tune the parameters of the security scoring model. Then, the d target structured queries and their corresponding d execution plans are input into the pre-trained security scoring model, which outputs security scores for the d target structured queries. Fine-tuning the parameters of the security scoring model can improve the accuracy of the output security scores.

[0087] The method further includes: obtaining n third structured query statements among the d target structured query statements whose security scores are lower than a preset threshold, and sending the n third structured query statements, where n is an integer, n is greater than 1 and n is less than d.

[0088] A warning can be issued for n third-party structured queries with security scores below a preset threshold. For example, these queries can be added to a watchlist, and alerts can be proactively sent via email and SMS to relevant parties to remind them to rectify the issues and monitor the progress. Specifically, this can include periodically performing regression analysis on the queries requiring rectification; if rectification is complete, they are automatically removed from the watchlist, completing an automated closed-loop process. If rectification is not completed by the deadline, a new reminder is issued.

[0089] By sending structured query statements with security scores below a preset threshold, proactive warnings and real-time responses to security risks can be achieved, significantly improving security operation and maintenance efficiency.

[0090] According to an embodiment of this application, a bytecode access tool intercepts and collects complete SQL statements when the program executes SQL. After local deduplication and distributed cache deduplication, a data stream message is asynchronously sent for further deduplication, and then the records are added to the data lake. The data lake table is continuously read in a streaming manner. The SQL statements in the returned record results are used as parameters to call a remote service interface provided by the tool to obtain the execution plan. The remote service interface provides the ability to query the production database and obtain the execution plan. The data lake is scanned, and the query result SQL statements and execution plans are used as input parameters to a security scoring model trained with the RAG knowledge base to generate a score for the secure SQL statement. If the score of the SQL statement is considered too low, and the SQL statement is considered a high-risk statement, the relevant content of the high-risk SQL statement is sent to relevant parties via email or SMS. After receiving the notification, relevant parties provide feedback on the governance plan. When the security scoring model continues scanning, it re-evaluates the SQL statement based on the feedback plan. If it is no longer a high-risk statement, the governance is considered complete and the loop is automatically closed, and no further notification is given; if it is still a high-risk statement, relevant parties are notified again.

[0091] The embodiments of this application reduce most redundant SQL transmissions through a three-level progressive deduplication process, thereby lowering transmission costs and improving transmission efficiency. Furthermore, asynchronous transmission of distributed data streams reduces the risk of business thread blocking. Overall, by refining the model input data, the technical means effectively improve the reliability and accuracy of the output data of the large security scoring model.

[0092] Figure 7 A schematic block diagram of a security score generation apparatus according to an embodiment of this application is shown.

[0093] like Figure 7 As shown, the device 700 includes: a first acquisition module 701, a second acquisition module 702, a third acquisition module 703, a fourth acquisition module 704, and a first generation module 705. The first acquisition module 701, the second acquisition module 702, and the third acquisition module 703 are included in a first deduplication module. The first deduplication module is used to acquire a original structured query statements, perform local deduplication, distributed cache deduplication, and event stream deduplication on the a original structured query statements, generate d target structured query statements, and store the d target structured query statements in a data lake. Here, a is an integer and a is greater than 1, d is an integer and d is greater than 0 and less than a.

[0094] The first acquisition module 701 is used to acquire *a* original structured query statements, load the *a* original structured query statements into a local cache, perform local deduplication on the *a* original structured query statements to generate *b* first structured query statements, and store the *b* first structured query statements in a data lake, wherein *a* is an integer greater than 1, and *b* is an integer greater than 0 and less than *a*. In one embodiment, the first acquisition module 701 can be used to execute step S201 described above.

[0095] The first acquisition module 701 includes: a fifth acquisition module and a second generation module.

[0096] The fifth acquisition module is used to acquire e repeated third structured query statements from the a original structured query statements within a first preset time period using a cache eviction algorithm, where e is an integer, greater than 0 and less than a. In one embodiment, the fifth acquisition module can be used to execute step S301 described above, which will not be repeated here.

[0097] The second generation module is used to merge and deduplicate e repeated third structured query statements within a first preset time period from the a original structured query statements to generate the b first structured query statements. In one embodiment, the second generation module can be used to execute step S302 described above, which will not be repeated here.

[0098] The second acquisition module 702 is configured to acquire the b first structured query statements from the data lake, load the b first structured query statements into a distributed cache, perform distributed cache deduplication on the b first structured query statements to generate c second structured query statements, and store the c second structured query statements in the data lake, where c is an integer, greater than 0 and less than b. In one embodiment, the second acquisition module 702 can be used to execute step S202 described above.

[0099] The second acquisition module 702 includes: a sixth acquisition module and a third generation module.

[0100] The sixth acquisition module is used to acquire f fourth structured query statements whose time-to-live has expired from the b first structured query statements using a distributed Bloom filter, where f is an integer, f is greater than 0 and less than b, and the sum of f and c equals b. In one embodiment, the sixth acquisition module can be used to execute step S401 described above, which will not be repeated here.

[0101] The third generation module is used to delete the f fourth structured query statements from the b first structured query statements and generate the c second structured query statements. In one embodiment, the third generation module can be used to execute step S402 described above, which will not be repeated here.

[0102] The third acquisition module 703 is used to acquire the c second structured query statements from the data lake, perform event stream deduplication on the c second structured query statements to generate d target structured query statements, and store the d target structured query statements in the data lake, where d is an integer, greater than 0 and less than c. In one embodiment, the third acquisition module 703 can be used to execute step S203 described above.

[0103] The third acquisition module 703 includes: the seventh acquisition module, the eighth acquisition module, and the fourth generation module.

[0104] The seventh acquisition module is used to acquire g fifth structured query statements that complete checkpoint fault recovery from the c second structured query statements through event flow detection, where g is an integer, g is greater than 0 and less than c, and the sum of g and d is equal to c. In one embodiment, the seventh acquisition module can be used to execute step S501 described above, which will not be repeated here.

[0105] The eighth acquisition module is used to acquire the g sixth structured query statements that have failed, corresponding to the g fifth structured query statements that have completed checkpoint fault recovery. In one embodiment, the eighth acquisition module can be used to execute step S502 described above, which will not be repeated here.

[0106] The fourth generation module is used to delete the g sixth structured query statements from the c second structured query statements and generate the d target structured query statements. In one embodiment, the fourth generation module can be used to execute step S503 described above, which will not be repeated here.

[0107] The fourth acquisition module 704 is used to acquire the d target structured query statements from the data lake, and to acquire d execution plans corresponding to the target structured query statements based on the d target structured query statements. In one embodiment, the fourth acquisition module 704 can be used to execute step S204 described above.

[0108] The fourth acquisition module 704 includes a ninth acquisition module. The ninth acquisition module is used to acquire the d target structured query statements from the data lake via a data stream through a remote service interface.

[0109] The first generation module 705 is used to input the d target structured query statements and the d execution plans corresponding to the target structured query statements into a pre-trained security scoring model, and output the security scores of the d target structured query statements. In one embodiment, the first generation module 705 can be used to execute step S205 described above.

[0110] The first generation module 705 includes: the fifth generation module, the sixth generation module, and the seventh generation module.

[0111] The fifth generation module is used to preset expert rules and construct a retrieval-enhanced knowledge base based on the expert rules. In one embodiment, the fifth generation module can be used to execute step S601 described above, which will not be repeated here.

[0112] The sixth generation module is used to obtain h historical structured query statements and their corresponding security scores, and to perform retrieval enhancement generation on the h historical structured query statements and their corresponding security scores based on the retrieval enhancement generation knowledge base, generating i historical structured query statements and their corresponding security scores, where h is an integer greater than 1, and i is an integer greater than h. In one embodiment, the sixth generation module can be used to execute step S602 described above, which will not be repeated here.

[0113] The seventh generation module is used to train the initial large model based on the i historical structured query statements and their corresponding security scores, thereby generating the security score large model. In one embodiment, the seventh generation module can be used to execute step S603 described above, which will not be repeated here.

[0114] The device further includes a tenth acquisition module. The tenth acquisition module is used to acquire n third structured query statements among the d target structured query statements whose security scores are lower than a preset threshold, and to send the n third structured query statements, where n is an integer, greater than 1 and less than d.

[0115] According to embodiments of this application, any plurality of modules among the first acquisition module 701, the second acquisition module 702, the third acquisition module 703, the fourth acquisition module 704, and the first generation module 705 can be combined into one module, or any one of these modules can be split into multiple modules. Alternatively, at least part of the functionality of one or more of these modules can be combined with at least part of the functionality of other modules and implemented in one module. According to embodiments of this application, at least one of the first acquisition module 701, the second acquisition module 702, the third acquisition module 703, the fourth acquisition module 704, and the first generation module 705 can be at least partially implemented as hardware circuitry, such as a field-programmable gate array (FPGA), a programmable logic array (PLA), a system-on-a-chip, a system-on-a-substrate, a system-on-package, an application-specific integrated circuit (ASIC), or implemented in hardware or firmware by any other reasonable means of integrating or packaging the circuitry, or implemented in any one of the three implementation methods of software, hardware, and firmware, or in a suitable combination of any of these. Alternatively, at least one of the first acquisition module 701, the second acquisition module 702, the third acquisition module 703, the fourth acquisition module 704, and the first generation module 705 can be implemented at least partially as a computer program module, which can perform corresponding functions when the computer program module is run.

[0116] Figure 8 A block diagram schematically illustrates an electronic device suitable for implementing a security score generation method according to an embodiment of this application.

[0117] like Figure 8As shown, an electronic device 800 according to an embodiment of this application includes a processor 801, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 802 or a program loaded from a storage portion 808 into a random access memory (RAM) 803. The processor 801 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or an associated chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 801 may also include onboard memory for caching purposes. The processor 801 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of this application.

[0118] RAM 803 stores various programs and data required for the operation of electronic device 800. Processor 801, ROM 802, and RAM 803 are interconnected via bus 804. Processor 801 executes various operations of the method flow according to embodiments of this application by executing programs in ROM 802 and / or RAM 803. It should be noted that the programs may also be stored in one or more memories other than ROM 802 and RAM 803. Processor 801 may also execute various operations of the method flow according to embodiments of this application by executing programs stored in said one or more memories.

[0119] According to embodiments of this application, the electronic device 800 may further include an input / output (I / O) interface 805, which is also connected to a bus 804. The electronic device 800 may also include one or more of the following components connected to the input / output (I / O) interface 805: an input section 806 including a keyboard, mouse, etc.; an output section 807 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 808 including a hard disk, etc.; and a communication section 809 including a network interface card such as a LAN card, modem, etc. The communication section 809 performs communication processing via a network such as the Internet. A drive 810 is also connected to the input / output (I / O) interface 805 as needed. A removable medium 811, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on the drive 810 as needed so that computer programs read from it can be installed into the storage section 808 as needed.

[0120] This application also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or it may exist independently and not assembled into the device / apparatus / system. The computer-readable storage medium carries one or more programs, which, when executed, implement the method according to the embodiments of this application.

[0121] According to embodiments of this application, the computer-readable storage medium can be a non-volatile computer-readable storage medium, such as including but not limited to: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this application, the computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. For example, according to embodiments of this application, the computer-readable storage medium may include ROM 802 and / or RAM 803 and / or one or more memories other than ROM 802 and RAM 803 described above.

[0122] Embodiments of this application also include a computer program product comprising a computer program containing program code for performing the methods shown in the flowchart. When the computer program product is run on a computer system, the program code is used to enable the computer system to implement the security score generation method provided in the embodiments of this application.

[0123] When the computer program is executed by the processor 801, it performs the functions defined in the system / apparatus of this application embodiment. According to the embodiments of this application, the systems, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0124] In one embodiment, the computer program may rely on a tangible storage medium such as an optical storage device or a magnetic storage device. In another embodiment, the computer program may also be transmitted and distributed in the form of signals over a network medium, and may be downloaded and installed via the communication section 809, and / or installed from a removable medium 811. The program code contained in the computer program can be transmitted using any suitable network medium, including but not limited to: wireless, wired, etc., or any suitable combination thereof.

[0125] In such an embodiment, the computer program can be downloaded and installed from a network via the communication section 809, and / or installed from the removable medium 811. When the computer program is executed by the processor 801, it performs the functions defined in the system of this application embodiment. According to the embodiments of this application, the systems, devices, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0126] According to embodiments of this application, program code for executing the computer programs provided in the embodiments of this application can be written in any combination of one or more programming languages. Specifically, these computational programs can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, languages ​​such as Java, C++, Python, "C", or similar programming languages. The program code can be executed entirely on the user's computing device, partially on the user's device, partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0127] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0128] Those skilled in the art will understand that the features described in the various embodiments of this application can be combined and / or combined in various ways, even if such combinations or combinations are not explicitly described in this application. In particular, the features described in the various embodiments of this application can be combined and / or combined in various ways without departing from the spirit and teachings of this application. All such combinations and / or combinations fall within the scope of this application.

Claims

1. A method for generating a security score, characterized in that, The method includes: Obtain a original structured query statements, perform local deduplication, distributed cache deduplication, and event stream deduplication on the a original structured query statements to generate d target structured query statements, and store the d target structured query statements in the data lake, where a is an integer and a is greater than 1, d is an integer and d is greater than 0 and less than a; Obtain the d target structured query statements from the data lake; and based on the d target structured query statements, obtain the d execution plans corresponding to the target structured query statements; and The d target structured query statements and the d execution plans corresponding to the target structured query statements are input into a pre-trained security scoring model, and the security scores of the d target structured query statements are output.

2. The method according to claim 1, characterized in that, Obtain *a* original structured query statements, and perform local deduplication, distributed cache deduplication, and event stream deduplication on the *a* original structured query statements to generate *d* target structured query statements, including: Obtain a original structured query statements, load the a original structured query statements into the local cache, perform local deduplication on the a original structured query statements to generate b first structured query statements, and store the b first structured query statements in the data lake, where b is an integer, b is greater than 0 and less than a; Obtain b first structured query statements from the data lake, load the b first structured query statements into a distributed cache, perform distributed cache deduplication on the b first structured query statements to generate c second structured query statements, and store the c second structured query statements in the data lake, where c is an integer, c is greater than 0 and less than b; The c second structured query statements are obtained from the data lake. The c second structured query statements are deduplicated using event streams to generate the d target structured query statements, where d is an integer, greater than 0 and less than c.

3. The method according to claim 2, characterized in that, Local deduplication is performed on the a original structured query statements to generate b first structured query statements, including: A cache eviction algorithm is used to obtain e repeated third structured query statements within a first preset time period from the a original structured query statements, where e is an integer, greater than 0 and less than a; and The e duplicate third structured query statements in the a original structured query statements within a first preset time period are merged and deduplicated to generate the b first structured query statements.

4. The method according to claim 2, characterized in that, Distributed caching and deduplication are performed on the b first structured query statements to generate c second structured query statements, including: A distributed Bloom filter is used to obtain f fourth-structured query statements whose time-to-live has expired from the b first-structured query statements, where f is an integer, greater than 0 and less than b, and the sum of f and c equals b; and Delete the f fourth structured query statements from the b first structured query statements to generate the c second structured query statements.

5. The method according to claim 2, characterized in that, The c second structured query statements are deduplicated using an event stream to generate d target structured query statements, including: The event flow detection obtains g fifth structured query statements that complete the checkpoint fault recovery from the c second structured query statements, where g is an integer, g is greater than 0 and less than c, and the sum of g and d is equal to c. Obtain the g sixth structured query statements that correspond to the g fifth structured query statements that have failed and the checkpoint fault recovery process; and Delete the g sixth structured query statements from the c second structured query statements to generate the d target structured query statements.

6. The method according to claim 1, characterized in that, Retrieving the d target structured query statements from the data lake includes: The d target structured query statements are obtained from the data lake via a data stream through a remote service interface.

7. The method according to claim 1, characterized in that, Training a large security scoring model includes: Pre-defined expert rules are used to construct a retrieval-enhanced knowledge base based on these expert rules. Obtain h historical structured query statements and their corresponding security scores. Based on the retrieval enhancement generation knowledge base, perform retrieval enhancement generation on the h historical structured query statements and their corresponding security scores to generate i historical structured query statements and their corresponding security scores, where h is an integer greater than 1, and i is an integer greater than h; and The initial large model is trained based on the i historical structured query statements and the security scores of the corresponding i historical structured query statements to generate the security score large model.

8. The method according to any one of claims 1 to 7, characterized in that, The method also includes: Obtain n third structured query statements from the d target structured query statements whose security scores are lower than a preset threshold, and send the n third structured query statements, where n is an integer, greater than 1 and less than d.

9. A safety score generation device, characterized in that, The device includes: The first deduplication module is used to obtain a original structured query statements, perform local deduplication, distributed cache deduplication, and event stream deduplication on the a original structured query statements to generate d target structured query statements, and store the d target structured query statements in the data lake, where a is an integer and a is greater than 1, d is an integer and d is greater than 0 and less than a; The fourth acquisition module is used to acquire the d target structured query statements from the data lake, and based on the d target structured query statements, acquire the d execution plans corresponding to the target structured query statements; and The first generation module is used to input the d target structured query statements and the d execution plans corresponding to the target structured query statements into a pre-trained security scoring model, and output the security scores of the d target structured query statements.

10. An electronic device, comprising: One or more processors; Memory, used to store one or more computer programs. The characteristic feature is that the one or more processors execute the one or more computer programs to implement the steps of the method according to any one of claims 1 to 8.

11. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program or instructions are executed by a processor, they implement the steps of the method according to any one of claims 1 to 8.

12. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 8.

Images