Target system verification method and device, equipment and storage medium

Abstract

The invention provides a target system verification method which can be applied to the technical field of artificial intelligence. The target system verification method comprises the following steps: in response to deployment of a target system to a production environment, obtaining change information corresponding to the target system; according to the change information, utilizing a pre-trained risk identification model to obtain a potential risk existing after the target system is deployed to the production environment; according to the potential risk and the information of the production environment where the target system is located, generating an index vector, and according to the index vector, retrieving knowledge fragments matched with the index vector from a pre-constructed knowledge base; fusing the knowledge fragment and the index vector to obtain a context prompt, and generating a verification script corresponding to the context prompt by using a large language model according to the context prompt; and verifying the target system according to the verification script to obtain a verification result. The invention further provides a target system verification device and equipment and a storage medium.

Description

Technical Field

[0001] This disclosure relates to the field of artificial intelligence technology, and more specifically to a method, apparatus, device, and storage medium for verifying a target system. Background Technology

[0002] With the rapid development of information technology, financial institutions' information systems are becoming increasingly complex, and their deployment, upgrades, and changes are becoming more frequent. To ensure that the system can operate stably, securely, and in a standardized manner after deployment, post-deployment technical verification has become an indispensable and crucial step.

[0003] However, existing post-production system technical verification methods typically only execute pre-defined, static verification processes. When faced with dynamically changing production environments, generating targeted verification solutions quickly, accurately, and comprehensively remains a challenge. Therefore, existing post-production system technical verification methods struggle to perform intelligent analysis and decision-making based on real-time change information and dynamic production environment data, resulting in low verification accuracy and efficiency. Summary of the Invention

[0004] In view of the above problems, this disclosure provides a method, apparatus, equipment and storage medium for verifying target systems to improve the accuracy and efficiency of target system verification.

[0005] According to a first aspect of this disclosure, a target system verification method is provided, comprising: in response to the deployment of the target system to a production environment, obtaining change information corresponding to the target system; based on the change information, using a pre-trained risk identification model to obtain potential risks existing after the target system is deployed to the production environment; generating an index vector based on the potential risks and the production environment information of the target system, and retrieving knowledge fragments matching the index vector from a pre-built knowledge base based on the index vector; fusing the knowledge fragments and the index vector to obtain contextual hints, and generating a verification script corresponding to the contextual hints using a large language model based on the contextual hints; and verifying the target system based on the verification script to obtain a verification result.

[0006] According to embodiments of this disclosure, the risk identification model is configured as a long short-term memory network model; wherein, based on change information, the potential risks existing after the target system is deployed to the production environment are obtained using a pre-trained risk identification model, including: extracting key attribute features from the change information that can reflect potential risks; concatenating the key attribute features in chronological order into a serialized multi-dimensional feature representation; and using the long short-term memory network model based on the multi-dimensional feature representation to identify the nonlinear relationships and temporal dependencies between the multi-dimensional features, thereby obtaining the identification result.

[0007] According to embodiments of this disclosure, generating an index vector based on potential risks and production environment information of the target system includes: determining an embedding model corresponding to the information type based on the information type of the potential risks and production environment information; and using the embedding model to vectorize the potential risks and production environment information to generate an index vector.

[0008] According to embodiments of this disclosure, a knowledge base is pre-built, including: acquiring background information related to technical verification after the target system is put into production, wherein the background information includes standard operating procedures, technical verification control tables, technical verification scripts, operation and maintenance manuals, and emergency plans; segmenting the background information to obtain multiple paragraphs; vectorizing the segmented paragraphs to obtain multiple vectorized paragraphs, and building a knowledge index for each vectorized paragraph to obtain the knowledge base.

[0009] According to embodiments of this disclosure, retrieving knowledge fragments matching the index vector from a pre-built knowledge base based on the index vector includes: calculating the cosine similarity between the index vector and all knowledge fragments in the knowledge base; sorting the retrieved knowledge fragments based on the cosine similarity; and selecting the top k knowledge fragments with the highest cosine similarity based on the sorted knowledge fragments as target knowledge fragments.

[0010] According to embodiments of this disclosure, retrieving knowledge fragments matching the index vector from a pre-built knowledge base based on the index vector includes: calculating the Euclidean distance between the index vector and all knowledge fragments in the knowledge base; sorting the retrieved knowledge fragments according to the Euclidean distance; and selecting the j knowledge fragments with the smallest Euclidean distance from the sorted knowledge fragments as target knowledge fragments.

[0011] According to embodiments of this disclosure, fusing knowledge fragments and index vectors to obtain contextual prompts includes: filling knowledge fragments and index vectors into a pre-constructed prompt word template and performing integration processing; and based on the integrated prompt word template, introducing thought chain constraints for optimization, wherein the thought chain constraints are used to guide the large language model to perform reasoning and argumentation before generating the verification script.

[0012] According to embodiments of this disclosure, the target system is verified according to a verification script to obtain verification results, including: calling a pre-configured automated verification tool according to the verification script to verify the target system and obtain verification data; generating a verification report based on the verification data and feeding it back to the user, wherein the verification report is used to display the results of this verification and the corresponding risk handling situation.

[0013] A second aspect of this disclosure provides a target system verification apparatus, comprising: an acquisition module for acquiring change information corresponding to the target system in response to the deployment of the target system to a production environment; an identification module for obtaining potential risks existing after the target system is deployed to the production environment based on the change information and using a pre-trained risk identification model; a retrieval module for generating an index vector based on the potential risks and the production environment information of the target system, and retrieving knowledge fragments matching the index vector from a pre-built knowledge base based on the index vector; a fusion module for fusing the knowledge fragments and the index vector to obtain contextual hints, and generating a verification script corresponding to the contextual hints using a large language model based on the contextual hints; and a verification module for verifying the target system according to the verification script to obtain a verification result.

[0014] A third aspect of this disclosure provides an electronic device comprising: one or more processors; and a memory for storing one or more computer programs, wherein the one or more processors execute the one or more computer programs to implement the steps of the method described above.

[0015] A fourth aspect of this disclosure also provides a computer-readable storage medium having a computer program or instructions stored thereon, which, when executed by a processor, implement the steps of the above-described method.

[0016] The fifth aspect of this disclosure also provides a computer program product, including a computer program or instructions that, when executed by a processor, implement the steps of the above-described method. Attached Figure Description

[0017] The foregoing contents, as well as other objects, features, and advantages of this disclosure, will become clearer from the following description of embodiments with reference to the accompanying drawings, in which:

[0018] Figure 1 This diagram schematically illustrates an application scenario of the target system verification method according to an embodiment of the present disclosure.

[0019] Figure 2 A flowchart illustrating a target system verification method according to an embodiment of the present disclosure is shown schematically;

[0020] Figure 3 The schematic diagram illustrates the principle of identifying potential risks based on a long short-term memory network model according to an embodiment of the present disclosure;

[0021] Figure 4 The diagram illustrates the principle of obtaining contextual hints by fusing knowledge fragments and index vectors according to an embodiment of the present disclosure;

[0022] Figure 5A schematic block diagram of a target system verification apparatus according to an embodiment of the present disclosure is shown; and

[0023] Figure 6 A block diagram schematically illustrates an electronic device suitable for implementing a target system verification method according to an embodiment of the present disclosure. Detailed Implementation

[0024] The embodiments of the present disclosure will now be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the disclosure. In the following detailed description, numerous specific details are set forth to provide a thorough understanding of the embodiments of the present disclosure for ease of explanation. However, it will be apparent that one or more embodiments may be practiced without these specific details. Furthermore, descriptions of well-known structures and techniques are omitted in the following description to avoid unnecessarily obscuring the concepts of the present disclosure.

[0025] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit this disclosure. The terms “comprising,” “including,” etc., as used herein indicate the presence of the stated features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0026] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein are to be interpreted in a manner consistent with the context of this specification, and not in an idealized or overly rigid way.

[0027] When using expressions such as "at least one of A, B and C", they should generally be interpreted in accordance with the meaning that is commonly understood by those skilled in the art (e.g., "a system having at least one of A, B and C" should include, but is not limited to, a system having A alone, a system having B alone, a system having C alone, a system having A and B, a system having A and C, a system having B and C, and / or a system having A, B and C, etc.).

[0028] It should be noted that the target system verification method and apparatus disclosed herein can be used in the financial field for target system verification, and can also be used in any field other than the financial field for target system verification. The application field of the target system verification method and apparatus disclosed herein is not limited.

[0029] It should be noted that in the embodiments disclosed herein, certain software, components, models, and other existing solutions in the industry may be mentioned. These should be considered as exemplary and are intended only to illustrate the feasibility of implementing the technical solutions disclosed herein. However, they do not mean that the applicant has used or necessarily used such solutions.

[0030] In the technical solution disclosed herein, the user information (including but not limited to user personal information, user image information, user device information, such as location information) and data (including but not limited to data used for analysis, stored data, and displayed data) involved are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, storage, use, processing, transmission, provision, disclosure, and application of related data all comply with relevant laws, regulations, and standards, and necessary measures have been taken to ensure that they do not violate public order and good morals. Corresponding operation entry points are provided for users to choose to authorize or refuse.

[0031] In scenarios involving automated decision-making using personal information, the methods, devices, and systems provided in this disclosure all offer users corresponding entry points for choosing to agree to or reject the automated decision-making results. If the user chooses to reject, the process proceeds to the expert decision-making stage. Here, "automated decision-making" refers to the activity of automatically analyzing and evaluating an individual's behavioral habits, interests, or economic, health, and credit status through computer programs, and then making a decision. Here, "expert decision-making" refers to the activity of making decisions by personnel who specialize in a particular field, possess specialized experience, knowledge, and skills, and have reached a certain level of professional expertise.

[0032] Embodiments of this disclosure provide a target system verification method, comprising: in response to the deployment of a target system to a production environment, obtaining change information corresponding to the target system; based on the change information, using a pre-trained risk identification model to obtain potential risks existing after the target system is deployed to the production environment; generating an index vector based on the potential risks and the production environment information of the target system, and retrieving knowledge fragments matching the index vector from a pre-built knowledge base based on the index vector; fusing the knowledge fragments and the index vector to obtain contextual hints, and generating a verification script corresponding to the contextual hints using a large language model based on the contextual hints; and verifying the target system based on the verification script to obtain a verification result.

[0033] Figure 1 The diagram illustrates an application scenario of the target system verification method according to an embodiment of the present disclosure.

[0034] like Figure 1 As shown, application scenario 100 according to this embodiment may include a first terminal device 101, a second terminal device 102, a third terminal device 103, a network 104, and a server 105. The network 104 serves as a medium for providing a communication link between the first terminal device 101, the second terminal device 102, the third terminal device 103, and the server 105. The network 104 may include various connection types, such as wired or wireless communication links, or fiber optic cables, etc.

[0035] Users can use the first terminal device 101, the second terminal device 102, and the third terminal device 103 to interact with the server 105 via the network 104 to receive or send messages, etc. Various communication client applications can be installed on the first terminal device 101, the second terminal device 102, and the third terminal device 103, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social media platform software, etc. (for example only).

[0036] The first terminal device 101, the second terminal device 102, and the third terminal device 103 can be various electronic devices with displays and support web browsing, including but not limited to smartphones, tablets, laptops, and desktop computers.

[0037] Server 105 can be a server that provides various services, such as a backend management server that supports websites browsed by users using the first terminal device 101, the second terminal device 102, and the third terminal device 103 (this is just an example). The backend management server can analyze and process data such as received user requests, and feed back the processing results (such as web pages, information, or data obtained or generated according to user requests) to the terminal devices.

[0038] It should be noted that the target system verification method provided in this embodiment can generally be executed by server 105. Correspondingly, the target system verification device provided in this embodiment can generally be located in server 105. The target system verification method provided in this embodiment can also be executed by a server or server cluster that is different from server 105 and capable of communicating with the first terminal device 101, the second terminal device 102, the third terminal device 103, and / or server 105. Correspondingly, the target system verification device provided in this embodiment can also be located in a server or server cluster that is different from server 105 and capable of communicating with the first terminal device 101, the second terminal device 102, the third terminal device 103, and / or server 105.

[0039] It should be understood that Figure 1 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.

[0040] The following will be based on Figure 1 The described scene, through Figures 2-4 The target system verification method of the present disclosure embodiments will be described in detail.

[0041] Figure 2 A flowchart illustrating a target system verification method according to an embodiment of the present disclosure is shown schematically.

[0042] like Figure 2 As shown, the target system verification method of this embodiment includes operations S210 to S250, and the target system verification method can be executed by a server.

[0043] In operation S210, in response to the deployment of the target system to the production environment, change information corresponding to the target system is obtained.

[0044] In operation S220, based on change information, a pre-trained risk identification model is used to identify potential risks that may exist after the target system is deployed to the production environment.

[0045] In operation S230, an index vector is generated based on potential risks and the production environment information of the target system, and knowledge fragments matching the index vector are retrieved from a pre-built knowledge base based on the index vector.

[0046] In operation S240, knowledge fragments and index vectors are fused to obtain contextual hints, and based on the contextual hints, a large language model is used to generate a verification script corresponding to the contextual hints.

[0047] When operating the S250, the target system is verified according to the verification script, and the verification results are obtained.

[0048] Post-production technical verification refers to a series of verification and validation activities carried out after an information system, software, or application has completed research and development, testing, and has been formally deployed to the production environment (i.e., "production"), in order to ensure that the system can operate stably, securely, and in a standardized manner under real business operation conditions.

[0049] In this embodiment of the disclosure, it is assumed that a financial institution needs to upgrade (change information) its core transaction system.

[0050] Once the new version is deployed to the production environment, the system will first automatically obtain the change information for this upgrade, including code commit records, configuration change list, and release notes.

[0051] Next, a pre-trained risk identification model (e.g., a machine learning model trained based on historical failure data and change types) is used to analyze this change information and identify potential risks that may exist in this upgrade, such as potential risks like "transaction response delay under high concurrency" and "database deadlock".

[0052] Simultaneously, real-time information about the production environment of the trading system is acquired, such as server core processor (CPU) utilization, memory usage, database connection pool status, and network latency. This production environment information and identified potential risk descriptions are then input into the embedding model to generate a comprehensive index vector.

[0053] Then, using this index vector, knowledge fragments related to "high-concurrency transaction system" and "database deadlock handling" are retrieved from a pre-built knowledge base (e.g., containing historical standard operating procedures (SOPs), verification scripts, failure cases, etc.).

[0054] These knowledge fragments, along with the index vectors, are fused together to form a contextual cue, which is then fed into the Large Language Model (LLM). Based on this cue, the LLM generates a detailed automated verification script, which includes specific verification points such as stress testing the trading system, monitoring database deadlock logs, and checking transaction response times. Finally, the verification script is executed to verify the upgraded trading system, and the verification results are recorded.

[0055] According to embodiments of this disclosure, a novel approach combines a knowledge base, a large language model, and prompt word engineering. Through a Retrieval-Augmented Generation (RAG) technique, the professional knowledge required for system verification is vectorized and indexed, and then combined with the user's actual needs (potential system risks and current production environment information). This enables automatic matching and generation of post-deployment technical verification scripts (verification schemes and verification points) for specific systems, thereby achieving system verification. Because this method can identify potential risks based on specific change information and real-time environmental information, and automatically retrieve relevant professional knowledge from the knowledge base, ensuring that the generated verification scripts cover key risk points, it greatly improves the accuracy and efficiency of system verification.

[0056] In this embodiment of the disclosure, the risk identification model is configured as a Long Short-Term Memory (LSTM) network model.

[0057] Specifically, operation S220 uses a pre-trained risk identification model based on the change information to determine the potential risks that the target system may pose after deployment to the production environment. These risks may include, for example:

[0058] Extract key attribute features from the change information that can reflect potential risks; concatenate the key attribute features in chronological order into a serialized multi-dimensional feature representation; based on the multi-dimensional feature representation, use a long short-term memory network model to identify the nonlinear relationships and temporal dependencies between the multi-dimensional features to obtain the identification results.

[0059] In this embodiment of the disclosure, it is further clarified that the risk identification model is trained using LSTM, and the specific process of feature extraction, serialization and identification of temporal dependencies of change information is described in detail.

[0060] Figure 3 The schematic diagram illustrates the principle of identifying potential risks based on a long short-term memory network model according to an embodiment of the present disclosure.

[0061] like Figure 3 As shown, the principle of identifying potential risks based on the Long Short-Term Memory (LSTM) network model in this embodiment is as follows: After obtaining version upgrade and change information of the core trading system, the system first preprocesses this raw information. For example, it extracts the change type (such as "adding an Application Programming Interface (API)" or "optimizing database queries"), the file paths involved, and the addition or deletion of lines of code from the code commit records; and it extracts key function descriptions from the release notes.

[0062] Next, the system vectorizes and encodes these extracted key attribute features (such as change type encoding, number of lines of code, and involved module ID).

[0063] Then, these features are concatenated in chronological order (e.g., multiple code commits involved in this change, or historical change records of related modules in recent times) into a serialized multi-dimensional feature representation. This serialized feature representation is then fed into a pre-trained LSTM model. Due to its unique gating mechanism, the LSTM model can identify the complex nonlinear relationships and temporal dependencies between these multi-dimensional features (e.g., discovering that if a high-concurrency feature is launched immediately after a certain type of database change, the risk of "database deadlock" will increase significantly). Finally, the LSTM model outputs the potential risks of this upgrade, such as the probability and type of "transaction response latency under high concurrency" and "database deadlock".

[0064] According to embodiments of this disclosure, LSTM models can learn complex patterns and time-series information in historical change data, thereby more accurately identifying potential risks. They are particularly suitable for complex system change scenarios. Compared to simple rules or traditional machine learning models, LSTM has better generalization ability when dealing with novel combinations of changes that have not been seen before.

[0065] In this embodiment of the disclosure, operation S230 generates an index vector based on potential risks and the production environment information of the target system, which may include, for example:

[0066] Based on the information types of potential risks and production environment information, an embedding model corresponding to the information type is determined; using the embedding model, the potential risks and production environment information are vectorized to generate index vectors.

[0067] In this embodiment of the disclosure, it is further clarified that during the vectorization process, an appropriate embedding model is selected according to the information type (textualized potential risks and environmental information), and different types of information are vectorized to finally generate a unified index vector.

[0068] For example, after identifying textual descriptions of potential risks such as "transaction response delay under high concurrency" and "database deadlock" and obtaining information about the production environment in which the transaction system operates, the system needs to convert this information into an index vector.

[0069] First, the system determines the embedding model based on the type of information.

[0070] For textual descriptions (such as potential risk descriptions and textualized environmental information), pre-trained text embedding models are typically chosen. Examples include general embedding models and domain-specific embedding models. General embedding models can map sentences or paragraphs to a high-dimensional vector space, and semantically similar texts are closer together in the vector space. If there is a large amount of textual data in IT operations or security, it is advisable to fine-tune the general model and use a domain-specific embedding model to improve performance in that specific domain.

[0071] For production environment information that includes non-text data such as images and log files, more complex multimodal embedding models are needed to ensure that specific tools can be used to transform the information into vectors.

[0072] Then, the vectors of potential risks and the vectors of production environment information are concatenated (or weighted and fused) to generate a comprehensive index vector.

[0073] According to embodiments of this disclosure, the embedding model can uniformly transform textual risk descriptions and textualized structured environmental information into high-dimensional numerical vectors, facilitating similarity calculation and processing by computers and providing high-quality input for subsequent knowledge base retrieval.

[0074] In this embodiment of the disclosure, a knowledge base may be pre-built, which may include, for example:

[0075] Obtain background information related to the technical verification after the target system is put into production. The background information includes standard operating procedures, technical verification control tables, technical verification scripts, operation and maintenance manuals, and emergency plans. The background information is segmented into multiple paragraphs. The segmented paragraphs are vectorized to obtain vectorized paragraphs. For each vectorized paragraph, a knowledge index is built to obtain a knowledge base.

[0076] In this embodiment of the disclosure, the construction process of the knowledge base is further clarified, highlighting the structure and searchability of the knowledge base.

[0077] For example, before system deployment, a large amount of background information related to the technical verification of the core trading system after production is collected. This information may include:

[0078] Standard Operating Procedures: Detailed documentation on the deployment, monitoring, and troubleshooting of the trading system.

[0079] Technical Verification Control List: Includes checklists and verification points for various system changes.

[0080] Technical verification scripts: Examples of scripts used in the past for stress testing, functional verification, and data consistency verification of trading systems.

[0081] Operations and Maintenance Manual: A document describing the architecture, dependencies, and configuration items of the transaction system.

[0082] Emergency plan: Recovery steps for common failures in the trading system.

[0083] The system automates the processing of these raw documents. First, each document is segmented into appropriately granular paragraphs; for example, an SOP document is segmented into multiple step descriptions, and a script example is segmented into script code blocks and comments. Next, each segmented paragraph is vectorized using the same embedding model used to generate the index vectors, resulting in corresponding paragraph vectors. Finally, the text content of these paragraphs and their corresponding vectors are stored in a vector database, and an efficient index structure is built for these vectors, thus forming a knowledge base that allows for rapid retrieval.

[0084] According to embodiments of this disclosure, the knowledge base transforms dispersed, unstructured professional knowledge into a vector form that can be understood and retrieved by machines, solving the problem of knowledge being difficult to share and reuse. Through vectorization and index building, it enables the ability to quickly and accurately retrieve relevant information from massive amounts of knowledge.

[0085] In this embodiment of the disclosure, operation S230 retrieves knowledge fragments matching the index vector from a pre-built knowledge base, which may include, for example:

[0086] Calculate the cosine similarity between the index vector and all knowledge fragments in the knowledge base; sort the retrieved knowledge fragments according to the cosine similarity; based on the sorted knowledge fragments, select the top k knowledge fragments with the highest cosine similarity as the target knowledge fragments.

[0087] In this embodiment of the disclosure, cosine similarity is used as a vector similarity metric, and combined with ranking and Top-k filtering mechanisms to achieve efficient and accurate knowledge fragment retrieval.

[0088] For example, after generating an index vector that integrates potential risks and production environment information, the system needs to retrieve matching knowledge fragments from a pre-built knowledge base.

[0089] First, the system calculates the cosine similarity between the index vector and all knowledge fragment vectors in the knowledge base. For example, if the index vector represents the query "transaction response latency under high concurrency, database version 8.0", the system will search the vector database for the knowledge fragment vector whose direction is closest to this query vector. Next, the system sorts all knowledge fragments from highest to lowest based on the calculated cosine similarity score. Finally, the system selects the top k (e.g., k=5) knowledge fragments with the highest cosine similarity from the sorted results as the target knowledge fragments for this query. These fragments include content related to "database version 8.0 performance optimization", "high-concurrency system tuning SOP", and "historical transaction system latency failure analysis".

[0090] According to embodiments of this disclosure, cosine similarity can effectively measure the semantic similarity of texts, ensuring that the retrieved knowledge fragments are highly relevant to the query in meaning. Combined with the index structure of the vector database, it can quickly locate the most relevant knowledge fragments in a large-scale knowledge base, filter out the top-k most relevant knowledge fragments, provide refined and highly relevant background information for the large language model, and avoid interference from irrelevant information.

[0091] In this embodiment of the disclosure, operation S230 retrieves knowledge fragments matching the index vector from a pre-built knowledge base, and may further include, for example:

[0092] Calculate the Euclidean distance between the index vector and all knowledge fragments in the knowledge base; sort the retrieved knowledge fragments according to the Euclidean distance; based on the sorted knowledge fragments, select the j knowledge fragments with the smallest Euclidean distance as the target knowledge fragments.

[0093] In this embodiment, another vector similarity metric—Euclidean distance—is provided, and combined with ranking and Top-j filtering mechanisms to achieve knowledge fragment retrieval.

[0094] In addition to using cosine similarity to retrieve knowledge fragments, Euclidean distance can also be used for retrieval.

[0095] For example, after generating an index vector that integrates potential risks and production environment information, the system needs to retrieve matching knowledge fragments from a pre-built knowledge base.

[0096] First, the Euclidean distance between the index vector and all knowledge fragment vectors in the knowledge base is calculated. Euclidean distance measures the straight-line distance between two points in a vector space. For example, if the index vector represents the query "transaction response latency under high concurrency, database version 8.0", the system will search the vector database for the knowledge fragment vector closest to this query vector. Next, the system sorts all knowledge fragments from lowest to highest distance based on the calculated Euclidean distance (smaller distances indicate greater similarity). Finally, the system selects the j knowledge fragments with the smallest Euclidean distance (e.g., j=5) from the sorted results as the target knowledge fragments for this query. These fragments include content related to "database version 8.0 performance optimization", "high-concurrency system tuning SOP", and "historical transaction system latency fault analysis".

[0097] According to embodiments of this disclosure, Euclidean distance geometrically reflects the proximity of vectors, providing an optional similarity calculation method for different application scenarios or model preferences, and increasing the flexibility of the method.

[0098] In this embodiment of the disclosure, operation S240 fuses knowledge fragments and index vectors to obtain contextual hints, which may include, for example:

[0099] Knowledge fragments and index vectors are populated into a pre-built prompt word template and integrated. Based on the integrated prompt word template, a thought chain constraint is introduced for optimization. The thought chain constraint is used to guide the large language model to perform reasoning and argumentation before generating the verification script.

[0100] In this embodiment, a Chain-of-Thought (CoT) constraint is introduced to guide the large language model to perform reasoning and argumentation before generating the verification script, thereby improving the reasoning ability and output quality of the large language model.

[0101] Figure 4 The diagram illustrates the principle of obtaining contextual hints by fusing knowledge fragments and index vectors according to an embodiment of the present disclosure.

[0102] like Figure 4As shown, the principle of obtaining contextual hints by fusing knowledge fragments and index vectors in this embodiment is as follows: after obtaining potential risks (such as "transaction response delay under high concurrency" and "database deadlock"), production environment information, and retrieved target knowledge fragments (such as "database version 8.0 connection pool optimization SOP" and "high concurrency stress test script example"), the system needs to integrate this information into contextual hints.

[0103] First, this information is populated into a pre-built prompt template to form an initial prompt. Based on this, the system further introduces thought chain constraints for optimization, guiding the large language model to perform reasoning and argumentation before generating the verification script. Upon receiving such a prompt, the large language model first outputs an analytical reasoning process (e.g., analyzing how high concurrency might lead to connection pool exhaustion and deadlock, and considering how to simulate high concurrency in the current environment), and then generates the specific verification script.

[0104] According to embodiments of this disclosure, the thought chain constraint can force a large language model to perform reasoning and argumentation, making the generated scripts more logical, more accurately covering risk points, and reducing "illusions" and unreasonable verification steps.

[0105] In this embodiment of the disclosure, operation S250 verifies the target system according to the verification script and obtains the verification result, which may include, for example:

[0106] According to the verification script, the pre-configured automated verification tool is invoked to verify the target system and obtain verification data. Based on the verification data, a verification report is generated and fed back to the user. The verification report is used to display the results of this verification and the corresponding risk handling.

[0107] In this embodiment of the disclosure, the execution method of the verification script (calling automation tools) and the result feedback mechanism (generating and feeding back the verification report) are further clarified, forming a complete closed loop from intelligent generation to automated execution and then to result feedback.

[0108] For example, once the large language model generates an automated verification script for the core trading system, the system will execute the script immediately.

[0109] The system invokes pre-configured automated verification tools (such as stress testing tools, monitoring system interfaces, database clients, etc.) to verify the target trading system according to the instructions in the script. During the verification process, the system collects verification data, such as stress test response time, transactions per second (TPS), error rate, real-time utilization of the database connection pool, and frequency of deadlock log occurrences.

[0110] Once verification is complete, the system automatically generates a detailed verification report based on the verification data. This report clearly shows the results of the verification (e.g., "stress test passed, response time meets expectations," "no database deadlocks found"), as well as the handling of identified risks (e.g., if response delays are found, it will suggest checking the database connection pool configuration). This report is then provided to operations personnel or the development team.

[0111] According to embodiments of this disclosure, the intelligently generated scripts are seamlessly integrated with existing automation tools, achieving full automation from verification scheme generation to execution. Detailed verification reports provide objective data support for operations and development teams, helping them quickly determine whether the system can be safely launched and what areas need further optimization.

[0112] Based on the above-described target system verification method, this disclosure also provides a target system verification apparatus. The following will be combined with... Figure 5 The device is described in detail.

[0113] Figure 5 A schematic block diagram of a target system verification apparatus according to an embodiment of the present disclosure is shown.

[0114] like Figure 5 As shown, the target system verification device 500 of this embodiment includes an acquisition module 510, an identification module 520, a retrieval module 530, a fusion module 540, and a verification module 550.

[0115] The acquisition module 510 is used to acquire change information corresponding to the target system in response to the deployment of the target system to the production environment. In one embodiment, the acquisition module 510 can be used to perform the operation S210 described above, which will not be repeated here.

[0116] The identification module 520 is used to identify potential risks to the target system after it is deployed to the production environment based on the change information and a pre-trained risk identification model. In one embodiment, the scanning module 520 can be used to perform the operation S220 described above, which will not be repeated here.

[0117] The retrieval module 530 generates an index vector based on potential risks and the production environment information of the target system, and retrieves knowledge fragments matching the index vector from a pre-built knowledge base. In one embodiment, the filtering module 530 can be used to perform the operation S230 described above, which will not be repeated here.

[0118] The fusion module 540 is used to fuse knowledge fragments and index vectors to obtain contextual hints, and based on the contextual hints, to generate a verification script corresponding to the contextual hints using a large language model. In one embodiment, the generation module 540 can be used to perform the operation S240 described above, which will not be repeated here.

[0119] The verification module 550 is used to verify the target system according to the verification script and obtain the verification result. In one embodiment, the generation module 550 can be used to perform the operation S250 described above, which will not be repeated here.

[0120] According to embodiments of this disclosure, any multiple modules among the acquisition module 510, identification module 520, retrieval module 530, fusion module 540, and verification module 550 can be combined into one module, or any one of these modules can be split into multiple modules. Alternatively, at least some of the functions of one or more of these modules can be combined with at least some of the functions of other modules and implemented in one module. According to embodiments of this disclosure, at least one of the acquisition module 510, identification module 520, retrieval module 530, fusion module 540, and verification module 550 can be at least partially implemented as hardware circuitry, such as a field-programmable gate array (FPGA), a programmable logic array (PLA), a system-on-a-chip, a system-on-a-substrate, a system-on-package, an application-specific integrated circuit (ASIC), or implemented in hardware or firmware by any other reasonable means of integrating or packaging the circuitry, or implemented in software, hardware, or firmware, or in any suitable combination of any of these three implementation methods. Alternatively, at least one of the acquisition module 510, identification module 520, retrieval module 530, fusion module 540 and verification module 550 may be implemented at least partially as a computer program module, which can perform corresponding functions when the computer program module is run.

[0121] Figure 6 A block diagram schematically illustrates an electronic device suitable for implementing a target system verification method according to an embodiment of the present disclosure.

[0122] like Figure 6As shown, an electronic device 600 according to an embodiment of this disclosure includes a processor 601, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage portion 608 into a random access memory (RAM) 603. The processor 601 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or an associated chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 601 may also include onboard memory for caching purposes. The processor 601 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of this disclosure.

[0123] RAM 603 stores various programs and data required for the operation of electronic device 600. Processor 601, ROM 602, and RAM 603 are interconnected via bus 604. Processor 601 performs various operations of the method flow according to embodiments of the present disclosure by executing programs in ROM 602 and / or RAM 603. It should be noted that the programs may also be stored in one or more memories other than ROM 602 and RAM 603. Processor 601 may also perform various operations of the method flow according to embodiments of the present disclosure by executing programs stored in said one or more memories.

[0124] According to embodiments of this disclosure, the electronic device 600 may further include an input / output (I / O) interface 605, which is also connected to a bus 604. The electronic device 600 may also include one or more of the following components connected to the input / output (I / O) interface 605: an input section 606 including a keyboard, mouse, etc.; an output section 607 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 608 including a hard disk, etc.; and a communication section 609 including a network interface card such as a LAN card, modem, etc. The communication section 609 performs communication processing via a network such as the Internet. A drive 610 is also connected to the input / output (I / O) interface 605 as needed. A removable medium 611, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on the drive 610 as needed so that computer programs read from it can be installed into the storage section 608 as needed.

[0125] This disclosure also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or it may exist independently and not assembled into the device / apparatus / system. The computer-readable storage medium carries one or more programs that, when executed, implement the method according to the embodiments of this disclosure.

[0126] According to embodiments of this disclosure, the computer-readable storage medium may be a non-volatile computer-readable storage medium, such as including, but not limited to: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this disclosure, the computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. For example, according to embodiments of this disclosure, the computer-readable storage medium may include ROM 602 and / or RAM 603 and / or one or more memories other than ROM 602 and RAM 603 described above.

[0127] Embodiments of this disclosure also include a computer program product comprising a computer program containing program code for performing the methods shown in the flowchart. When the computer program product is run on a computer system, the program code is used to cause the computer system to implement the methods provided in the embodiments of this disclosure.

[0128] When the computer program is executed by the processor 601, it performs the functions defined in the system / apparatus of this disclosure embodiments. According to embodiments of this disclosure, the systems, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0129] In one embodiment, the computer program may rely on a tangible storage medium such as an optical storage device or a magnetic storage device. In another embodiment, the computer program may also be transmitted and distributed in the form of signals over a network medium, and downloaded and installed via the communication section 609, and / or installed from the removable medium 611. The program code contained in the computer program can be transmitted using any suitable network medium, including but not limited to: wireless, wired, etc., or any suitable combination thereof.

[0130] In such an embodiment, the computer program can be downloaded and installed from a network via the communication section 609, and / or installed from the removable medium 611. When the computer program is executed by the processor 601, it performs the functions defined in the system of this disclosure embodiment. According to embodiments of this disclosure, the systems, devices, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0131] According to embodiments of this disclosure, program code for executing the computer programs provided in embodiments of this disclosure can be written in any combination of one or more programming languages. Specifically, these computational programs can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, languages ​​such as Java, C++, Python, "C", or similar programming languages. The program code can execute entirely on a user's computing device, partially on a user's device, partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0132] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0133] Those skilled in the art will understand that the features described in the various embodiments of this disclosure can be combined and / or combined in various ways, even if such combinations or combinations are not explicitly described in this disclosure. In particular, the features described in the various embodiments of this disclosure can be combined and / or combined in various ways without departing from the spirit and teachings of this disclosure. All such combinations and / or combinations fall within the scope of this disclosure.

[0134] The embodiments of this disclosure have been described above. However, these embodiments are for illustrative purposes only and are not intended to limit the scope of this disclosure. Although various embodiments have been described above, this does not mean that the measures in the various embodiments cannot be used advantageously in combination. Various substitutions and modifications can be made by those skilled in the art without departing from the scope of this disclosure, and all such substitutions and modifications should fall within the scope of this disclosure.

Claims

1. A method for verifying a target system, characterized in that, The method includes: In response to the deployment of the target system to the production environment, obtain the change information corresponding to the target system; Based on the change information, a pre-trained risk identification model is used to obtain the potential risks that the target system may pose after being deployed to the production environment; Based on the potential risks and the production environment information of the target system, an index vector is generated, and based on the index vector, knowledge fragments matching the index vector are retrieved from a pre-built knowledge base. By fusing the knowledge fragments with the index vectors, contextual hints are obtained, and based on the contextual hints, a large language model is used to generate a verification script corresponding to the contextual hints. The target system is verified according to the verification script to obtain the verification result.

2. The method according to claim 1, characterized in that, The risk identification model is configured as a long short-term memory network model; The step of obtaining the potential risks of the target system after deployment to the production environment based on the change information and using a pre-trained risk identification model includes: Extract the key attribute features from the change information that can reflect potential risks; The key attribute features are concatenated in chronological order to form a serialized multi-dimensional feature representation; Based on the multi-dimensional feature representation, the nonlinear relationships and temporal dependencies between the multi-dimensional features are identified using the long short-term memory network model, and the identification results are obtained.

3. The method according to claim 2, characterized in that, The step of generating an index vector based on the potential risks and the production environment information of the target system includes: Based on the potential risks and the information types of the production environment information, determine the embedding model corresponding to the information type; Using the embedding model, the potential risks and the production environment information are vectorized to generate an index vector.

4. The method according to claim 1, characterized in that, The knowledge base is pre-built, including: Obtain background information related to the technical verification after the target system is put into production, wherein the background information includes standard operating procedures, technical verification control tables, technical verification scripts, operation and maintenance manuals, and emergency plans; The background information is segmented to obtain multiple paragraphs; The segmented paragraphs are vectorized to obtain vectorized paragraphs. For each vectorized paragraph, a knowledge index is constructed to obtain the knowledge base.

5. The method according to claim 4, characterized in that, The step of retrieving knowledge fragments matching the index vector from a pre-built knowledge base includes: Calculate the cosine similarity between the index vector and all knowledge fragments in the knowledge base; The retrieved knowledge fragments are sorted according to the cosine similarity. Based on the sorted knowledge fragments, the top k knowledge fragments with the highest cosine similarity are selected as the target knowledge fragments.

6. The method according to claim 4, characterized in that, The step of retrieving knowledge fragments matching the index vector from a pre-built knowledge base includes: Calculate the Euclidean distance between the index vector and all knowledge fragments in the knowledge base; The retrieved knowledge fragments are sorted according to the Euclidean distance. Based on the sorted knowledge fragments, the j knowledge fragments with the smallest Euclidean distance are selected as the target knowledge fragments.

7. The method according to claim 1, characterized in that, The process of fusing the knowledge fragment with the index vector to obtain contextual hints includes: The knowledge fragments and index vectors are populated into a pre-built prompt word template and then integrated. Based on the integrated prompt word template, a thought chain constraint is introduced for optimization. The thought chain constraint is used to guide the large language model to perform reasoning and argumentation before generating the verification script.

8. The method according to claim 7, characterized in that, The step of verifying the target system according to the verification script and obtaining the verification result includes: The process involves calling a pre-configured automated verification tool based on the verification script to verify the target system and obtain verification data. Based on the verification data, a verification report is generated and sent back to the user. The verification report is used to display the results of this verification and the corresponding risk handling situation.

9. A target system verification device, characterized in that, The device includes: The acquisition module is used to acquire change information corresponding to the target system in response to the deployment of the target system to the production environment; The identification module is used to determine the potential risks of the target system after it is deployed to the production environment based on the change information and a pre-trained risk identification model. The retrieval module is used to generate an index vector based on the potential risks and the production environment information of the target system, and to retrieve knowledge fragments that match the index vector from a pre-built knowledge base based on the index vector. The fusion module is used to fuse the knowledge fragments and the index vectors to obtain contextual hints, and to generate a verification script corresponding to the contextual hints using a large language model based on the contextual hints. The verification module is used to verify the target system according to the verification script and obtain the verification result.

10. An electronic device, comprising: One or more processors; Memory, used to store one or more computer programs. The characteristic feature is that the one or more processors execute the one or more computer programs to implement the steps of the method according to any one of claims 1 to 8.

11. A computer-readable storage medium having a computer program or instructions stored thereon, characterized in that, When the computer program or instructions are executed by a processor, they implement the steps of the method according to any one of claims 1 to 8.

12. A computer program product, comprising a computer program or instructions, characterized in that, When the computer program or instructions are executed by a processor, they implement the steps of the method according to any one of claims 1 to 8.

Images