Data analysis scenario generation method, device, equipment and storage medium

Abstract

The present invention discloses a method, device, equipment and storage medium for generating a data analysis scenario. The method comprises: receiving a scenario requirement file to be processed, performing semantic analysis and data extraction processing on the scenario requirement file using a pre-trained language model, and obtaining data analysis scenario information corresponding to the scenario requirement file; based on the data analysis scenario information, obtaining a set of candidate scenario information that matches the scenario requirement file from a data analysis scenario pool; and assembling data query statements and front-end component codes for the data analysis scenario corresponding to the scenario requirement file based on the candidate scenario information in the candidate scenario information set. The technical solution of the present invention achieves the beneficial effect of automatically creating an adaptive data analysis scenario and a front-end code framework based on the scenario requirement file.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a method, apparatus, device, and storage medium for generating a data analysis scenario. Background Art

[0002] In transaction management systems, business data analysis scenarios for each business line are designed and implemented mainly through manual analysis of scenarios and manual search for data sources.

[0003] However, in the existing business data analysis requirements analysis process, business personnel lack knowledge of data analysis, limiting the scope of business data analysis scenarios they can propose. This has limited the development of data analysis capabilities. Because the limited scenarios proposed result in limited data coverage, a large amount of data and data tables remain unused. Summary of the Invention

[0004] The present invention provides a method, apparatus, device and storage medium for generating a data analysis scenario to solve the problem of limited business data analysis scenarios that can be proposed when manually analyzing business data.

[0005] According to one aspect of the present invention, a method for generating a data analysis scenario is provided, comprising:

[0006] Receive a scenario requirement file to be processed, perform semantic analysis and data extraction on the scenario requirement file using a pre-trained language model, and obtain data analysis scenario information corresponding to the scenario requirement file;

[0007] Based on the data analysis scenario information, obtaining a candidate scenario information set matching the scenario requirement file from a data analysis scenario pool;

[0008] According to the candidate scenario information in the candidate scenario information set, a data query statement and a front-end component code of the data analysis scenario corresponding to the scenario requirement file are assembled.

[0009] According to another aspect of the present invention, there is provided a device for generating a data analysis scenario, comprising:

[0010] A receiving module is used to receive a scenario requirement file to be processed, perform semantic analysis and data extraction on the scenario requirement file using a pre-trained language model, and obtain data analysis scenario information corresponding to the scenario requirement file;

[0011] An acquisition module, configured to acquire, based on the data analysis scenario information, a set of candidate scenario information that matches the scenario requirement file from a data analysis scenario pool;

[0012] An assembly module is used to execute data query statements and front-end component codes for assembling the data analysis scenario corresponding to the scenario requirement file based on the candidate scenario information in the candidate scenario information set.

[0013] According to another aspect of the present invention, an electronic device is provided, comprising:

[0014] at least one processor; and

[0015] a memory communicatively connected to the at least one processor; wherein,

[0016] The memory stores a computer program that can be executed by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the method for generating a data analysis scenario described in any embodiment of the present invention.

[0017] According to another aspect of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the method for generating a data analysis scenario according to any embodiment of the present invention when executed.

[0018] The technical solution of the embodiment of the present invention receives a scenario requirement file to be processed, uses a pre-trained language model to perform semantic analysis and data extraction processing on the scenario requirement file, and obtains data analysis scenario information corresponding to the scenario requirement file; based on the data analysis scenario information, obtains a set of candidate scenario information matching the scenario requirement file from a data analysis scenario pool; according to the candidate scenario information in the candidate scenario information set, assembles data query statements and front-end component codes for the data analysis scenario corresponding to the scenario requirement file, thereby solving the problem of limited business data analysis scenarios that can be proposed when manually analyzing business data, and achieving the beneficial effect of automatically creating adaptive data analysis scenarios and front-end code frameworks according to scenario requirement files.

[0019] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present invention, nor is it intended to limit the scope of the present invention. Other features of the present invention will become readily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0021] Figure 1 is a flowchart of a method for generating a data analysis scenario according to the first embodiment of the present invention;

[0022] Figure 2 is a flowchart of another method for generating a data analysis scenario according to the second embodiment of the present invention;

[0023] Figure 3 1 is a schematic structural diagram of a device for generating a data analysis scenario according to a third embodiment of the present invention;

[0024] Figure 4 It is a structural diagram of an electronic device for implementing the method for generating a data analysis scenario according to an embodiment of the present invention. DETAILED DESCRIPTION

[0025] In order to enable those skilled in the art to better understand the solutions of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the embodiments described are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of the present invention.

[0026] It should be noted that the terms "first", "target", "candidate", etc. in the description and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0027] Example 1

[0028] Figure 1 This is a flow chart of a method for generating a data analysis scenario according to the first embodiment of the present invention. This embodiment is applicable to newly added scenario requirement files, and can automatically generate data analysis scenarios and front-end code frameworks that are adapted thereto. This method can be executed by a data analysis scenario generation device, which can be implemented in the form of hardware and / or software, and can be configured in an electronic device. Figure 1 As shown, the method includes:

[0029] S110: Receive a scenario requirement file to be processed, perform semantic analysis and data extraction on the scenario requirement file using a pre-trained language model, and obtain data analysis scenario information corresponding to the scenario requirement file.

[0030] Scenario requirement files can include newly added data tables, newly added requirements documents, unused historical design documents, and log data. Pretrained language models are designed to learn semantically based word embedding vectors for natural language processing tasks. Pretrained language models capture information such as word semantics, contextual semantics, syntactic structure, semantic roles, and referential relationships, making them a crucial technology for extracting information from text data. Data analysis scenario information can be understood as semantic expressions relevant to the current scenario.

[0031] Data analysis primarily involves leveraging massive amounts of user data to extract information with commercial value, through modeling and abstraction, that can provide direct or indirect guidance for business operations. For example, "user profiles," crucial for understanding target users, and metrics like "activity," "retention," "conversion," and "profit margin"—all crucial for analyzing a system's business value—are typical examples of information output from business data analysis.

[0032] In this embodiment, if the scenario requirement document is a newly added data table or unused log data, in order to automatically generate a data analysis scenario applicable to the new data table, a pre-trained language model can be used to perform semantic analysis and data extraction on the new data table to obtain the field information and data dictionary information of the data table, and use this information as the data analysis scenario information corresponding to the scenario requirement document. For newly added data analysis scenario requirement documents or unused historical design documents, in order to perform data traceability analysis and automatically generate a data analysis scenario applicable to the new requirement document, the data analysis scenario requirement text is input into the pre-trained language model for semantic parsing and data extraction, and the extracted semantic parsing results are used as the newly added data analysis scenario information.

[0033] S120. Based on the data analysis scenario information, obtain a candidate scenario information set that matches the scenario requirement file from a data analysis scenario pool.

[0034] Among them, the data analysis scenario pool stores the description information of the data analysis scenarios initially designed manually, as well as the description information of the data analysis scenarios that are subsequently automatically matched to unused historical demand files based on artificial intelligence algorithms.

[0035] Optionally, based on the data analysis scenario information, obtaining a candidate scenario information set that matches the scenario requirement file from the data analysis scenario pool may include: finding a target historical requirement file that has the same elements as the scenario requirement file from all historical requirement files; for all target historical requirement files, querying the data analysis scenario that uses the data in the target historical requirement file for data analysis from the data analysis scenario pool, and adding it to the candidate scenario information set.

[0036] In this embodiment, since the data analysis scenarios corresponding to similar scenario requirement files will also have more similarities, after obtaining the data analysis scenario information corresponding to the currently processed scenario requirement file, the target historical requirement file with the same fields as the currently processed scenario requirement file can be searched from the existing historical requirement files, or the target historical requirement file with fields calculated as high similarity by the pre-trained language model as the currently processed scenario requirement file can be searched. Among them, there is no fixed restriction on the definition of high similarity, and it can be set accordingly according to the solution requirements. For example, when the similarity between fields is higher than 90%, it is considered that the two are highly similar. For all target historical requirement files, the scenario using its data for data analysis is queried in the data analysis scenario pool, and the description information of the scenario is included in the candidate scenario information set.

[0037] Optionally, after obtaining a set of candidate scenario information that matches the scenario requirement file from the data analysis scenario pool based on the data analysis scenario information, it may also include: screening out target data query statements whose maximum frequent item set is less than a preset threshold in the candidate scenario information set; and deleting the candidate scenario information corresponding to the target data query statement from the candidate scenario information set.

[0038] Frequent itemsets are sets of subsequences that appear frequently in data. Frequent itemsets are often used in association rule learning scenarios, such as discovering frequently appearing shopping combinations in a customer's shopping list.

[0039] In this embodiment, to further improve the accuracy of matching data analysis scenarios, after determining one or more candidate scenarios corresponding to the scenario requirement document, the data query statements for each candidate scenario are verified one by one, and candidate scenario information whose fields in the data query statements do not meet the requirements is removed from the candidate scenario information set. In other words, candidate scenario information whose maximum frequent item set of the data query statement is less than a preset threshold is removed, so that the remaining data analysis scenarios in the set are those with a high degree of similarity to the currently processed scenario requirement document. The preset threshold value can be 2, 3, or other numerical values.

[0040] S130. Assembling data query statements and front-end component codes of the data analysis scenario corresponding to the scenario requirement file according to the candidate scenario information in the candidate scenario information set.

[0041] In this embodiment, based on multiple candidate scenario information, it is possible to integrate multi-dimensional data across multiple requirement files, assemble data query statements for the currently processed scenario requirement file in the data analysis scenario, and intelligently generate front-end page code through slot-filling code based on the fields, table names, description information and other data in the candidate scenario information to achieve page rendering.

[0042] Optionally, assembling data query statements and front-end component codes for the data analysis scenarios corresponding to the scenario requirement file based on the candidate scenario information in the candidate scenario information set may include: assembling data query statements corresponding to the scenario requirement file based on the fields, description information, and table names in the candidate scenario information set; sending the fields and table names in the candidate scenario information set to the front-end component code as parameters; calculating historical data analysis scenarios whose similarity with the description information in the candidate scenario information set is greater than a first threshold, and sending the most frequently used visualization method of the historical data analysis scenarios to the front-end component code as a parameter for page rendering.

[0043] In this embodiment, through intelligent design and result retrospective calculation, available data analysis scenarios are automatically generated for newly added scenario requirement documents and newly added data tables, the feasibility of data analysis is fully measured, and a data retrieval method that spans multiple tables and integrates multi-dimensional data is designed, effectively eliminating the "data gap" phenomenon where data actually exists but cannot be obtained by the demander. At the same time, through intelligent generation of slot-filling code, historical data is fully mined, the most suitable visualization chart is automatically selected, and the corresponding front-end page code is generated. The connection between data analysis and page development is opened up, significantly reducing the selection and communication costs of data analysts and developers.

[0044] The technical solution of the embodiment of the present invention receives a scenario requirement file to be processed, uses a pre-trained language model to perform semantic analysis and data extraction processing on the scenario requirement file, and obtains data analysis scenario information corresponding to the scenario requirement file; based on the data analysis scenario information, obtains a set of candidate scenario information matching the scenario requirement file from a data analysis scenario pool; according to the candidate scenario information in the candidate scenario information set, assembles data query statements and front-end component codes for the data analysis scenario corresponding to the scenario requirement file, thereby solving the problem of limited business data analysis scenarios that can be proposed when manually analyzing business data, and achieving the beneficial effect of automatically creating adaptive data analysis scenarios and front-end code frameworks according to scenario requirement files.

[0045] Example 2

[0046] Figure 2 This is a flow chart of another method for generating a data analysis scenario according to the second embodiment of the present invention. Based on the above embodiment, this embodiment further provides specific steps for model training of a pre-trained language model. Figure 2 As shown, the method includes:

[0047] S210: Extract mapping relationship data under different data subject domains from the historical demand file.

[0048] Historical requirements documents refer to all existing requirements documents. They can include scenario requirements documents or data tables that have not yet been used for data analysis, as well as scenario requirements documents or data tables that have already been used for data analysis. A data subject domain is typically a collection of closely related data topics, such as data table fields from the same source, business scenarios with similar logical relationships, similar page designs for data visualization, and data query scripts with similar structures but different elements.

[0049] Optionally, the mapping relationship data under different data subject domains include: mapping relationship data between data and data dictionary description; mapping relationship data between data analysis scenarios and business rules; mapping relationship data between data analysis scenarios and data query statements; mapping relationship data between data query statements in data analysis scenarios and visualization methods.

[0050] For example, the mapping relationship data between data and data dictionary descriptions is in the form of "[Field C1 in Table 1] represents [Basic Bond Information]". The mapping relationship data between data analysis scenarios and business rules is in the form of "[Business Rule] in the [Overseas Branch Bond Index Details Query] scenario: Query the details of foreign currency bond investment indicators of overseas institutions based on date, bond code, currency, and transaction combination". The mapping relationship data between data analysis scenarios and data query statements is in the form of "[Query statement] in the Overseas Branch Bond Index Details Analysis scenario is: "SELECT A1, B2, C3, D4 FROM TABLE1, TABLE2"". The mapping relationship data between data query statements and visualization methods in data analysis scenarios is in the form of "[Visualization method of the query statement] in the Overseas Branch Bond Index Details Analysis scenario is: scatter plot".

[0051] In this embodiment, through a highly abstract main framework, the data analysis needs of the entire database are compatible in a flexibly configurable and expandable manner, data dictionary information is introduced as important training data for artificial intelligence algorithms, and the newly added data table information is fully utilized for data analysis scenario design.

[0052] Considering that if two data subject domains have similar logical structures and metadata designs, data analysis logic can often be reused with minor adjustments, the requirements document, based on existing technologies, does not effectively connect the scenarios with the backend logic and data sources. Therefore, this embodiment integrates the three dimensions of information: requirements documents, metadata mapping, and data dictionary, structuring historical design experience information into structured data assets that are easy to analyze and utilize, effectively supporting the construction of a data analysis system in the context of the present invention.

[0053] S220: Perform data cleaning and data extraction operations on the mapping relationship data, and perform ambiguity avoidance processing on the mapping relationship data after the operations.

[0054] Optionally, the data cleaning and data extraction operations are performed on the mapping relationship data, and ambiguity avoidance processing is performed on the mapping relationship data after the operation, which may include: performing data cleaning and data extraction operations on the mapping relationship data between the data analysis scenario and the business rules, and the mapping relationship data between the data analysis scenario and the data query statement, to obtain scenario description information; combining the field data dictionary information in the data query statement with the corresponding scenario description information, and combining the second-order features generated by the combination with all mapping relationship data.

[0055] It should be noted that in the mapping relationship data, the mapping relationship data between data and data dictionary description, and the mapping relationship data between data analysis scenarios and data query statements are used as mapping logic information. The mapping relationship data between data analysis scenarios and business rules are used as training data for the pre-training corpus after data cleaning and processing. The query statements in the mapping relationship data between data analysis scenarios and data query statements are used as training data for the pre-training corpus after data cleaning and data extraction (mainly extracting dictionary information corresponding to table names and fields). The mapping relationship data between data query statements and visualization methods in data analysis scenarios are used as key parameters for determining the type of visualization chart to be selected for data analysis scenarios.

[0056] The purpose of the pre-training corpus is to model the data analysis scenario descriptions in existing requirements documents, thereby obtaining a pre-trained language model suitable for this system. The training data for the pre-training corpus includes: data dictionary descriptions of fields, data dictionary descriptions of fields after keyword substitution; and the data for text structured mining includes: data query statements after keyword substitution.

[0057] In this embodiment, the common content of the data analysis scenario and the data table is the field information involved. However, field information with the same semantic representation (such as the same text description in the data dictionary) in different data analysis scenario information may be mapped to different data tables, which is similar to the ambiguity problem in natural language processing. To solve this problem, this embodiment uses the data dictionary information of the field in the data query statement to map the data dictionary information to the data query statement. m Text description information of the corresponding data analysis scenario I n Combine the combined second-order features Combined with the text description information of the original data analysis scenario, it serves as the training input for the pre-training model.

[0058] S230: Using the processed mapping relationship data as training data, perform model training on the pre-trained language model based on the multi-head attention mechanism.

[0059] In this embodiment, feature input data for each scenario is processed based on the processed mapping relationship data, and pre-trained language models are trained separately to obtain pre-trained language models suitable for each subject domain. Using this language model, this embodiment can perform similarity calculations and use the language model as a natural language feature extractor to support text classification tasks. This allows the identification of existing data analysis scenarios that are most semantically similar to the current scenario's requirement document.

[0060] Among them, the multi-head attention mechanism is the core processing mechanism in mainstream deep learning natural language processing models, such as the Transformer model and the BERT model. The multi-head attention mechanism uses matrix operations of multiple attention head devices and, after matrix linear transformation operations, obtains a vector representation of the text containing contextual semantics. In this embodiment, the multi-head attention mechanism mainly leverages its significant advantages in processing the contextual semantics of long texts, improving the accuracy of tasks such as historical document mining.

[0061] S240: Receive a scenario requirement file to be processed, perform semantic analysis and data extraction on the scenario requirement file using a pre-trained language model, and obtain data analysis scenario information corresponding to the scenario requirement file.

[0062] S250: Based on the data analysis scenario information, obtain a candidate scenario information set that matches the scenario requirement file from a data analysis scenario pool.

[0063] S260. Assemble, based on the candidate scenario information in the candidate scenario information set, a data query statement and a front-end component code for the data analysis scenario corresponding to the scenario requirement file.

[0064] The technical solution of the embodiment of the present invention processes and mines unused historical scenario requirement files and log data, integrates the field mapping relationships under various data subject domains, adopts a multi-head attention mechanism deep learning algorithm to establish a pre-trained language model, and then uses the pre-trained language model to perform complex logical semantic analysis and abstract extraction on the newly added scenario requirement files, matches historical requirement files with high similarity based on the extraction results of the semantic analysis, and extracts their key information to automatically generate scenario information and code parts.

[0065] Example 3

[0066] Figure 3 Schematic diagram of a data analysis scenario generation device according to the third embodiment of the present invention. Figure 3 As shown, the device includes:

[0067] A receiving module 310 is configured to receive a scenario requirement file to be processed, perform semantic analysis and data extraction on the scenario requirement file using a pre-trained language model, and obtain data analysis scenario information corresponding to the scenario requirement file;

[0068] An acquisition module 320 is configured to acquire, based on the data analysis scenario information, a candidate scenario information set matching the scenario requirement file from a data analysis scenario pool;

[0069] The assembly module 330 is used to execute, based on the candidate scenario information in the candidate scenario information set, data query statements and front-end component codes for the data analysis scenario corresponding to the scenario requirement file.

[0070] Optionally, the device further includes: a model training module for performing semantic analysis and data extraction processing on the scenario requirement file using a pre-trained language model before receiving the scenario requirement file to be processed and obtaining data analysis scenario information corresponding to the scenario requirement file.

[0071] Extract mapping relationship data under different data subject domains from historical demand documents;

[0072] Performing data cleaning and data extraction operations on the mapping relationship data, and performing ambiguity avoidance processing on the mapping relationship data after the operations;

[0073] The processed mapping relationship data is used as training data to train the pre-trained language model based on the multi-head attention mechanism.

[0074] Optionally, the mapping relationship data under different data subject domains includes:

[0075] Mapping relationship data between data and data dictionary description;

[0076] Mapping relationship data between data analysis scenarios and business rules;

[0077] Mapping relationship data between data analysis scenarios and data query statements;

[0078] Mapping relationship data between data query statements and visualization methods in data analysis scenarios.

[0079] Optionally, the model training module is used to:

[0080] Performing data cleaning and data extraction operations on the mapping relationship data between the data analysis scenario and the business rules, and the mapping relationship data between the data analysis scenario and the data query statement, to obtain scenario description information;

[0081] The field data dictionary information in the data query statement is combined with the corresponding scene description information, and the second-order features generated by the combination are combined with all mapping relationship data.

[0082] Optionally, the acquisition module 320 is configured to:

[0083] Find the target historical requirement file that has the same elements as the scenario requirement file from all historical requirement files;

[0084] For all target historical requirement files, data analysis scenarios that use the data in the target historical requirement files to perform data analysis are queried from the data analysis scenario pool and added to the candidate scenario information set.

[0085] Optionally, the device further includes:

[0086] A verification module is used to, after obtaining a set of candidate scenario information matching the scenario requirement file from the data analysis scenario pool based on the data analysis scenario information, filter out a target data query statement whose maximum frequent item set is less than a preset threshold in the candidate scenario information set; and delete the candidate scenario information corresponding to the target data query statement from the candidate scenario information set.

[0087] Optionally, assembly module 330 is used to:

[0088] Assembling a data query statement corresponding to the scenario requirement file based on the fields, description information, and table names in the candidate scenario information set;

[0089] Send the fields and table names in the candidate scene information set to the front-end component code as parameters;

[0090] Calculate the historical data analysis scenarios whose similarity with the description information in the candidate scenario information set is greater than a first threshold, and send the most frequently used visualization method of the historical data analysis scenarios to the front-end component code as a parameter for page rendering.

[0091] The data analysis scenario generation device provided in the embodiment of the present invention can execute the data analysis scenario generation method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0092] Example 4

[0093] Figure 4 A schematic diagram of the structure of an electronic device 10 that can be used to implement an embodiment of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or claimed herein.

[0094] like Figure 4 As shown, the electronic device 10 includes at least one processor 11 and a memory, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., which is communicatively connected to the at least one processor 11. The memory stores a computer program that can be executed by the at least one processor. The processor 11 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or the computer program loaded from the storage unit 18 into the random access memory (RAM) 13. Various programs and data required for the operation of the electronic device 10 can also be stored in the RAM 13. The processor 11, ROM 12, and RAM 13 are connected to each other via a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0095] Multiple components in the electronic device 10 are connected to the I / O interface 15, including an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a magnetic disk, an optical disk, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks.

[0096] The processor 11 can be any general-purpose and / or specialized processing component with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various specialized artificial intelligence (AI) computing chips, various processors that run machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 11 executes the various methods and processes described above, such as the method for generating a data analysis scenario.

[0097] In some embodiments, the method for generating a data analysis scenario may be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 10 via the ROM 12 and / or the communication unit 19. When the computer program is loaded into the RAM 13 and executed by the processor 11, one or more steps of the method for generating a data analysis scenario described above may be performed. Alternatively, in other embodiments, the processor 11 may be configured to execute the method for generating a data analysis scenario in any other appropriate manner (e.g., by means of firmware).

[0098] Various embodiments of the systems and techniques described herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), system-on-chip systems (SOCs), programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include being implemented in one or more computer programs that are executable and / or interpreted on a programmable system that includes at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0099] Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when the computer program is executed by the processor, the functions / operations specified in the flowcharts and / or block diagrams are implemented. The computer program may be executed entirely on the machine, partially on the machine, as a stand-alone software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0100] In the context of the present invention, computer-readable storage media can be tangible media that can contain or store a computer program for use with an instruction execution system, device or equipment or used in combination with an instruction execution system, device or equipment. Computer-readable storage media can include but are not limited to electronic, magnetic, optical, electromagnetic, infrared or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. Alternatively, computer-readable storage media can be machine-readable signal media. More specific examples of machine-readable storage media can include electrical connections based on one or more lines, portable computer disks, hard disks, random access memories (RAM), read-only memories (ROM), erasable programmable read-only memories (EPROM or flash memory), optical fibers, portable compact disk read-only memories (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0101] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0102] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0103] A computing system may include clients and servers. The clients and servers are typically remote from each other and typically interact via a communication network. This client-server relationship arises through computer programs running on the respective computers, creating a client-server relationship. The server may be a cloud server, also known as a cloud computing server or cloud host. This server is a hosting product within the cloud computing service ecosystem that addresses the management difficulties and limited scalability of traditional physical hosting and VPS services.

[0104] It should be understood that the various forms of the processes shown above can be used to reorder, add, or delete steps. For example, the steps described in the present invention can be performed in parallel, sequentially, or in a different order, as long as the desired results of the technical solution of the present invention can be achieved. This is not limited herein.

[0105] The above specific embodiments do not limit the scope of protection of the present invention. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions may be made based on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention are intended to be included within the scope of protection of the present invention.

Claims

1. A method for generating a data analysis scenario, characterized in that: include: Receive a scenario requirement file to be processed, perform semantic analysis and data extraction on the scenario requirement file using a pre-trained language model, and obtain data analysis scenario information corresponding to the scenario requirement file; The scenario requirement file includes: a newly added data table or a newly added requirement document; the data analysis scenario information includes: field information and data dictionary information of the newly added data table, or the semantic parsing result of the newly added requirement document; Based on the data analysis scenario information, obtaining a candidate scenario information set matching the scenario requirement file from a data analysis scenario pool; Assembling, based on the candidate scenario information in the candidate scenario information set, a data query statement and a front-end component code for the data analysis scenario corresponding to the scenario requirement file; Wherein, obtaining a candidate scenario information set matching the scenario requirement file from a data analysis scenario pool based on the data analysis scenario information includes: Find the target historical requirement file that has the same elements as the scenario requirement file from all historical requirement files; For all target historical requirement files, data analysis scenarios that use the data in the target historical requirement files to perform data analysis are queried from the data analysis scenario pool and added to the candidate scenario information set.

2. The method according to claim 1, characterized in that Before receiving the scenario requirement file to be processed and performing semantic analysis and data extraction processing on the scenario requirement file using the pre-trained language model to obtain data analysis scenario information corresponding to the scenario requirement file, the method further includes: Extract mapping relationship data under different data subject domains from historical demand documents; Performing data cleaning and data extraction operations on the mapping relationship data, and performing ambiguity avoidance processing on the mapping relationship data after the operations; The processed mapping relationship data is used as training data to train the pre-trained language model based on the multi-head attention mechanism.

3. The method according to claim 2, characterized in that The mapping relationship data under different data subject domains include: Mapping relationship data between data and data dictionary description; Mapping relationship data between data analysis scenarios and business rules; Mapping relationship data between data analysis scenarios and data query statements; Mapping relationship data between data query statements and visualization methods in data analysis scenarios.

4. The method according to claim 3, characterized in that The performing of data cleaning and data extraction operations on the mapping relationship data, and performing ambiguity avoidance processing on the mapping relationship data after the operations, includes: Performing data cleaning and data extraction operations on the mapping relationship data between the data analysis scenario and the business rules, and the mapping relationship data between the data analysis scenario and the data query statement, to obtain scenario description information; The field data dictionary information in the data query statement is combined with the corresponding scene description information, and the second-order features generated by the combination are combined with all mapping relationship data.

5. The method according to claim 1, wherein After acquiring a candidate scenario information set matching the scenario requirement file from a data analysis scenario pool based on the data analysis scenario information, the method further includes: Filtering out target data query statements whose maximum frequent itemset is less than a preset threshold value from the candidate scene information set; The candidate scene information corresponding to the target data query statement is deleted from the candidate scene information set.

6. The method according to claim 1, characterized in that The step of assembling a data query statement and a front-end component code for a data analysis scenario corresponding to the scenario requirement file based on the candidate scenario information in the candidate scenario information set includes: Assembling a data query statement corresponding to the scenario requirement file based on the fields, description information, and table names in the candidate scenario information set; Send the fields and table names in the candidate scene information set to the front-end component code as parameters; Calculate the historical data analysis scenarios whose similarity with the description information in the candidate scenario information set is greater than a first threshold, and send the most frequently used visualization method of the historical data analysis scenarios to the front-end component code as a parameter for page rendering.

7. A device for generating a data analysis scenario, characterized in that: include: A receiving module is used to receive a scenario requirement file to be processed, perform semantic analysis and data extraction on the scenario requirement file using a pre-trained language model, and obtain data analysis scenario information corresponding to the scenario requirement file; The scenario requirement file includes: a newly added data table or a newly added requirement document; the data analysis scenario information includes: field information and data dictionary information of the newly added data table, or the semantic parsing result of the newly added requirement document; An acquisition module, configured to acquire, based on the data analysis scenario information, a set of candidate scenario information that matches the scenario requirement file from a data analysis scenario pool; An assembly module, configured to execute, based on the candidate scenario information in the candidate scenario information set, a data query statement and a front-end component code for assembling the data analysis scenario corresponding to the scenario requirement file; The acquisition module is used to: Find the target historical requirement file that has the same elements as the scenario requirement file from all historical requirement files; For all target historical requirement files, data analysis scenarios that use the data in the target historical requirement files to perform data analysis are queried from the data analysis scenario pool and added to the candidate scenario information set.

8. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor, and the computer program is executed by the at least one processor to enable the at least one processor to execute the method for generating a data analysis scenario according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the method for generating a data analysis scenario according to any one of claims 1 to 6 when executed.

Images