Big model-based data processing method and device, electronic equipment and storage medium

Abstract

The present disclosure provides a large model-based data processing method and device, electronic equipment and storage medium, relating to the field of artificial intelligence, and especially to the application fields of generative search, document intelligent editing, intelligent assistant, virtual assistant, intelligent e-commerce, etc. The specific implementation scheme is: receiving data registration information; wherein the data registration information includes processing rule information for N data sources and mode description of target data; N is an integer greater than or equal to 1; based on the processing rule information and the mode description, generating an execution code adapted to a preset data protocol; based on the execution code, processing the N data sources to obtain the target data and storing the target data in a target database. The present disclosure embodiment can reduce the technical threshold for submitting data to the data center and reduce the labor cost of application development.

Description

Technical Field

[0001] This disclosure relates to the field of artificial intelligence, particularly to applications such as generative search, intelligent document editing, intelligent assistants, virtual assistants, and intelligent e-commerce. Background Technology

[0002] Data open platforms allow application developers to submit data and applications, bringing them significant traffic distribution. However, they also have limitations. Specifically, data open platforms only allow application developers to submit data in specified formats, making the process relatively limited, and requiring application developers to invest considerable time and effort in protocol adaptation. Summary of the Invention

[0003] This disclosure provides a data processing method, apparatus, electronic device, and storage medium based on a large model.

[0004] According to one aspect of this disclosure, a data processing method based on a large model is provided, comprising:

[0005] Receive data registration information; wherein, the data registration information includes processing rule information for N data sources and a pattern description of the target data; N is an integer greater than or equal to 1;

[0006] Based on the processing rule information and the mode description, generate execution code that is compatible with the preset data protocol;

[0007] Based on the execution code, the N data sources are processed to obtain the target data, and the target data is stored in the target database.

[0008] According to another aspect of this disclosure, a data processing apparatus based on a large model is provided, comprising:

[0009] The registration module is used to receive data registration information; wherein, the data registration information includes processing rule information for N data sources and a pattern description of the target data; N is an integer greater than or equal to 1;

[0010] The code generation module is used to generate executable code that is compatible with the preset data protocol based on the processing rule information and the mode description.

[0011] The data import module is used to process the N data sources based on the executed code to obtain the target data and store the target data in the target database.

[0012] According to another aspect of this disclosure, an electronic device is provided, comprising:

[0013] At least one processor; and

[0014] The memory is communicatively connected to the at least one processor; wherein,

[0015] The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform any of the methods described in the present disclosure.

[0016] According to another aspect of this disclosure, a non-transitory computer-readable storage medium is provided storing computer instructions, wherein the computer instructions are used to cause the computer to perform any of the methods according to embodiments of this disclosure.

[0017] According to another aspect of this disclosure, a computer program product is provided, including a computer program that, when executed by a processor, implements any of the methods according to embodiments of this disclosure.

[0018] This disclosure supports application developers in submitting data by submitting data registration information. The data submission method is more flexible and can reduce the technical threshold for data submission and reduce the labor cost of application development.

[0019] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0020] The accompanying drawings are provided to better understand this solution and do not constitute a limitation of this disclosure. Wherein:

[0021] Figure 1 This is a schematic flowchart of a data processing method based on a large model provided in an embodiment of this disclosure;

[0022] Figure 2 This is a schematic diagram illustrating an application example of a data processing method based on a large model;

[0023] Figure 3 This is a schematic diagram of the logical topological relationship of the data in the application example;

[0024] Figure 4 This is a schematic diagram of the storage structure of the target database in an embodiment of this disclosure;

[0025] Figure 5 This is a flowchart illustrating a data processing method based on a large model provided in another embodiment of this disclosure;

[0026] Figure 6 This is a schematic diagram of online retrieval data in an application example;

[0027] Figure 7This is a schematic diagram illustrating an application example of semantic recall.

[0028] Figure 8 This is a schematic block diagram of a data processing apparatus based on a large model provided in an embodiment of the present disclosure;

[0029] Figure 9 This is a schematic block diagram of a data processing apparatus based on a large model provided in another embodiment of this disclosure;

[0030] Figure 10 This is a block diagram of an electronic device used to implement the large-model-based data processing method of the embodiments of this disclosure. Detailed Implementation

[0031] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0032] Figure 1 A flowchart illustrating a large-model-based data processing method according to an embodiment of this disclosure is shown. This method can be applied to a large-model-based data processing device, which can be deployed in an electronic device. The electronic device can be a single-machine or multi-machine terminal, server, or other processing device. The terminal can be a mobile device, a personal digital assistant (PDA), a handheld device, a computing device, an in-vehicle device, a wearable device, or other user equipment (UE). In some possible implementations, the electronic device can be applied to a data center; for example, the electronic device can be a computing engine device in a data center. In some possible implementations, the method can also be implemented by a processor calling computer-readable instructions stored in memory. Figure 1 As shown, the method may include:

[0033] S110. Receive data registration information; wherein, the data registration information includes processing rule information for N data sources and a pattern description of the target data; N is an integer greater than or equal to 1;

[0034] S120. Based on processing rule information and pattern description, generate execution code that is compatible with the preset data protocol;

[0035] S130. Based on the execution code, process N data sources to obtain target data, and store the target data in the target database.

[0036] It is understood that data registration information is the information that users (e.g., application developers) need to submit when registering data in the data center. In this embodiment of the disclosure, the data registration information includes processing rule information for N data sources; here, the data source refers to the data source that is accessed or submitted to the data center, and this embodiment of the disclosure supports access to one or more data sources.

[0037] Optionally, the submission format for processing rule information is not limited; for example, it does not need to be in JSON / xml format, thus saving users a lot of time and effort in implementing protocol adaptation.

[0038] Optionally, the submission format for processing rule information can be SQL (Structured Query Language) statements. SQL statements are designed for users of different database systems. By submitting processing rule information using SQL statements, data migration between various database systems and data centers can be achieved without having to relearn a completely new language, thereby reducing users' time and labor costs.

[0039] It should be noted that the SQL statements in this embodiment may not be strictly standardized SQL statements; that is, they may be SQL-like statements. SQL-like statements refer to statements that are the same as or similar to SQL statements in some aspects. They may not fully comply with the SQL standard, but they are similar to SQL in syntax and function to facilitate user migration and learning.

[0040] In this embodiment of the disclosure, the processing rule information is used to indicate how the data from the data source is processed, including common processing of various fields, such as character set encoding and URL normalization. When the data registration information includes processing rule information for multiple data sources, this processing rule information can also be used to indicate how to concatenate or merge data from different data sources.

[0041] In this embodiment of the disclosure, the data registration information also includes a schema description of the target data. This schema description is used to indicate what kind of target data can be obtained based on N data sources. That is, it is used to describe one or more types of information such as the definition, description, and format of the target data, thereby guiding the electronic device to obtain or verify the target data.

[0042] According to embodiments of this disclosure, after obtaining data registration information, the electronic device can generate execution code adapted to a preset data protocol based on the processing rules and the pattern description of the target data therein. The execution code then processes the data source to obtain the target data for storage in the target database. For example, the preset data protocol could be a JSON data protocol. In other words, the user only needs to submit data registration information, and the electronic device can generate corresponding unified data protocol execution code based on that information to process the target data. This eliminates the need for the user to write their own execution code to process the data source according to the data protocol requirements of the data center. This provides a more flexible data submission method and lowers the technical threshold for data submission, as well as reducing the labor costs of application development.

[0043] In some embodiments, the processing rule information includes SQL statements, which describe the concatenation rules of N data sources. Accordingly, step S120 above, based on the processing rule information and the pattern description, generates execution code adapted to a preset data protocol, including:

[0044] The SQL statement is parsed using a syntax tree to obtain a structured JSON configuration; the JSON configuration is used to represent the logical topological relationship between N data sources and the target data.

[0045] Based on JSON configuration, generate executable code that is compatible with the preset data protocol.

[0046] Here, logical topology refers to the generation dependency relationship between different data sources and between the data source and the target data. In this embodiment, multiple data sources can be submitted. When the data registration information includes processing rule information for multiple data sources, the processing rule information includes splicing rules for multiple data sources, that is, describing how to splice data from different data sources to obtain the target data through step-by-step splicing processing.

[0047] For example, N data sources include data source 1, data source 2, and data source 3. The SQL statement entered by the user can describe how to obtain field C based on field A in data source 1 and field B in data source 2, and then obtain the target data E based on field C and field D in data source 3. After receiving the data registration information, the electronic device will parse the SQL statement using a syntax tree to obtain a structured JSON configuration. This configuration information, which meets the requirements of the preset data protocol, represents the logical topological relationship between data source 1, data source 2, data source 3, target data E, and the intermediate fields C and D. Based on this configuration information, it generates executable code that is compatible with the preset data protocol.

[0048] According to the above method, users only need to provide processing rule information based on SQL statements, and electronic devices can use the SQL statements to perform concatenation processing of multiple data sources, reducing the technical threshold for data submission and reducing the labor cost of application development.

[0049] In some embodiments, the data registration information further includes the import configuration of N data sources; the import configuration includes the import method of each of the N data sources and the parameter information associated with the import method. Accordingly, the above-described data processing method based on a large model further includes:

[0050] Based on the configuration of importing N data sources, data from N data sources is imported into the data pipeline for processing.

[0051] In other words, the above method allows users to import or register data by configuring the import method and configuration parameters. Here, the import method can refer to the way data is read or retrieved, such as batch scan import or streaming import. Specifically, batch scan import can refer to writing data in batches by scanning remote file systems / data systems; streaming import can refer to writing updated or incremental data from the data source to a transit data queue via a data queue or a Remote Procedure Call. The parameter information associated with the import method can include the type of data system associated with the import method, the data protocol type, etc.

[0052] According to the above embodiments, users can import data by configuring the import method and configuration parameter information, which simplifies the data registration process. Users do not need to describe the data import method in terms of protocol adaptation, which can reduce the technical threshold for data submission and reduce the labor cost of application development.

[0053] In some embodiments, the N data sources include a trigger source and M concatenation sources for concatenating with the trigger source, where M is an integer greater than or equal to 1; the trigger source is introduced in a batch scan manner and / or a streaming manner; each of the M concatenation sources is introduced in a manner including at least one of local dictionary introduction, remote key-value (KV) introduction, table storage introduction, and database introduction.

[0054] The trigger source, as the name suggests, is the original data accessed by the business. Users can use the trigger data as the original data to associate and concatenate data with the concatenation source, or they can choose not to concatenate. Business data access must have a trigger source, and each business must have at least one trigger source. The introduction of trigger sources can include batch introduction and / or streaming introduction.

[0055] For example, the trigger source can take one or more of the following forms:

[0056] Batch triggering: The data sources associated with batch triggering are similar to MR (MapReduce) tasks that scan the full data of the corresponding business. This method supports scanning remote file systems (common AFS, HDFS), as well as data systems that support full scans, such as HBase and Table type storage.

[0057] Scheduled triggering: Essentially a special type of batch triggering, it executes a special command within a specified time range to trigger the event. Here, a new process has been introduced that integrates a scheduled task registration mechanism to trigger the event on a scheduled basis by executing task commands.

[0058] Streaming triggering: Streaming triggering is also a common business practice. Business data updates are written to a relay data queue via a data queue or RPC.

[0059] Trigger-triggered: This method is commonly used in internal database data synchronization, such as SQL Server's CDC synchronization and MySQL's DTS synchronization.

[0060] The purpose of the splicing source is to supplement the original trigger source with data. The splicing source cannot exist independently of the trigger source. Its main function is to use the data sent by the original trigger source and combine it with association rules to splice additional source data. In this embodiment, the splicing source can have one or more of the following forms:

[0061] Local dictionary: Users bring online dictionaries delivered to the system via data stream. This mainly supports several special dictionary formats. Generally, dictionaries are very small and take up little space. When splicing these, they can be loaded through memory or through LSM storage structure.

[0062] Remote KV: This is the most common way of concatenating data. It uses the key of the original data to query and supplement the data. Although the latency is slightly higher than that of the local dictionary, the amount of data that can be stored is not limited by local resources.

[0063] Table storage: This storage structure can also be used as a batch trigger source. However, since it supports both Scan and Seek data, it can be used as a source to supplement the query key information during the data concatenation process.

[0064] Databases: Databases are the most common type of storage in the traditional sense, serving as the source of data. These databases include various relational and non-relational databases, such as MongoDB, MySQL (PostGreSql), and SQL Server.

[0065] The above embodiments provide diverse data import methods, basically covering all data import methods in related technologies. This allows users to import data by selecting configurations, reducing users' time and labor costs, optimizing the user experience in the data center, and facilitating data aggregation and application.

[0066] To facilitate understanding of the above technical solution, an application example is provided below. Figure 2 This is a schematic diagram illustrating an example of the application. (For example...) Figure 2 As shown, the data imported by users is broadly divided into two categories: trigger sources and concatenation sources. The triggering methods (import methods) of trigger sources mainly include timed triggering, batch triggering, traffic triggering, and trigger-based triggering. Concatenation sources include local dictionaries, remote key-value pairs, table storage, and databases. Refer to the descriptions in the preceding embodiments for details.

[0067] In this application example, user data processing essentially involves completing data registration, which includes registering several pieces of information:

[0068] Trigger source 210: Describes the original method of importing business data (batch, streaming) and data protocol (PB, JSON);

[0069] Concatenation Source 220: Describes the data source required for concatenating the original data. If the concatenation source dictionary supports global reuse and already exists, it does not need to be registered again. If the required dictionary is not available globally, it will be added.

[0070] Processing rule 230 (including concatenation rules): A trigger source can be configured with data from multiple concatenation sources for supplementary concatenation. Concatenation rules are described using a SQL-like language, which can process and expand the original data set, including common basic processing of fields, such as character set encoding, URL normalization, forward and reverse order processing, and concatenation with the corresponding concatenation source to associate with new resources.

[0071] Output Schema Description 240: User-defined protocol and schema for distributing business data. The data fields distributed can not only be extracted from the original trigger source, but can also be changed through Join (the process of combining rows from two or more database tables based on one or more common attributes) and any fields in the data transformation process.

[0072] There are two main operating modes for the service, where all processing logic runs in a consistent manner, including:

[0073] Batch mode: mainly uses STDIN input and STDOUT output, primarily supporting batch calculation methods such as MR;

[0074] Streaming mode: Deployed as a general-purpose service, supporting subscriptions to the above. Figure 2Various service modules.

[0075] The following describes the overall data processing procedure:

[0076] 1. The original trigger source first sends data requests one by one in a specified way, transforming data from different data sources into data that is distributed to the data pipeline one by one;

[0077] 2. Data in the data pipeline is processed through specified business logic. Within each business logic, the data is transformed into an actual execution topology based on the splicing rules of the original business configuration. Note that the execution topology is a logical topology relationship generated based on the original data dependencies of the business. Each data source (including trigger source + splicing source) is abstracted into a table in the database. The association relationship configured by the business is directly represented by the database JOIN relationship. In short, regardless of the trigger source, the data will eventually be transformed into a unified and special JSON data protocol.

[0078] 3. Ultimately, all related data will be aggregated into a multi-dimensional field (target data), and users can select the data fields they need to distribute the data.

[0079] The following are specific examples using product information:

[0080] The trigger source is the original product information, including: product category number, product number, product name, and warehouse name.

[0081] The splicing sources include:

[0082] Source 1: Mapping information between warehouse name and ID, stored using a local dictionary that can be imported into the data center;

[0083] Source 2: Warehouse details, including warehouse ID, warehouse name, region ID, staff information, and quantity of goods, stored in a table;

[0084] Source 3: Regional information, including regional ID and warehouse ranking, imported via remote key-value pairs;

[0085] Source 4: Product Source Information: Product ID, source information (including province, city, district / county, address, etc.), imported via remote key-value store (KV).

[0086] In addition to configuring the data source mentioned above, users also need to configure the output schema (a schema description of the target data). The output schema configuration can be used for final data validation. As a form of double validation, if the user does not explicitly configure it, it can be generated through processing rules. The output schema can be: the target data includes product name, product category number, source information, warehouse location, and warehouse ranking.

[0087] Within the computing engine, the topological relationships between data need to be identified, and the concatenation operation is implemented based on simple user-written statements (such as SQL statements). Specifically, users need to configure concatenation rules, such as associating the concatenation source with the mapping information between warehouse name and ID, warehouse details, regional information, and product source information based on the trigger source of the original product information, and finally generating the target data output table. The above configuration is expressed through SQL statements, and the general concatenation process is described as follows:

[0088] 1. Combine the original product information (trigger source) and the mapping information between the warehouse name and ID (combination source 1) to generate virtual table 1. Virtual table 1 has two columns: product number and warehouse ID.

[0089] 2. By associating virtual table 1 with warehouse details (concatenation source 2), configure and generate virtual table 2, which has 3 columns: product number, warehouse ID, and region ID;

[0090] 3. By associating virtual table 2 with regional information (splicing source 3), configure and generate virtual table 3, which has two columns: product number and warehouse ranking;

[0091] 4. By linking the original student information and the product source information (splicing source 4), configure and generate a virtual table 4 with 5 columns: product number, province, city, district / county, and address;

[0092] 5. By directly extracting the original product information and virtual tables 3 and 4, all the final output fields, i.e., the target data, are generated.

[0093] The logical topological relationships of the above data are as follows: Figure 3 As shown. Since the generation of each field is dependent on others, a new flowchart can be constructed by deducing the data lineage of input and output. Specifically, the calculation engine can parse the user's original SQL into a structured JSON configuration using a syntax tree, and then generate executable code to process the data source and obtain the target data.

[0094] In some embodiments, step S130 of the above-described large model-based data processing method, storing the target data in the target database, includes: determining the descriptive information of each field in the target data based on the schema description of the target data; and storing the target data and the descriptive information of each field in association in the target database; wherein the descriptive information of each field is used to call the target data based on the language model.

[0095] In this embodiment of the disclosure, the language model can be a large model, such as a Large Language Model (LLM) for processing multimodal data. An LLM is a natural language processing model with massive parameters built using deep learning techniques, capable of learning and understanding the complex characteristics of language, including but not limited to syntax and semantics. Optionally, the large language model can also be applied as a conversational generation model, capable of conducting dialogues based on natural language, thereby processing input information based on natural language to facilitate the retrieval of target data.

[0096] In the above embodiments, by storing the descriptive information of each field in association when storing the target data, the descriptive information of each field can be used to assist the language model in understanding each field, thereby optimizing the accuracy of retrieving the target data. In other words, the above data storage method can facilitate AI-friendly implementation.

[0097] In one application example, the target data mentioned above is industry data, specifically in the form of a data table. Industry data needs to be easily accessible to upstream businesses and models. For businesses, the data hierarchy can be slightly more complex, the key being to clearly explain the meaning of the data fields within that industry, facilitating business understanding, processing, and subscription. For models, the data hierarchy should be minimized, ideally with an automatically flattened structure, and each field should include detailed descriptions and usage notes to achieve AI friendliness. Here, the detailed descriptions and usage notes of fields are collectively referred to as data description information (also known as schema).

[0098] For example, the target database for storing industry data may include a non-relational database called TableStore. This database meets the following conditions:

[0099] 1. Maintain a consistent schema across all industries, and upstream entities need not concern themselves with data dependencies. This is because industry data production often relies on the combination of multiple datasets, which are often interdependent. To minimize the business's understanding cost, these relationships should be hidden, allowing the business to see the complete picture of industry data, ready to use out of the box.

[0100] 2. Supports data subscription by different industry dimensions and indicators.

[0101] 3. It supports dynamic expansion, has sufficient disaster recovery capabilities, is easy to migrate, has timeliness guarantees, and its cost is significantly lower than that of relational databases.

[0102] Figure 4 This is a schematic diagram of the storage structure of the target database in an embodiment of this disclosure. Figure 4 In this context, the industry is the table name (e.g., the education industry is a table), LG (Locality Group) is a physical layer concept, mainly used for data isolation. Taking the macroeconomic industry as an example, LG0 and LG1 can respectively identify GDP and population; CF (Column Family) represents the set of indicators under each sub-industry; and a column is the specific indicator information. In relational databases, a single or multiple columns combined together constitute a specific row of data.

[0103] To facilitate data retrieval, a row key must be included in the data table, similar to the primary key in a relational database.

[0104] For example, the generation rule for row_key can be: (dimension level:dimension value;)*; for instance: if the regional dimension level is: province, the regional dimension value is: Shandong, the time dimension level is: quarter, and the time dimension value is: 2023Q4, then the key value would be: province: Shandong; quarter: 2023Q4. This allows you to retrieve the corresponding records from a specific indicator based on its dimensions.

[0105] In the above application example, a table can be maintained to show the relationship between industry IDs and data descriptions (i.e., a table that maintains the correspondence between record fields and description information). When an upstream business subscribes, it can directly obtain detailed descriptions of the data for each industry, including descriptions of the fields, to facilitate model understanding. The data relationship is maintained on the platform, and in the table, it is a specific remote lookup schema address, which allows the business to make changes directly on the platform without needing to perform a full data replay to update the schema.

[0106] Based on the database design described above, data subscription methods are divided into incremental and full subscription. For full subscription, it is generally a direct connection to the database, and for the data table, it provides a way to scan by specifying CF / row_key. For incremental subscription, the data is generally sent to a message queue, and the downstream listens to the message queue to complete the triggered data processing or further storage into the database.

[0107] In some embodiments, the large-model-based data processing method also includes methods for retrieving data from the target database. Specifically, such as... Figure 5 As shown, the above-mentioned data processing methods based on large models also include:

[0108] S510, Receive query information input by the target object;

[0109] S520. Based on the query information, retrieve the indicator terms that match the query information and the explanatory information of the indicator terms from the target database;

[0110] S530. Using the first language model, the system obtains the return results corresponding to the query information based on the query information, indicator words, and the description information of the indicator words, and displays the return results to the target object.

[0111] It is understood that the method of storing data in the database can refer to the aforementioned embodiment, which will associate and store fields (i.e., index terms) with descriptive information. Based on this, when the query information input by the target object is received, the associated index terms and corresponding descriptive information can be retrieved, so as to use the descriptive information and the first language model to obtain the returned result (e.g., the specific value of a certain row in the data table corresponding to the index term).

[0112] Optionally, the target object can be a user conducting an online search. The query information entered by the target object is the input query. In the search process of this embodiment, the index terms and their descriptive information are first retrieved based on the input query. For example, in conjunction with the foregoing embodiments, the target database can store a table that records the correspondence between fields and descriptive information, where the fields are index terms, and the index terms and their corresponding descriptive information can be retrieved from the table based on the input query.

[0113] For example, semantic recall can be used to retrieve indicator terms and their descriptive information. Specifically, firstly, indicator terms and their common synonyms are vectorized and stored in a semantic database. Then, when an input query is received, it is also vectorized. The semantic indexing engine can then find the vector representation of the indicator term that most closely matches the input query's vector representation, and thus obtain the matching indicator term. Next, based on the indicator term mapping dictionary, various synonyms can be mapped to standard indicator names. Optionally, after retrieving the indicator term, the descriptive information can be retrieved from a table in the target database that records the corresponding descriptive information. Alternatively, the descriptive information can be included in the summary information during the indicator term database creation phase, allowing direct retrieval of the descriptive information using the input query's vector representation.

[0114] In this embodiment, after obtaining the indicator terms and their descriptive information, the rich syntactic and semantic information can be used to obtain the return results corresponding to the query information using a first language model, and then the return results can be displayed to the target object. It can be seen that the above embodiment utilizes the descriptive information of the indicator terms and the language model, which can improve the accuracy of the returned results.

[0115] In some embodiments, step S530 above, using a first language model, to obtain the return result corresponding to the query information based on query information, indicator words, and explanatory information of the indicator words, includes:

[0116] Based on the query information, indicator terms, and descriptions of the indicator terms, the input information is obtained;

[0117] Input information into the first language model to obtain multiple subtasks output by the first language model and the execution order among the multiple subtasks;

[0118] Based on the execution order, multiple subtasks are executed, and the returned results are obtained.

[0119] Specifically, in the above embodiments, the input information of the first language model includes query information, retrieved index words, and corresponding explanatory information; the processing result of the first language model is multiple sub-tasks planned according to the input information and the execution order between the multiple sub-tasks.

[0120] In other words, in the above embodiments, a large model is used to break down subtasks, and then the results are obtained by executing the subtasks sequentially. By breaking down the subtasks, the needs of the target object are fully understood, thereby enabling more accurate acquisition of the return results corresponding to the query information.

[0121] In some embodiments, the input information is also obtained based on the description information of a plurality of preset tool interfaces. That is, the input information is obtained based on query information, indicator terms, and descriptions of the indicator terms, including: obtaining input information based on query information, indicator terms, descriptions of the indicator terms, and descriptions of a plurality of preset tool interfaces. For example, prompt information can be constructed based on the above information and used as input information for the model. This prompt information can be a conversational language used to indicate and explain to the model the callable tool interfaces, the query information of the current retrieval task, the associated indicator terms, and the descriptions of the indicator terms. Thus, the model can plan multiple sub-tasks that the electronic device is capable of executing based on the callable tool interfaces, and obtain the returned results through these multiple sub-tasks.

[0122] In other words, according to the execution order, multiple subtasks are executed to obtain the return result, including: according to the execution order, calling the tool interface corresponding to each of the multiple subtasks to execute the multiple subtasks and obtain the return result.

[0123] The above embodiments provide the model with descriptions of multiple tool interfaces, thereby enabling the model to plan the sub-tasks to be executed more accurately and thus improving the accuracy of the returned results.

[0124] In some embodiments, step S530 above, displaying the returned result to the target object, includes: using a second language model to determine the chart style of the returned result based on the query information; using a third language model to generate a target chart based on the chart style and the returned result; and displaying the target chart to the target object.

[0125] In the embodiments of this disclosure, the first language model, the second language model, and the third language model can all be the same model or different models, and this disclosure does not limit this. Furthermore, optionally, each of the above language models can be a large model, an LLM, or a conversational generation model.

[0126] According to the above embodiments, a language model can be used to determine the chart style of the returned results based on query information, and a target icon can be generated based on the returned results. The returned results are then displayed to the target audience through a chart. This achieves the automatic generation of easy-to-understand charts based on user data and needs, enhancing data readability and making it easier for users to extract useful information from the charts.

[0127] To facilitate understanding of the above technical solutions, the following will be combined with... Figure 6 Provide a specific application example.

[0128] In this application example, such as Figure 6 As shown, online data retrieval includes three stages:

[0129] Phase 1: Obtain the indicators and their schema descriptions;

[0130] Phase 2: Subtask Breakdown and Data Acquisition

[0131] Phase 3: Presentation style organization.

[0132] In Phase 1, based on the user's input query, index terms and their schema descriptions that satisfy the query are retrieved. This phase is achieved through semantic recall. See details for further information. Figure 7This is a schematic diagram of the semantic recall method. Specifically, firstly, the indicator words and their common synonyms are vectorized and built into a semantic database. Then, for a user-input query, such as "What is the typical rainfall in area A in July?", it is also vectorized using a vector encoder 71. In the semantic indexing engine 72, the vector representation of the indicator word closest to the query's vector representation can be found (e.g., finding rainfall among the indicator words rainfall, temperature, and humidity), and the corresponding indicator word can be obtained accordingly. Finally, based on the indicator word mapping dictionary, various synonyms can be mapped to standard indicator names. During the indicator word database construction phase, the schema description information corresponding to the indicator word can be incorporated into the summary information. Thus, when the indicator word is obtained, the schema description information can be retrieved through the summary information.

[0133] In Phase 2, subtasks are broken down and data is acquired based on the user query and the key terms and schema obtained in the previous phase. This phase is implemented using a Generative Large Language Model (LLM) and multiple tool interfaces. The tool interfaces include:

[0134] 1. Data query interface: A data retrieval service built on a data schema, capable of querying and filtering data based on fields in the schema;

[0135] 2. Calculator service, which provides numerical calculations with units, and supports the types of calculations including addition, subtraction, multiplication and division;

[0136] 3. Location normalization service, which can retrieve standard location names (country, province, city, district, county) based on the input location;

[0137] 4. Time normalization service, which can obtain a standard representation of time based on the input time description;

[0138] 5. Ranking service, which can calculate the ranking of a given value 'a' in a given array of values ​​'A'.

[0139] Specifically, the process in Phase 2 is as follows:

[0140] 1. After the model receives the user's query, corresponding keyword, and data schema, it independently considers how to filter data according to the given schema under this keyword to answer the user's query.

[0141] 2. The order in which the tools are called based on the thinking results, where each API call includes the API name and the parameters to be filled in. For example, assuming the user query is: Which city has a larger area, Nanjing or Suzhou? Based on the area indicator obtained in Phase 1, the order in which the model generates the tools, or the order in which the subtasks are executed, is as follows:

[0142] By calling the location normalization service, Nanjing and Suzhou are normalized to the cities of Nanjing and Suzhou.

[0143] Use the data query interface to query the latest area data for Nanjing and Suzhou.

[0144] The size relationship between two area data can be determined by calling the calculator service.

[0145] 3. Execute the interface calls sequentially according to the order given by the model to obtain the final return result.

[0146] In Phase 3, data is presented based on the data generated in Phase 2. This phase is also based on LLM. After seeing the user's query, the model analyzes it to determine the appropriate chart style to be displayed, mainly including: bar charts, line charts, pie charts, leaderboards, and short text answers. For different chart styles, the model generates corresponding chart elements.

[0147] As can be seen, this embodiment of the disclosure uses multiple steps to break down the task, which can solve complex data requirements and achieve the needs that cannot be met by a single query by calling multiple tools. Finally, the readability of the obtained data is improved by displaying it in charts.

[0148] It is understandable that the above methods can be used in application scenarios such as generative search, intelligent document editing, intelligent assistants, virtual assistants, and intelligent e-commerce.

[0149] According to embodiments of this disclosure, this disclosure also provides a data processing apparatus based on a large model. Figure 8 A schematic block diagram of a large-model-based data processing apparatus provided in an embodiment of this disclosure is shown, such as... Figure 8 As shown, the device includes:

[0150] The registration module 810 is used to receive data registration information; wherein, the data registration information includes processing rule information for N data sources and a pattern description of the target data; N is an integer greater than or equal to 1;

[0151] The code generation module 820 is used to generate executable code that is compatible with the preset data protocol based on processing rule information and pattern description.

[0152] The data import module 830 is used to process N data sources based on the executed code, obtain the target data, and store the target data in the target database.

[0153] In some embodiments, the processing rule information includes a Structured Query Language (SQL) statement, which describes the concatenation rules of N data sources.

[0154] Code generation module 820 is used for:

[0155] The SQL statement is parsed using a syntax tree to obtain a structured JSON configuration; the JSON configuration is used to represent the logical topological relationship between N data sources and the target data.

[0156] Based on JSON configuration, generate executable code that is compatible with the preset data protocol.

[0157] In some embodiments, the data registration information further includes the import configuration of N data sources; the import configuration includes the import method of each of the N data sources and the parameter information associated with the import method;

[0158] like Figure 9 As shown, the data processing device based on the large model also includes:

[0159] The data distribution module 910 is used to import data from N data sources into the data pipeline based on the import configuration of N data sources, so as to process the N data sources.

[0160] In some embodiments, the N data sources include a trigger source and M concatenation sources for concatenating with the trigger source, where M is an integer greater than or equal to 1; the trigger source is introduced in a batch scan manner and / or a streaming manner; each of the M concatenation sources is introduced in a manner including at least one of local dictionary introduction, remote key-value introduction, table storage introduction, and database introduction.

[0161] In some embodiments, the data import module 830 is used for:

[0162] Based on the pattern description of the target data, determine the descriptive information of each field in the target data;

[0163] The target data and the descriptive information of each field are stored together in the target database; the descriptive information of each field is used to call the target data based on the language model.

[0164] In some embodiments, such as Figure 9 As shown, the data processing device based on the large model also includes:

[0165] The query input module 920 is used to receive query information input by the target object;

[0166] The indicator term retrieval module 930 is used to retrieve indicator terms that match the query information and their descriptions from the target database based on the query information.

[0167] The model processing module 940 is used to utilize the first language model to obtain the return results corresponding to the query information based on the query information, indicator words, and explanatory information of the indicator words, and to display the return results to the target object.

[0168] In some embodiments, the model processing module 940 is used for:

[0169] Based on the query information, indicator terms, and descriptions of the indicator terms, the input information is obtained;

[0170] Input information into the first language model to obtain multiple subtasks output by the first language model and the execution order among the multiple subtasks;

[0171] Based on the execution order, multiple subtasks are executed, and the returned results are obtained.

[0172] In some embodiments, the input information is also obtained based on description information of a plurality of preset tool interfaces;

[0173] Model processing module 940 is used for:

[0174] Based on the execution order, the tool interface corresponding to each of the multiple subtasks is called to execute the multiple subtasks and obtain the return results.

[0175] In some embodiments, the model processing module 940 is further configured to:

[0176] Based on the query information, the chart style of the returned results is determined using a second language model;

[0177] The target chart is generated based on the chart style and the returned results using a third-language model.

[0178] Display the target chart to the target audience.

[0179] The specific functions and examples of each module and submodule of the apparatus in this disclosure can be found in the relevant descriptions of the corresponding steps in the above method embodiments, and will not be repeated here.

[0180] The acquisition, storage, and application of user personal information involved in the technical solution disclosed herein comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0181] According to embodiments of this disclosure, this disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0182] Figure 10 A schematic block diagram of an example electronic device 1000 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0183] like Figure 10 As shown, device 1000 includes a computing unit 1001, which can perform various appropriate actions and processes according to a computer program stored in read-only memory (ROM) 1002 or a computer program loaded from storage unit 1008 into random access memory (RAM) 1003. The RAM 1003 may also store various programs and data required for the operation of device 1000. The computing unit 1001, ROM 1002, and RAM 1003 are interconnected via bus 1004. Input / output (I / O) interface 1005 is also connected to bus 1004.

[0184] Multiple components in device 1000 are connected to I / O interface 1005, including: input unit 1006, such as keyboard, mouse, etc.; output unit 1007, such as various types of monitors, speakers, etc.; storage unit 1008, such as disk, optical disk, etc.; and communication unit 1009, such as network card, modem, wireless transceiver, etc. Communication unit 1009 allows device 1000 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0185] The computing unit 1001 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 1001 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 1001 performs the various methods and processes described above, such as large model-based data processing methods. For example, in some embodiments, the large model-based data processing method may be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 1008. In some embodiments, part or all of the computer program may be loaded and / or installed on device 1000 via ROM 1002 and / or communication unit 1009. When the computer program is loaded into RAM 1003 and executed by the computing unit 1001, one or more steps of the large model-based data processing method described above may be performed. Alternatively, in other embodiments, the computing unit 1001 may be configured to perform a large model-based data processing method by any other suitable means (e.g., by means of firmware).

[0186] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0187] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0188] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0189] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0190] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with embodiments of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0191] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact via communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, servers in distributed systems, or servers incorporating blockchain technology.

[0192] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0193] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A data processing method based on a large model, comprising: Receive data registration information; wherein, the data registration information includes processing rule information for N data sources and a pattern description of the target data; N is an integer greater than or equal to 1; Based on the processing rule information and the mode description, generate execution code that is compatible with the preset data protocol; Based on the execution code, the N data sources are processed to obtain the target data, and based on the pattern description of the target data, the descriptive information of each field in the target data is determined; The target data and the descriptive information of each field are associated and stored in the target database; wherein, the descriptive information of each field is used to call the target data based on the language model; Receive query information input by the target object; Based on the query information, retrieve the index terms that match the query information and the explanatory information of the index terms from the target database; Using a first language model, the system obtains the return results corresponding to the query information based on the query information, the indicator words, and the description information of the indicator words, and displays the return results to the target object.

2. The method according to claim 1, wherein, The processing rule information includes Structured Query Language (SQL) statements, which are used to describe the concatenation rules of the N data sources; The step of generating execution code adapted to a preset data protocol based on the processing rule information and the mode description includes: The SQL statement is parsed using a syntax tree to obtain a structured JSON configuration; wherein the JSON configuration is used to represent the logical topological relationship between the N data sources and the target data; Based on the JSON configuration, generate executable code that is compatible with the preset data protocol.

3. The method according to claim 1 or 2, wherein, The data registration information also includes the import configuration of the N data sources; the import configuration includes the import method of each of the N data sources and the parameter information associated with the import method; The method further includes: Based on the configuration of the N data sources, the data from the N data sources is introduced into the data pipeline for processing.

4. The method according to claim 3, wherein, The N data sources include trigger sources and M concatenation sources for concatenating with the trigger sources, where M is an integer greater than or equal to 1; the trigger sources are introduced in batch scanning and / or streaming methods; each of the M concatenation sources is introduced in at least one of the following methods: local dictionary introduction, remote key-value introduction, table storage introduction, and database introduction.

5. The method according to claim 1 or 2, wherein, The step of using a first language model to obtain the return result corresponding to the query information based on the query information, the indicator terms, and the explanatory information of the indicator terms includes: Based on the query information, the indicator terms, and the description information of the indicator terms, the input information is obtained; The input information is input into the first language model to obtain multiple subtasks output by the first language model and the execution order among the multiple subtasks; The multiple subtasks are executed according to the execution order to obtain the return result.

6. The method according to claim 5, wherein, The input information is also obtained based on the description information of multiple preset tool interfaces; The step of executing the multiple subtasks according to the execution order to obtain the return result includes: According to the execution order, the tool interface corresponding to each of the multiple subtasks is called to execute the multiple subtasks and obtain the return result.

7. The method according to claim 1 or 2, wherein, The step of displaying the returned result to the target object includes: The chart style of the returned results is determined based on the query information using a second language model; The target chart is generated using a third-language model based on the chart style and the returned results; The target chart is displayed to the target object.

8. A data processing device based on a large model, comprising: The registration module is used to receive data registration information; wherein, the data registration information includes processing rule information for N data sources and a pattern description of the target data; N is an integer greater than or equal to 1; The code generation module is used to generate executable code that is compatible with the preset data protocol based on the processing rule information and the mode description. The data import module is used to process the N data sources based on the execution code to obtain the target data, and to determine the descriptive information of each field in the target data based on the pattern description of the target data. The target data and the descriptive information of each field are associated and stored in the target database. The descriptive information of each field is used to call the target data based on the language model. The query input module is used to receive query information input by the target object; The indicator term retrieval module is used to retrieve, based on the query information, indicator terms that match the query information and explanatory information of the indicator terms from the target database; The model processing module is used to utilize a first language model to obtain the return results corresponding to the query information based on the query information, the indicator words, and the explanatory information of the indicator words, and to display the return results to the target object.

9. The apparatus according to claim 8, wherein, The processing rule information includes Structured Query Language (SQL) statements, which are used to describe the concatenation rules of the N data sources; The code generation module is used for: The SQL statement is parsed using a syntax tree to obtain a structured JSON configuration; wherein the JSON configuration is used to represent the logical topological relationship between the N data sources and the target data; Based on the JSON configuration, generate executable code that is compatible with the preset data protocol.

10. The apparatus according to claim 8 or 9, wherein, The data registration information also includes the import configuration of the N data sources; the import configuration includes the import method of each of the N data sources and the parameter information associated with the import method; The device further includes: The data distribution module is used to introduce data from the N data sources into the data pipeline based on the import configuration of the N data sources, so as to process the N data sources.

11. The apparatus according to claim 10, wherein, The N data sources include trigger sources and M concatenation sources for concatenating with the trigger sources, where M is an integer greater than or equal to 1; the trigger sources are introduced in batch scanning and / or streaming methods; each of the M concatenation sources is introduced in at least one of the following methods: local dictionary introduction, remote key-value introduction, table storage introduction, and database introduction.

12. The apparatus according to claim 8 or 9, wherein, The model processing module is used for: Based on the query information, the indicator terms, and the description information of the indicator terms, the input information is obtained; The input information is input into the first language model to obtain multiple subtasks output by the first language model and the execution order among the multiple subtasks; The multiple subtasks are executed according to the execution order to obtain the return result.

13. The apparatus according to claim 12, wherein, The input information is also obtained based on the description information of multiple preset tool interfaces; The model processing module is used for: According to the execution order, the tool interface corresponding to each of the multiple subtasks is called to execute the multiple subtasks and obtain the return result.

14. The apparatus according to claim 8 or 9, wherein, The model processing module is also used for: The chart style of the returned results is determined based on the query information using a second language model; The target chart is generated using a third-language model based on the chart style and the returned results; The target chart is displayed to the target object.

15. An electronic device comprising: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-7.

16. A non-transitory computer-readable storage medium storing computer instructions, wherein, The computer instructions are used to cause the computer to perform the method according to any one of claims 1-7.

17. A computer program product comprising a computer program that, when executed by a processor, implements the method according to any one of claims 1-7.

Images