Data management method and device, electronic equipment and computer program product

Abstract

The invention discloses a data management method, a data management device, electronic equipment and a computer program product. The method comprises the steps that at least two corpora of a target project are constructed, and different corpora are constructed based on different data sources; respectively constructing a first database corresponding to each corpus through a large language model; all the first databases are subjected to data analysis, a second database is constructed according to an analysis result, the data analysis comprises conflict analysis and coincidence degree analysis, and the structures of the second database and all the first databases are kept consistent. According to the scheme, the reliability and accuracy of the database are improved, so that the database can be applied to scenes with high reliability requirements.

Description

Technical Field

[0001] This application belongs to the field of database technology, and in particular relates to a data management method, a data management device, an electronic device, and a computer program product. Background Technology

[0002] With the widespread application of Large Language Models (LLMs), knowledge extraction and database construction from massive corpora have become common techniques. Current technologies typically extract structured information from datasets from single or limited sources to construct corresponding databases or knowledge graphs, supporting subsequent database applications such as information retrieval, document generation, and intelligent question answering. However, these methods often rely on a one-way data generation process, meaning the database content depends primarily on the initial data sources. Due to the difficulty in ensuring the reliability of these data sources, confusion and / or errors can easily occur during database application, resulting in low database universality and making it difficult to apply to scenarios with high reliability requirements. Summary of the Invention

[0003] This application provides a data management method, a data management device, an electronic device, and a computer program product, which can improve the reliability and accuracy of the database, enabling the database to be applied to scenarios with high reliability requirements.

[0004] Firstly, this application provides a data management method, including: Construct at least two corpora for the target project, with each corpus built based on a different data source; The first database corresponding to each corpus is constructed using a large language model; Data analysis is performed on all first databases to construct a second database based on the analysis results. The data analysis includes conflict analysis and overlap analysis. The structure of the second database is consistent with that of each first database.

[0005] Secondly, this application provides a data management device, comprising: The first construction module is used to build at least two corpora for the target project, wherein the different corpora are built based on different data sources; The second construction module is used to construct the first database corresponding to each corpus using the large language model; The third construction module is used to perform data analysis on all the first databases in order to build the second database based on the analysis results. The data analysis includes conflict analysis and overlap analysis. The structure of the second database and each of the first databases is consistent.

[0006] Thirdly, this application provides an electronic device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the steps of the method described in the first aspect.

[0007] Fourthly, this application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the method described in the first aspect above.

[0008] Fifthly, this application provides a computer program product comprising a computer program that, when executed by one or more processors, implements the steps of the method described in the first aspect.

[0009] The advantages of this application compared to existing technologies are as follows: This application constructs at least two corpora for the target project, each based on different data sources, and uses a large language model to construct a first database corresponding to each corpus, enabling the content stored in the first database to reflect the differences between data from different sources. Based on this, conflict and overlap analyses are performed on all first databases, and a second database is constructed based on the analysis results. This second database maintains a consistent structure with each of the first databases, achieving dynamic fusion and consistency optimization of multi-source data under a unified structure. Therefore, this application can improve the accuracy and reliability of database content under multi-source conditions, improve the problem of easily distorted database results in existing one-way data generation processes, and enable the database to be applied to scenarios with high reliability requirements.

[0010] It is understood that the beneficial effects of the second to fifth aspects mentioned above can be found in the relevant descriptions in the first aspect mentioned above, and will not be repeated here. Attached Figure Description

[0011] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0012] Figure 1 This is a schematic diagram illustrating the implementation flow of the data management method provided in the embodiments of this application; Figure 2 This is an example architecture diagram of the data management system provided in the embodiments of this application; Figure 3 This is a structural block diagram of the data management device provided in the embodiments of this application; Figure 4 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0013] The embodiments of the technical solution of this application will now be described in detail with reference to the accompanying drawings. These embodiments are only used to more clearly illustrate the technical solution of this application and are therefore merely examples, and should not be used to limit the scope of protection of this application.

[0014] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains; the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the application; the terms “comprising” and “having”, and any variations thereof, in the specification, claims, and foregoing description of the drawings are intended to cover non-exclusive inclusion.

[0015] In the description of the embodiments of this application, technical terms such as "first" and "second" are used only to distinguish different objects and should not be construed as indicating or implying relative importance or implicitly indicating the number, specific order, or primary and secondary relationship of the indicated technical features.

[0016] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0017] In the description of the embodiments in this application, the term "and / or" is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Additionally, the character " / " in this document generally indicates that the preceding and following related objects have an "or" relationship.

[0018] In the description of the embodiments of this application, the term "multiple" refers to two or more (including two), unless otherwise expressly and specifically defined.

[0019] Projects typically involve process nodes such as product initiation, hardware development, software development, testing, deployment, and maintenance. For the same product model, different process nodes focus on different details and use different text files. Furthermore, as the project progresses, the design and development of the same function often generates a large number of documents. Taking the PoE (Power over Ethernet) function of a switch development project as an example, the product initiation process node often focuses on the number of PoE ports, the total power supply, the maximum supported port power, whether Fast Persistent PoE is supported, and whether Fast Persistent PoE has power limitations. The hardware development process node focuses on whether EEPROM is needed, and if so, its selection, and the selection of the PSE chip. However, each process node usually only has its own internal data, which is obtained from multiple channels. When obtaining information, it is often necessary to query documents provided by the responsible persons of other process nodes. Moreover, errors in the same function are often introduced during the query process due to various reasons, resulting in wasted design time or resources. To address this issue, this application proposes a data management method, data management device, electronic device, and computer program product, which can achieve dynamic fusion and consistency optimization of database content, thereby improving the reliability and accuracy of the database. Using the database constructed in this application, each process node of a project can quickly find the data it needs. The following specific embodiments will explain and illustrate the solution proposed in this application.

[0020] This application proposes a data management method. This data management method can be applied to electronic devices, such as servers, that can integrate, deploy, or access databases; the specific type of electronic device is not limited here. Please refer to [link to relevant documentation]. Figure 1 , Figure 1 The implementation flow of the data management method applied to this electronic device is presented, and detailed below: Step 101: Construct at least two corpora for the target project.

[0021] During business development, multiple different projects may be involved. For example, under the business of switch development, there may be N different projects, each corresponding to the development of a specific switch model. In some examples, the project to which each file belongs can be distinguished by extracting tags from each original text file; alternatively, the project to which each file belongs can be distinguished by extracting keywords from each original text file; or, the project to which each original text file belongs can be manually distinguished. This application does not limit the means of distinguishing the project to which the original text file belongs. It is understood that any project with management needs under all business operations of an enterprise can be identified as a target project to apply the data management method proposed in this application.

[0022] Electronic devices can classify and organize raw text files belonging to the target project according to their data sources, thereby constructing corpora corresponding to different data sources. Specifically, the corpus construction process includes: preprocessing the raw text files to obtain key information, including formatting, noise reduction, sentence segmentation, and / or word segmentation; annotating the preprocessed key information, including but not limited to project identifiers, file sources, and / or timestamps; and storing the annotated key information into different corpora according to their data sources, thus constructing different corpora and distinguishing the differences in data sources during subsequent analysis.

[0023] Through this step, the electronic device establishes a corpus from multiple data sources, laying the foundation for subsequent comparison and fusion of multi-source data. This ensures that the final database content reflects information differences from different stages or departments within the same target project. Furthermore, the construction of the corpus guarantees, to a certain extent, the structured and diverse input of data sources, thereby ensuring sufficient data coverage for subsequent database construction.

[0024] Step 102: Construct the first database corresponding to each corpus using the large language model.

[0025] Electronic devices can utilize large language models to extract information and perform structured mapping on each corpus, thereby forming multiple corresponding databases. For ease of distinction, this database can be referred to as the first database. In fact, the first database is a database constructed based on a corpus from a single data source. Its internal structure is organized according to a preset key-value pair format, such as "parameter name (key) - parameter value (value)" and "function item (key) - support status (value)", etc. This application embodiment does not limit the structure of the first database.

[0026] Specifically, during the construction process, the large language model can be used to perform the following operations: extract key information from the corpus (such as functional descriptions, parameter values, and / or attribute relationships); perform semantic analysis and synonym recognition to normalize the content of different expressions; and map the analysis results to a standardized database structure.

[0027] Through this step, electronic devices can convert unstructured text in a corpus into structured data in a database, standardizing text information from different data sources, providing comparability and a unified format for subsequent data analysis, and ensuring that subsequent database analysis can be conducted on the same structure.

[0028] Step 103: Perform data analysis on all first databases to construct a second database based on the analysis results.

[0029] The electronic device can continue to perform unified data analysis on the data in each of the first databases. This data analysis includes two operations: conflict analysis and overlap analysis. Based on the analysis results, the electronic device can generate a comprehensive second database. It can be understood that the first database is at the bottom layer, essentially a single-layer database; the second database is at the top layer, essentially a two-layer database.

[0030] Conflict analysis refers to identifying and labeling discrepancies when records of the same corpus (i.e., the same key-value items) in different first databases are inconsistent. For example, if the first database based on an email corpus records "POE support: No", while the first database based on a design corpus records "POE support: Yes", the system will identify the conflict in conflict analysis and set a discrepancy label in the second database.

[0031] Overlap analysis refers to calculating the degree of consistency between results for the same corpus across different primary databases to determine the reliability of the data. In some examples, statistical methods (such as majority rule or weighted confidence calculation) can be used to determine the final result. For instance, in three primary databases, if two records for a certain function are "supported" and one is "not supported," the overlap analysis result would be "supported," with a high confidence level.

[0032] Based on the analysis results of each of the first databases, the electronic device can generate a comprehensive database, which can be referred to as the second database for easy distinction. It can be understood that the structure of the second database is consistent with that of each of the first databases; however, compared with the first database, the second database adds a conflict identifier field to the original structure. This field is used to record whether there are source conflicts or insufficient overlap of items.

[0033] Through this step, the electronic device integrates multi-source data while maintaining a unified structure, improving the accuracy and confidence of the database results, avoiding bias caused by data from a single source, realizing dynamic integration and consistency judgment of data content, and obtaining a relatively reliable second database.

[0034] It is understandable that there is a close logical dependency between the above steps: First, the multi-source construction of the corpus provides diverse data input for the creation of the first database; second, the establishment of the first database enables the transformation of unstructured text into structured data that can be uniformly analyzed; finally, the construction of the second database achieves result fusion and conflict identification among multiple first databases. Through the above steps, the embodiments of this application realize dynamic updating and consistency optimization of database content, effectively solving the problem of database content confusion and errors caused by unidirectional data flow; furthermore, the second database constructed in the embodiments of this application can be widely used in scenarios requiring the fusion of multiple data sources, such as project management, document generation, and data verification.

[0035] In some embodiments, based on the constructed database, electronic devices can automate file generation. The steps for generating this file may include: A1, Get the template file.

[0036] Electronic devices can first read a template file. A template file refers to a predefined file that marks the content to be filled using placeholders or similar methods. This template file may also contain default values ​​and / or format constraints for the content to be filled. In some examples, the template file may specifically be a text template, a structured template, or a configuration template; a text template may be a Word template or a Markdown template, a structured template may be a JSON template, a YAML template, or an XML template, and a configuration template may be a device configuration script, etc. This application does not limit the specific type of the template file. Through the template file, the output carrier can be standardized, thereby clarifying the location and data type of the content to be filled, providing a consistent interface for subsequent automatic filling based on a second database.

[0037] A2, extract the first keyword from the template file.

[0038] The electronic device can parse the template file, identify and extract all placeholder names. Each placeholder name is a key name corresponding to a placeholder, which can be used to retrieve the corresponding value in the database. For ease of description, this placeholder name is referred to as the first keyword. It can be understood that through this step, generally more than one first keyword can be obtained, thus forming a list of first keywords. As described above, the template file may also contain format constraints for the content to be filled. Therefore, when extracting the first keyword, the electronic device can also record the format constraints corresponding to each first keyword, such as numerical units and rounding rules. This embodiment of the application does not limit this.

[0039] In some examples, electronic devices may use template parsers, regular expressions, or lightweight template engines for parsing; however, this application does not limit the parsing methods used.

[0040] By extracting the first keyword, electronic devices can map unstructured placeholders in template files to searchable keys in the database, thereby establishing a mapping relationship between template files and database keys and ensuring consistency between retrieval and data filling.

[0041] A3, based on each primary keyword, searches the second database to obtain the first search result.

[0042] For each primary keyword, the electronic device can perform a search in the second database using that primary keyword as the key, and the resulting search result can be recorded as the primary search result. In some examples, the primary search result contains the target value found; in addition, the primary search result may also contain: confidence level, timestamp (if there are time-series records) and / or difference annotation information (if the corresponding key-value pair has difference annotations), etc., which are not limited here.

[0043] Thus, electronic devices can obtain reference data for filling from a first database that has been integrated and has traceable information, while also retaining meta-information for subsequent review or human intervention.

[0044] A4, based on the first search result, fill in the content to be filled for each first keyword in the template file.

[0045] The electronic device can fill the target value from the first search result into the corresponding position in the template file according to the format constraints corresponding to the first keyword. After filling, the electronic device can also perform format verification operations, including but not limited to: unit consistency verification, value range verification, and parameter interdependence verification, etc., which are not limited in this embodiment.

[0046] Based on this, if the first search result also carries a confidence level that is lower than a preset confidence threshold, and / or if the difference annotation information indicates that the corresponding key-value pair has a difference annotation, the electronic device may also trigger a specific strategy. In some examples, the specific strategy may include, but is not limited to: filling in a default value in the content to be filled corresponding to the first keyword; generating a preset annotation message to indicate that the content to be filled needs to be manually reviewed; calling the source tracing function to retrieve more context from each original text file, etc.

[0047] At this point, the electronic device can automatically generate the document, and the process incorporates confidence level and difference annotation to ensure the accuracy of the results.

[0048] In some embodiments, based on the constructed database, electronic devices can automate document verification. The document verification steps may include: B1, retrieve the document to be verified.

[0049] The file to be verified refers to a file that already exists and the user expects to compare its data status to ensure consistency with database records. Electronic devices can obtain the file to be verified (i.e., the file to be verified) from the project file library or the user's upload location. Similar to template files, this file to be verified can also support various formats, including but not limited to text, tables, and configuration files, etc., which are not limited here.

[0050] B2, extract the second keyword from the document to be verified.

[0051] The electronic device can extract information from the document to be verified, identifying the text item in the document that corresponds to the key in the second database. This text item can be denoted as the second keyword. In other words, the second keyword refers to the field identifier or semantic item extracted from the document to be verified that corresponds to the key in the second database. Through this extraction operation, the electronic device can convert unstructured or semi-structured information in the document to be verified into comparable structured data, facilitating the automation of subsequent verification operations.

[0052] In some examples, electronic devices can extract information from the documents to be checked using template parsers, regular expressions, table field mapping, or large language models to obtain second keywords. In addition, during the extraction process, electronic devices can also record contextual text and location (including but not limited to page numbers and / or rows and columns).

[0053] B3. Search the second database based on each secondary keyword to obtain the second search results.

[0054] For each second keyword, the electronic device can perform a search in the second database using that second keyword as the key. The resulting search is referred to as the second search result. Similar to the first search result, the second search result contains the found target value. In addition, the second search result may also include: confidence level, timestamp (if there are time-series records), and / or difference annotation information (if the corresponding key-value pair has difference annotations), etc., which are not limited here. It should be noted that if the second search result contains multiple target values, each corresponding to a different time-series record, the latest value or the value at a specified time point can be used as the final target value based on actual needs. It can be understood that the second search result can serve as a reference for subsequent verification.

[0055] B4. Based on the second search results, verify the content to be confirmed corresponding to each second keyword in the document to be checked.

[0056] Each secondary keyword in its corresponding file to be verified is accompanied by a corresponding text value, which is the content to be confirmed in the file corresponding to that secondary keyword. The electronic device can compare the content to be confirmed for each secondary keyword with the target value carried by the corresponding secondary search result.

[0057] If the comparison is successful, the content to be confirmed and its corresponding second keyword can be marked as successful.

[0058] Conversely, if the comparison is inconsistent, the inconsistent content to be confirmed can be recorded as erroneous content. The electronic device can perform source tracing analysis in various first databases based on the second keyword corresponding to the erroneous content to confirm the cause of the error. In addition, the electronic device can record a verification event, which includes the following: erroneous content, the corresponding target value, the confidence level of the target value, the source information of the target value, the timestamp of the target value, and the verification time. Of course, this verification event can also include more or less content; this is not limited here.

[0059] Specifically, the source tracing analysis process can be as follows: using the second keyword corresponding to the erroneous content as the search key, traverse all first databases, and retrieve the historical value, timestamp, source document ID, and context fragment for that key in each first database to obtain the source tracing results. In some examples, these source tracing results can be summarized in chronological order or by source priority.

[0060] Through source tracing analysis, electronic devices can trace the occurrence time, modification history, and original source (i.e., the original text file) of data related to the erroneous content in the file to be verified from different sources (i.e., the primary database). After merging the source tracing results from various primary databases, the electronic device can preliminarily locate the time and source of the erroneous content based on factors such as timeline and change frequency, thereby determining the cause of the error, such as outdated data, entry errors, or data omissions. If the electronic device cannot determine the cause of the error, it can generate a manual review task, attaching the obtained source tracing results as evidence for users to make manual judgments and corrections.

[0061] In some embodiments, in the event of discrepancies, the electronic device may first determine the type of difference between the erroneous content and the target value. If it is a formal difference (including but not limited to format differences and capitalization differences), the erroneous content can be automatically corrected; otherwise, if it is a substantive difference (including but not limited to numerical deviations and semantic inconsistencies), the relevant process of source tracing analysis proposed above can be triggered, which will not be elaborated here.

[0062] In some embodiments, if a source tracing analysis is performed at the current point in time during the management cycle of a target project, the results of this analysis can be used to correct erroneous data in the database when building / maintaining the database (including the first and second databases) at the next point in time, thereby further ensuring the accuracy of the database. It can be understood that through the above operations, the reliability and accuracy of the database can be gradually improved, especially during project management of the target project; the closer to the end of the management cycle, the higher the reliability and accuracy of the data.

[0063] In this way, electronic devices can complete automated document verification, reducing the cost of manual inspection and enabling rapid error correction, tracking and correction of data.

[0064] In some embodiments, to ensure the reliability of the second database and enable it to provide a consistent reference baseline for template filling, document verification, and traceability, step 103 may specifically include: C1, for each key-value pair, iterates through all the first databases to determine the value corresponding to the key in each first database.

[0065] As described earlier, the first and second databases share the same structure, meaning they are all built based on the same structural template. The electronic device can read this structural template to obtain the key set. Specifically, the electronic device can retrieve the key set for any key k in the key set from the first database D. i D is read from i [k], where D iLet D represent the i-th first database. i [k] represents the value corresponding to the i-th key k in the first database. For example, if D appears... i If [k] does not exist, it can be recorded as null and the source can be marked as missing. Of course, during the traversal, the electronic device can also collect metadata of each value, including but not limited to file source, timestamp, and ID of the original text file, which is not limited here.

[0066] C2 checks whether the values ​​corresponding to the keys in each of the first databases are consistent.

[0067] For key k, the electronic device can obtain multiple corresponding values, namely D1[k], D... 2[ k]until D N [k], where N is the total number of values ​​in the first database. The electronic device can compare these N values ​​to determine whether they are consistent. It can be understood that the electronic device's judgment here is specifically to determine whether there are different values ​​(i.e., whether at least two values ​​are different), rather than to determine whether these values ​​are inconsistent pairwise.

[0068] C3, when the values ​​corresponding to the key differ in at least two first databases, determines the target value based on a preset weighting strategy.

[0069] In D1[k], D 2[ k]……D N If at least two values ​​differ in [k], then there is a data conflict between different data sources for key k. Considering that the second database can only store one specific value, the electronic device can be pre-set with a weighting strategy. Specifically, this weighting strategy can be a static weighting strategy. For example, the weight w1 of the first database D1 is set to a preset first weight value, the weight w2 of the first database D2 is set to a preset second weight value, the weight w3 of the first database D3 is set to a preset third weight value, and the weight w4 of the first database D4 is set. Alternatively, this weighting strategy can also be a dynamic weighting strategy. For example, the weights of each first database can be adjusted based on the historical source tracing error rate, or the weights of each first database can be adjusted based on the reliability of the source of the first database. This application embodiment does not limit this.

[0070] Let the values ​​that differ be denoted as V. j Based on the weighting strategy, the weight score of this value can be calculated using the following formula: score(V j ) = Σw i (if D i [k] ==V j ) Electronic devices can determine the target value as the value with the highest weighted score.

[0071] C4 populates the target value into the key-value pairs in the second database.

[0072] The electronic device can write the selected target value to the corresponding key in the second database. Furthermore, the electronic device can record metadata about each value involved in the fusion decision-making process; this metadata can be used for subsequent source tracing. In some examples, this metadata includes the source of each value involved in the calculation, the weight of each value, and the filling time, etc., but this is not limited here.

[0073] In addition, since the target value is selected when data differences exist, the electronic device can also add difference annotations to the key-value pairs corresponding to the target value in the second database. These difference annotations indicate that the key-value pairs differ across different first databases.

[0074] It is understandable that, based on the difference annotations in the key-value pairs, electronic devices can trigger various subsequent operations, including but not limited to: periodically performing manual review of the key-value pairs, using them as input for dynamic weight adjustment in the first database, or presenting them to reviewers as "confirmation required" reminders when generating documents. This can further improve the data quality of the second database and make weight and data conflict handling part of closed-loop optimization, while also facilitating verification or querying by electronic devices in subsequent traceability.

[0075] In some embodiments, to improve the efficiency and accuracy of building the first database, step 102 may include: D1 extracts data from each corpus using a trained large language model and a pre-set database.

[0076] For each corpus, the electronic device can first determine a suitable base model. This base model can be open-source or self-developed by the enterprise; no specific limitations are made here. To improve model performance, this base model can be fine-tuned or optimized using a sample set from the target project's domain. This sample set may include, but is not limited to, historical project documents, industry standards, labeled key-value pair training sets, and manually reviewed examples, etc., which will not be elaborated upon here. Ultimately, a trained large language model is obtained. This large language model is fine-tuned or designed using specific hints within the target project's domain, enabling it to better understand the domain's terminology and thus better perform specified feature extraction tasks, including but not limited to synonym conversion, keyword context extraction, and semantic analysis.

[0077] Synonym conversion refers to the ability of a large language model to map different expressions to a unified vocabulary.

[0078] Keyword context extraction refers to extracting the attributes associated with a keyword from the text in which the keyword is found and establishing the key-value relationship; that is, extracting numerical or state information related to the semantics of the keyword from the context of the keyword as key-value pairs to be filled.

[0079] Semantic analysis refers to inferring actual parameters or restrictions from implicit statements or complex sentences and storing them as structured annotations or restriction rules in a first database.

[0080] Of course, electronic devices can also perform manual verification of the extraction results of large language models, and statistically analyze the extraction error rate based on the verification results. Error samples can be fed back into the training set to fine-tune the large language model or update the prompting strategy, thereby improving the extraction effect of the large language model.

[0081] D2, construct the first database corresponding to the corpus based on the extraction results.

[0082] Electronic devices can write extracted results, such as key-value pairs output by a large language model, into the corresponding first database. It's important to note that the correspondence between the corpus and the first database must be maintained during the writing process. For example, if the extracted results originate from an email corpus, then those results should be written to the first database corresponding to that email corpus. In practical applications, the same key may be extracted multiple times within the same corpus. In such cases, the electronic device can record the time series data in the corresponding first database to reflect the temporal characteristics.

[0083] In this way, electronic devices can establish a structured data layer (i.e., multiple first databases) based on different data sources, which provides detailed original records and chains of evidence for data merging in the second database and subsequent possible tracing operations.

[0084] Please see Figure 2 , Figure 2 An architectural example of the data management system proposed in the embodiments of this application is given. For example... Figure 2 As shown, an email corpus can be formed based on email correspondence related to the target project; a technical corpus can be formed based on technical documents related to the target project; and a design corpus can be formed based on the preliminary design documents of the target project. This application does not limit the possible data sources or corresponding corpora. Based on this, the electronic device can further generate a corresponding first database based on each corpus. Finally, through data analysis of the first database and data fusion, a second database integrating the contents of each first database is obtained. This second database can be used for the automated generation or verification of documents; and, by combining the first and second databases, traceability operations can be performed when necessary.

[0085] As can be seen from the above, this embodiment constructs at least two corpora for the target project, each based on different data sources, and uses a large language model to construct a first database corresponding to each corpus. This ensures that the content stored in the first database reflects the differences between data from different sources. Based on this, conflict and overlap analyses are performed on all first databases, and a second database is constructed according to the analysis results. This second database maintains a consistent structure with all first databases, achieving dynamic fusion and consistency optimization of multi-source data under a unified structure. Therefore, this embodiment can improve the accuracy and reliability of database content under multi-source conditions, addressing the problem of easily distorted database results in existing one-way data generation processes, and enabling the database to be applied to scenarios with high reliability requirements.

[0086] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0087] Corresponding to the data management method provided above, this application also provides a data management device. Please refer to... Figure 3 The data management device 3 in this embodiment includes: The first construction module 301 is used to construct at least two corpora for the target project, wherein the different corpora are constructed based on different data sources; The second construction module 302 is used to construct the first database corresponding to each corpus through the large language model; The third construction module 303 is used to perform data analysis on all the first databases in order to construct the second database based on the analysis results. The data analysis includes conflict analysis and overlap analysis. The structure of the second database and each of the first databases is consistent.

[0088] In some embodiments, the data management device 3 further includes: The first acquisition module is used to acquire template files; The first extraction module is used to extract the first keyword from the template file; The first search module is used to search the second database based on each first keyword to obtain the first search results; The fill module is used to fill in the content to be filled for each first keyword in the template file based on the first search result.

[0089] In some embodiments, the data management device 3 further includes: The second acquisition module is used to acquire the file to be verified. The second extraction module is used to extract the second keyword from the document to be verified. The second search module is used to search the second database based on each second keyword and obtain the second search results. The verification module is used to verify the content to be confirmed corresponding to each second keyword in the document to be verified based on the second search results.

[0090] In some embodiments, the data management device 3 further includes: The source tracing module is used to perform source tracing analysis in each of the first databases based on the second keyword corresponding to the erroneous content when the verification results indicate that there is erroneous content, in order to confirm the cause of the error.

[0091] In some embodiments, the third building module 303 includes: The traversal unit is used to traverse all first databases for each key-value pair to determine the value corresponding to the key in each first database. The detection unit is used to detect whether the values ​​corresponding to the keys in each of the first databases are consistent; The determining unit is used to determine the target value based on a preset weighting strategy when there are differences in the values ​​corresponding to the key in at least two first databases. The filler unit is used to fill the target value into key-value pairs in the second database.

[0092] In some embodiments, the third building module 303 further includes: The annotation module is used to perform difference annotation on the key-value pairs corresponding to the target value in the second database. The difference annotation is used to indicate that the key-value pairs are different in different first databases.

[0093] In some embodiments, the second building module 302 includes: The extraction unit is used to extract data from each corpus using a trained large language model and a pre-set database. The large language model is trained based on the domain of the target project. The data extraction includes at least one of the following: synonym conversion, keyword context extraction, and semantic analysis. The construction unit is used to build the first database corresponding to the corpus based on the extraction results.

[0094] As can be seen from the above, this embodiment constructs at least two corpora for the target project, each based on different data sources, and uses a large language model to construct a first database corresponding to each corpus. This ensures that the content stored in the first database reflects the differences between data from different sources. Based on this, conflict and overlap analyses are performed on all first databases, and a second database is constructed according to the analysis results. This second database maintains a consistent structure with all first databases, achieving dynamic fusion and consistency optimization of multi-source data under a unified structure. Therefore, this embodiment can improve the accuracy and reliability of database content under multi-source conditions, addressing the problem of easily distorted database results in existing one-way data generation processes, and enabling the database to be applied to scenarios with high reliability requirements.

[0095] Corresponding to the data management method provided above, this application also provides an electronic device. Please refer to... Figure 4 The electronic device 4 in this application embodiment includes: a memory 401, and one or more processors 402. Figure 4 (Only one is shown in the image) and a computer program stored in memory 401 and executable on the processor. Specifically, the processor 402 performs the following steps by running the aforementioned computer program stored in memory 401: Construct at least two corpora for the target project, with each corpus built based on a different data source; The first database corresponding to each corpus is constructed using a large language model; Data analysis is performed on all first databases to construct a second database based on the analysis results. The data analysis includes conflict analysis and overlap analysis. The structure of the second database is consistent with that of each first database.

[0096] Assuming the above is the first possible implementation, in the second possible implementation provided based on the first possible implementation, after performing data analysis on all the first databases to construct the second database based on the analysis results, the processor 402 further performs the following steps when running the computer program stored in the memory 401: Get the template file; Extract the first keyword from the template file; Based on each primary keyword, a search is performed in the second database to obtain the primary search results; Based on the first search result, the content to be filled in corresponding to each first keyword in the template file is filled in.

[0097] In a third possible implementation based on the first possible implementation described above, after performing data analysis on all the first databases to construct a second database based on the analysis results, the processor 402 further performs the following steps when running the computer program stored in the memory 401: Obtain the documents to be verified; Extract the second keyword from the document to be verified; The second search results are obtained by searching the second database based on each of the secondary keywords. Based on the second search results, the content to be confirmed corresponding to each second keyword in the document to be checked is verified.

[0098] In the fourth possible implementation provided based on the third possible implementation described above, after verifying the content to be confirmed corresponding to each second keyword of the document to be verified based on the second search result, the processor 402 further performs the following steps when running the computer program stored in the memory 401: If the verification results indicate that there is erroneous content, a source tracing analysis is performed in each of the first databases based on the second keyword corresponding to the erroneous content to confirm the cause of the error.

[0099] In a fifth possible implementation provided based on the first, second, third, or fourth possible implementations described above, data analysis is performed on all first databases to construct a second database based on the analysis results, including: For each key-value pair, iterate through all the first databases to determine the value corresponding to the key in each first database; Check whether the values ​​corresponding to the key in each of the first databases are consistent; If the values ​​corresponding to the key differ in at least two first databases, the target value is determined based on a preset weighting strategy. The target value is populated into the key-value pairs in the second database.

[0100] In the sixth possible implementation provided based on the fifth possible implementation described above, after determining the target value based on a preset weighting strategy, the processor 402 further performs the following steps when running the computer program stored in the memory 401: In the second database, the key-value pairs corresponding to the target value are marked with differences. The differences are used to indicate that the key-value pairs differ in different first databases.

[0101] In a seventh possible implementation provided based on the first, second, third, or fourth possible implementations described above, a first database corresponding to each corpus is constructed using a large language model, including: For each corpus, data is extracted from the corpus using a trained large language model and a pre-defined database. The large language model is trained based on the domain of the target project. Data extraction includes at least one of the following: synonym conversion, keyword context extraction, and semantic analysis. Based on the extraction results, a first database corresponding to the corpus is constructed.

[0102] It should be understood that, in the embodiments of this application, the processor 402 may be a central processing unit (CPU), but it may also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor.

[0103] Memory 401 may include read-only memory and random access memory, and provides instructions and data to processor 402. Some or all of memory 401 may also include non-volatile random access memory. For example, memory 401 may also store device type information.

[0104] As can be seen from the above, this embodiment constructs at least two corpora for the target project, each based on different data sources, and uses a large language model to construct a first database corresponding to each corpus. This ensures that the content stored in the first database reflects the differences between data from different sources. Based on this, conflict and overlap analyses are performed on all first databases, and a second database is constructed according to the analysis results. This second database maintains a consistent structure with all first databases, achieving dynamic fusion and consistency optimization of multi-source data under a unified structure. Therefore, this embodiment can improve the accuracy and reliability of database content under multi-source conditions, addressing the problem of easily distorted database results in existing one-way data generation processes, and enabling the database to be applied to scenarios with high reliability requirements.

[0105] This application also provides a computer program product that, when run on an electronic device, enables the electronic device to perform the steps described in the various method embodiments above.

[0106] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the above device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0107] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0108] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of external device software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0109] In the embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the system embodiments described above are merely illustrative. For instance, the division of modules or units described above is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection between devices or units through some interfaces, and may be electrical, mechanical, or other forms.

[0110] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0111] If the integrated units described above are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments can also be implemented by a computer program instructing associated hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable storage medium can include: any entity or device capable of carrying the computer program code, a recording medium, a USB flash drive, a portable hard drive, a magnetic disk, an optical disk, a computer-readable storage device, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium, etc. It should be noted that the contents of the aforementioned computer-readable storage media may be appropriately added to or subtracted from the contents according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable storage media may not include electrical carrier signals and telecommunication signals.

[0112] The above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A data management method characterized by, include: Construct at least two corpora for the target project, wherein the different corpora are constructed based on different data sources; A first database corresponding to each of the aforementioned corpora is constructed using a large language model; Data analysis is performed on all the first databases to construct a second database based on the analysis results. The data analysis includes conflict analysis and overlap analysis. The structure of the second database and each of the first databases is consistent.

2. The data management method of claim 1, wherein, After performing data analysis on all the first databases to construct a second database based on the analysis results, the data management method further includes: Get the template file; Extract the first keyword from the template file; Based on each of the first keywords, a search is performed in the second database to obtain the first search result; Based on the first search result, the content to be filled in corresponding to each of the first keywords in the template file is filled in.

3. The data management method of claim 1, wherein, After performing data analysis on all the first databases to construct a second database based on the analysis results, the data management method further includes: Obtain the documents to be verified; Extract the second keyword from the document to be verified; Based on each of the second keywords, a search is performed in the second database to obtain the second search results; Based on the second search result, the content to be confirmed corresponding to each of the second keywords in the document to be verified is checked.

4. The data management method of claim 3, wherein, After verifying the content to be confirmed corresponding to each of the second keywords in the document to be verified based on the second search result, the data management method further includes: If the verification results indicate that there is erroneous content, a source tracing analysis is performed in each of the first databases based on the second keyword corresponding to the erroneous content to confirm the cause of the error.

5. The data management method according to any one of claims 1 to 4, wherein, The step of performing data analysis on all of the first databases to construct a second database based on the analysis results includes: For each key-value pair, iterate through all the first databases to determine the value corresponding to the key in each of the first databases; Detect whether the values ​​corresponding to the key in each of the first databases are consistent; If the values ​​corresponding to the key differ in at least two of the first databases, a target value is determined based on a preset weighting strategy. The target value is populated into the key-value pair in the second database.

6. The data management method of claim 5, wherein, After determining the target value based on a preset weighting strategy, the data management method further includes: In the second database, the key-value pairs corresponding to the target value are marked with differences, which are used to indicate that the key-value pairs differ in different first databases.

7. The data management method according to any one of claims 1 to 4, wherein, The construction of a first database corresponding to each of the corpora using a large language model includes: For each corpus, data is extracted from the corpus using a trained large language model and a pre-defined database. The large language model is trained based on the domain of the target project. The data extraction includes at least one of the following: synonym conversion, keyword context extraction, and semantic analysis. Based on the extraction results, a first database corresponding to the corpus is constructed.

8. A data management apparatus characterized by comprising: include: The first construction module is used to construct at least two corpora for the target project, wherein the different corpora are constructed based on different data sources; The second construction module is used to construct a first database corresponding to each of the corpora using a large language model; The third construction module is used to perform data analysis on all the first databases in order to construct a second database based on the analysis results. The data analysis includes conflict analysis and overlap analysis. The structure of the second database and each of the first databases is consistent.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the method as described in any one of claims 1 to 7.

10. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by one or more processors, implements the method as described in any one of claims 1 to 7.

Images