Database construction method, device, electronic device, storage medium and program product

Abstract

The present application provides a method, device, electronic device, storage medium and program product for constructing a database; the method includes: obtaining multiple data to be stored, identifying the association of the multiple data to be stored, and obtaining an identification result; based on the identification result, integrating the target data to be stored that does not have data association among the multiple data to be stored, and obtaining integrated data, wherein the integrated data includes at least one sub-integrated data, and the sub-integrated data includes at least part of the data in each target data to be stored; based on each sub-integrated data, constructing a target database, wherein the target database includes a mapping relationship corresponding to each sub-integrated data, and the mapping relationship uses the data feature of the sub-integrated data as an index, and uses the encrypted data of each sub-integrated data as the data content. Through the present application, the query performance of the database can be improved while effectively improving the data security of the database.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a database construction method, device, electronic device, storage medium and program product. Background Art

[0002] Artificial Intelligence (AI) is a comprehensive discipline that covers a wide range of fields, including both hardware-level and software-level technologies. The basic technologies of artificial intelligence generally include sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, pre-trained model technology, operation / interaction systems, mechatronics, etc. Among them, the pre-trained model is also called a large model or a basic model. After fine-tuning, it can be widely used in downstream tasks in various major directions of artificial intelligence. Artificial intelligence software technology mainly includes computer vision technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0003] In the related art, when constructing a database, the data to be stored or the encrypted data corresponding to the data to be stored is usually directly stored. Directly storing the data to be stored will result in poor data security of the data storage, while storing the encrypted data corresponding to the data to be stored will result in poor data operation performance of the stored encrypted data because the data has been encrypted. Moreover, directly storing multiple data to be stored that do not have data correlation in the database will result in poor query performance of the constructed database. Therefore, the related art cannot effectively improve the query performance of the database while effectively improving the data security of the database. Summary of the invention

[0004] The embodiments of the present application provide a database construction method, device, electronic device, computer-readable storage medium and computer program product, which can effectively improve the data security of the database while improving the query performance of the database.

[0005] The technical solution of the embodiment of the present application is implemented as follows:

[0006] The present application embodiment provides a method for constructing a database, including:

[0007] Acquire multiple data to be stored, perform correlation identification on the multiple data to be stored, and obtain an identification result;

[0008] Based on the recognition result, target data to be stored that do not have data association among the multiple data to be stored are integrated to obtain integrated data, wherein the integrated data includes at least one sub-integrated data, and the sub-integrated data includes at least part of the data in each of the target data to be stored;

[0009] Based on each of the sub-integration data, a target database is constructed, wherein the target database includes mapping relationships corresponding to each of the sub-integration data, wherein the mapping relationships use data features of the sub-integration data as indexes and encrypted data of each of the sub-integration data as data content.

[0010] The present application provides a database construction device, including:

[0011] An identification module is used to obtain a plurality of data to be stored, perform correlation identification on the plurality of data to be stored, and obtain an identification result;

[0012] an integration module, configured to integrate target data to be stored that do not have data association among the plurality of data to be stored based on the identification result, to obtain integrated data, wherein the integrated data includes at least one sub-integrated data, and the sub-integrated data includes at least part of the data in each of the target data to be stored;

[0013] A construction module is used to construct a target database based on each of the sub-integration data. The target database includes a mapping relationship corresponding to each of the sub-integration data. The mapping relationship uses the data features of the sub-integration data as an index and the encrypted data of each of the sub-integration data as data content.

[0014] In the above scheme, the above identification module is also used to obtain the data source of each of the data to be stored; for any two of the multiple data to be stored, when the data sources of the any two data to be stored are the same, it is determined that there is data correlation between the any two data to be stored; when the data sources of the any two data to be stored are different, it is determined that there is no data correlation between the any two data to be stored.

[0015] In the above scheme, when the data to be stored is data in an application, the above identification module is also used to obtain the generation time of each data to be stored in the application; for any two data to be stored among the multiple data to be stored, when the generation times of the any two data to be stored are respectively before and after the startup time of the application, it is determined that there is no data correlation between the any two data to be stored; when the generation times of the any two data to be stored are both before the startup time of the application, it is determined that there is data correlation between the any two data to be stored.

[0016] In the above scheme, the above identification module is also used to perform the following processing for any two data to be stored among the multiple data to be stored: calling the correlation identification model to predict the correlation between the any two data to be stored to obtain the correlation between the any two data to be stored; when the correlation is less than the correlation threshold, determining that there is no data correlation between the any two data to be stored; when the correlation reaches the correlation threshold, determining that there is data correlation between the any two data to be stored.

[0017] In the above scheme, the above-mentioned construction module is also used to merge the to-be-stored data with the data association among the multiple to-be-stored data in response to the identification result indicating that none of the multiple to-be-stored data are the target to-be-stored data, so as to obtain at least one merged data; when the number of the merged data is one, construct a first database based on each to-be-stored data in the merged data; when the number of the merged data is multiple, integrate each merged data to obtain merged integrated data, and construct a second database based on each sub-merged integrated data in the merged integrated data.

[0018] In the above scheme, the above construction module is also used to extract features of each of the data to be stored to obtain data features of each of the data to be stored, and to encrypt each of the data to be stored to obtain encrypted data of each of the data to be stored; based on the data features of the data to be stored and the encrypted data of the data to be stored, the first database is constructed; the first database includes a first mapping relationship corresponding to each of the data to be stored, and the first mapping relationship uses the data features of the data to be stored as an index and the encrypted data of the data to be stored as data content.

[0019] In the above scheme, the above integration module is also used to, in response to the identification result indicating that each of the multiple data to be stored is the target data to be stored, integrate the target data to be stored to obtain the integrated data; in response to the identification result indicating that some of the multiple data to be stored are the target data to be stored, merge the multiple data to be stored that have the data association to obtain at least one merged data; integrate the target data to be stored and the merged data to obtain the integrated data.

[0020] In the above scheme, the data to be stored includes at least one sub-data, and different sub-data in the data to be stored correspond to different data types. The above-mentioned integration module is also used to obtain indication information corresponding to the data type for each data type. When the indication information indicates that sub-data corresponding to the data type exists in at least part of the target data to be stored, the sub-integration data corresponding to the data type is determined based on the sub-data corresponding to the data type; and the integrated data is constructed based on the sub-integration data.

[0021] In the above scheme, the above integrated module is also used to determine the data type as the target data type, and for each target data to be stored, compare the target data type with the data type of each sub-data in the target data to be stored to obtain a first comparison result corresponding to the target data to be stored; when the first comparison result indicates that sub-data of the target data type exists in the target data to be stored, the first comparison result is determined as the target comparison result; when the target comparison result exists, the indication information is determined as sub-data corresponding to the data type exists in at least part of the target data to be stored.

[0022] In the above scheme, the above integration module is also used to, when the number of sub-data corresponding to the data type is one, determine the sub-data corresponding to the data type as the sub-integrated data corresponding to the data type; when the number of sub-data corresponding to the data type is multiple, splice the sub-data corresponding to the data type to obtain the sub-integrated data corresponding to the data type.

[0023] In the above scheme, the above integration module is also used to determine the sub-integration data as the integrated data when the number of the sub-integration data is one; when the number of the sub-integration data is multiple, construct the integrated data based on each of the sub-integration data.

[0024] In the above scheme, when the data to be stored is data generated during the operation of the application, the data to be stored includes multiple sub-data, and the sub-data in the data to be stored corresponds one-to-one to the running time of the application; the above-mentioned integration module is also used to splice the sub-data corresponding to the running time in each target data to be stored for each running time of the application, so as to obtain the sub-integrated data corresponding to the running time; the sub-integrated data corresponding to each running time is constructed as the integrated data.

[0025] In the above scheme, the above integration module is also used to identify the format of each target data to be stored to obtain the data format of each target data to be stored; for each target data to be stored, based on the data format of the target data to be stored and the preset data format, determine the candidate data corresponding to the target data to be stored, and the candidate data is in the preset data format; integrate each candidate data to obtain the integrated data.

[0026] In the above scheme, the above integrated module is also used to compare the data format of the target data to be stored with the preset data format to obtain a second comparison result; when the second comparison result indicates that the data format of the target data to be stored is different from the preset data format, based on the preset data format, the target data to be stored is format converted to obtain the candidate data; when the second comparison result indicates that the data format of the target data to be stored is the same as the preset data format, the target data to be stored is determined as the candidate data.

[0027] In the above scheme, the preset data format is a text data format, and the above integrated module is also used to extract features of the target data to be stored to obtain data features of the target data to be stored; call the text conversion model, and based on the data features of the target data to be stored, perform format conversion on the target data to be stored to obtain the candidate data.

[0028] In the above scheme, the construction device of the above database also includes: a response module, which is used to respond to a data processing request carrying preset data, extract features from the preset data, and obtain data features of the preset data; determine from the target database a first data feature whose similarity with the data feature of the preset data reaches a similarity threshold, and determine from the mapping relationship the first encrypted data that has a mapping relationship with the first data feature; and respond to the data processing request based on the first encrypted data.

[0029] An embodiment of the present application provides an electronic device, including:

[0030] A memory for storing computer executable instructions or computer programs;

[0031] The processor is used to implement the database construction method provided in the embodiment of the present application when executing the computer executable instructions or computer programs stored in the memory.

[0032] An embodiment of the present application provides a computer-readable storage medium storing computer-executable instructions for causing a processor to execute the instructions to implement the database construction method provided in the embodiment of the present application.

[0033] The embodiment of the present application provides a computer program product, which includes a computer program or a computer executable instruction, and the computer program or the computer executable instruction is stored in a computer-readable storage medium. The processor of the electronic device reads the computer executable instruction from the computer-readable storage medium, and the processor executes the computer executable instruction, so that the electronic device executes the database construction method described in the embodiment of the present application.

[0034] The embodiments of the present application have the following beneficial effects:

[0035] By acquiring multiple data to be stored, identifying the association of the multiple data to be stored, obtaining the identification result, integrating the target data to be stored that do not have data association among the multiple data to be stored based on the identification result, obtaining the integrated data, and constructing the target database based on the sub-integrated data, the target database includes the mapping relationship corresponding to each sub-integrated data, the mapping relationship uses the data feature of the sub-integrated data as the index, and uses the encrypted data of each sub-integrated data as the data content. In this way, by identifying the association of the multiple data to be stored, obtaining the identification result, integrating the target data to be stored that do not have data association among the multiple data to be stored based on the identification result, obtaining the integrated data, so that the obtained integrated data can realize the unified management and analysis of the target data to be stored that do not have data association, and by constructing the target database based on the sub-integrated data, the data that can be retrieved in the obtained target database is the integrated sub-integrated data, and because the sub-integrated data includes at least part of the data of each target data to be stored, the target database can realize the rapid retrieval of at least part of each target data to be stored, thereby effectively improving the query performance of the determined target database. Since the data features corresponding to the sub-integration data can accurately reflect the data characteristics of the sub-integration data, and the data features corresponding to the sub-integration data are in vector form, the data features in vector form are not readable, while the sub-integration data are usually readable, therefore, the readable sub-integration data can be encrypted, and the unreadable data features are not encrypted, a mapping relationship between the encrypted data and the data features is established, and a target database is constructed based on the mapping relationship, thereby effectively improving the data security of the target database. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 It is a schematic diagram of the architecture of the database construction system provided in the embodiment of the present application;

[0037] Figure 2 is a schematic diagram of the structure of an electronic device for building a database provided in an embodiment of the present application;

[0038] Figure 3 This is a schematic diagram of the process of constructing a database provided in the embodiment of the present application. Figure 1 ;

[0039] Figure 4 This is a schematic diagram of the process of constructing a database provided in the embodiment of the present application. Figure 2 ;

[0040] Figure 5 This is a schematic diagram of the process of constructing a database provided in the embodiment of the present application. Figure 3 ;

[0041] Figure 6 This is a schematic diagram of the process of constructing a database provided in the embodiment of the present application. Figure 4 ;

[0042] Figure 7 This is a schematic diagram of the process of constructing a database provided in the embodiment of the present application. Figure 5 ;

[0043] Figure 8 This is a schematic diagram of the principle of the database construction method provided in the embodiment of the present application. Figure 1 ;

[0044] Fig. 9 This is a schematic diagram of the principle of the database construction method provided in the embodiment of the present application. Figure 2 ;

[0045] Fig.10 It is a schematic diagram of the system architecture of the method for constructing a database provided in an embodiment of the present application;

[0046] Fig.11 This is a schematic diagram of the process of constructing a database provided in the embodiment of the present application. Figure 6 ;

[0047] Fig.12 This is a schematic diagram of the process of constructing a database provided in the embodiment of the present application. Figure 7 ;

[0048] Fig.13 It is a schematic diagram of a configuration file of a method for constructing a database provided in an embodiment of the present application. DETAILED DESCRIPTION

[0049] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings. The described embodiments should not be regarded as limiting the present application. All other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of this application.

[0050] In the following description, reference is made to “some embodiments”, which describe a subset of all possible embodiments, but it will be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0051] In the following description, the terms "first\second\third" involved are merely used to distinguish similar objects and do not represent a specific ordering of the objects. It can be understood that "first\second\third" can be interchanged with a specific order or sequence where permitted, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.

[0052] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0053] Before further describing the embodiments of the present application in detail, the nouns and terms involved in the embodiments of the present application are explained. The nouns and terms involved in the embodiments of the present application are subject to the following interpretations.

[0054] 1) Database: In short, it can be regarded as an electronic file cabinet where electronic files are stored. Users can add, query, update, delete, etc. the data in the files. The so-called database is a collection of data that is stored together in a certain way, can be shared with multiple users, has as little redundancy as possible, and is independent of the application.

[0055] 2) Big Data: refers to a collection of data that cannot be captured, managed, and processed by conventional software tools within a certain time frame. It is a massive, high-growth, and diverse information asset that requires new processing models to have stronger decision-making power, insight discovery, and process optimization capabilities. With the advent of the cloud era, big data has also attracted more and more attention. Big data requires special technologies to effectively process large amounts of data within a tolerable time frame. Technologies suitable for big data include large-scale parallel processing databases, data mining, distributed file systems, distributed databases, cloud computing platforms, the Internet, and scalable storage systems.

[0056] 3) In response to: used to indicate the conditions or states on which the executed operations depend. When the dependent conditions or states are met, one or more operations executed may be in real time or with a set delay. Unless otherwise specified, there is no restriction on the order in which the multiple operations executed are executed.

[0057] 4) Artificial Intelligence (AI): It is a comprehensive discipline that covers a wide range of fields, including both hardware-level and software-level technologies. The basic technologies of artificial intelligence generally include sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, pre-trained model technology, operation / interaction systems, mechatronics, etc. Among them, the pre-trained model is also called a large model or a basic model. After fine-tuning, it can be widely used in downstream tasks in various major directions of artificial intelligence. Artificial intelligence software technology mainly includes computer vision technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0058] 5) Machine Learning (ML): It is a multi-disciplinary interdisciplinary subject involving probability theory, statistics, approximation theory, convex analysis, algorithm complexity theory and other disciplines. It specializes in studying how computers simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to make computers intelligent. Its applications are spread across all areas of artificial intelligence. Machine learning and deep learning usually include artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and learning by induction. The pre-trained model is the latest development of deep learning, which integrates the above technologies.

[0059] 6) Natural Language Processing (NLP): It is an important direction in the fields of computer science and artificial intelligence. It studies various theories and methods that can achieve effective communication between people and computers using natural language. Natural language processing is a science that integrates linguistics, computer science, and mathematics. Therefore, research in this field will involve natural language, that is, the language used by people in daily life, so it is closely related to the study of linguistics, but there are important differences. Natural language processing is not a general study of natural language, but the development of computer systems that can effectively realize natural language communication, especially the software systems therein. Therefore, it is a part of computer science. Natural language processing is mainly used in machine translation, automatic summarization, opinion extraction, text classification, question answering, text semantic comparison, speech recognition, etc.

[0060] 7) Retrieval Augmented Generation (RAG): When using a large language model for text or image generation, it is difficult to cover the amount of information in the private data because the huge corpus used in the training of the large language model does not contain the private data of the current user. Through the retrieval augmented generation technology, when the user uses the large language model, it will first search based on the input data, retrieve the relevant private data and input it into the large model together, so as to enhance or optimize the output of the large model.

[0061] 8) Message Queue (MQ): Also known as message queue middleware, it is an important component in distributed systems. Its main function is to serve as a communication method between multiple asynchronous services. The main problems it solves are: asynchronous processing, flow control, front-end elimination, and asynchronous processing when sending large amounts of message data. The system can be divided into three parts: Producer (message generator) is responsible for producing and sending messages to Broker; the message processing center is responsible for message storage, confirmation, and retry, and will include multiple queues; the message consumer is responsible for obtaining messages from Broker and processing business logic.

[0062] 9) Key Management Services (KMS): Key Management Services is a one-stop key management and data encryption service platform for users, providing simple, reliable, secure, and compliant data encryption protection capabilities. KMS helps users reduce the procurement, operation, and R&D expenses on cryptographic infrastructure and data encryption and decryption products so that users can focus on the business itself. In a nutshell: it allows users to create and manage secret public and private keys to protect user data.

[0063] 10) Vector database (Vector DB): is a database that stores information in the form of vectors, which are digital representations of data objects, also known as vector embeddings. It can use the power of these vector embeddings to index and search massive datasets of unstructured and semi-structured data (such as images, text, or sensor data). Vector databases are built specifically for managing vector embeddings, so they provide a complete solution for managing unstructured and semi-structured data. The common way is to calculate the similarity of two vectors by using a similarity calculation function, such as Cosine_Similarity. The similarity between two vectors represents the similarity between two texts or data, and then retrieve the data based on the similarity.

[0064] 11) Online service (Embedding service, Embedding): It is the process of mapping a data set to a vector space and vectorizing the data, that is, a method of "representing" an object (Object) with a numerical vector. "Entity object" can be an image, word, etc., and the "numerical representation" is a coding vector. For example, a three-element vector (R, G, B) is used to encode an entity object such as "color". Embedding can also be understood as projecting a discrete target onto a point in a continuous space. The numerical embedding vector itself is meaningless, and the relative relationship between different vectors is meaningful. Embedding Service refers to deploying the Embedding model and providing online services (OpenAPI interface or grpc interface) for users to calculate the Embedding value of the query text.

[0065] 12) Data Evaluator Service: Data evaluation refers to the use of specific data evaluation algorithms or large models to score and rate data quality. The scores or ratings are used for subsequent data processing and optimization. Further, Data Evaluator Service refers to the deployment of the computational logic of lecture data evaluation and the provision of online services (OpenAPI interface or Grpc interface) for users to rate and process data.

[0066] 13) Advanced Encryption Standard (AES): A widely used symmetric encryption algorithm, which was released by the National Institute of Standards and Technology (NIST) in 2001 to replace the original Data Encryption Standard (DES). The main features include: Symmetric encryption: AES is a symmetric encryption algorithm, that is, it uses the same key for encryption and decryption. Block encryption: AES is a block encryption algorithm that divides the input (plaintext) into multiple fixed-size blocks and then encrypts each block. The block size of AES is fixed to 128 bits. Security: AES is widely regarded as a very secure encryption algorithm.

[0067] 14) Large Language Model (LLM): A machine learning model designed to understand and generate human language. They are trained on large amounts of text data and can perform a wide range of tasks, including text summarization, translation, sentiment analysis, etc. Large language models are characterized by their large scale and billions of parameters that help them learn complex patterns in language data. They are usually based on deep learning architectures. Large language models are also called large language models or large models. Large language models refer to deep learning models trained with large amounts of text data, containing billions or even larger parameters. They can be used to generate natural language text and understand the meaning of natural language text. Through training, the model can learn the statistical laws and semantic associations of language to build a huge language knowledge base, thereby simulating human language understanding and generation capabilities.

[0068] 15) Stream Computing: Also known as streaming computing or real-time stream processing, it is mainly designed to meet the real-time application requirements of the scenario and can be widely used in multiple fields such as the Internet and the Internet of Things. The entire data stream processing process is often completed within milliseconds. In stream computing, data is presented in groups in the form of continuous data streams and continuously integrated into the big data stream computing platform. The implementation of its computing function is described in the form of a directed task graph. Before the corresponding calculation results are generated in real time, the data stream will flow through the directed task graph, and the corresponding data will be gradually acquired in an incremental manner at different time points in the stream data. Common stream computing engines include: Apache Flink, Apache Storm, Spark Streaming, etc.

[0069] 16) ETL: Extract, Transform, Load. The ETL process is the core of data integration and involves extracting data from different sources, converting it into a format suitable for analysis, and then loading it into a target system such as a data warehouse or data lake. Extract: Extract raw data from various data sources. These data sources may include relational databases, file systems, APIs, streaming data, etc. Transform: Transform the data. The transformation process may include data cleaning (removing errors and inconsistent data), data integration (merging data from different sources), data aggregation (summarizing data to reduce the amount of data), data validation (ensuring that the data meets specific standards), etc. Load: Load the transformed data into a target system such as a data warehouse, data lake, or a specific data store.

[0070] 17) Data Integration Tool: Aims to provide users with a more comprehensive set of programming interfaces (APIs). This tool allows users to elegantly define their ETL (Extract, Transform, Load) processes in the form of YAML configuration files, and assists users in automatically generating customized Flink operators and submitting Flink jobs. Flink CDC has been optimized during the task submission process, and some advanced features have been added, such as automatic synchronization of table structure changes (Schema Evolution), data transformation (Data Transformation), full database synchronization (Full Database Synchronization), and exactly-once semantics.

[0071] During the implementation of the embodiments of the present application, the applicant discovered that the related technology has the following problems:

[0072] In the related art, when constructing a database, the data to be stored or the encrypted data corresponding to the data to be stored is usually directly stored. Directly storing the data to be stored will result in poor data security of the data storage, while storing the encrypted data corresponding to the data to be stored will result in poor data operation performance of the stored encrypted data because the data has been encrypted. Moreover, directly storing multiple data to be stored that do not have data correlation in the database will result in poor query performance of the constructed database. Therefore, the related art cannot effectively improve the query performance of the database while effectively improving the data security of the database.

[0073] The embodiments of the present application provide a method, device, electronic device, computer-readable storage medium and computer program product for constructing a database, which can effectively improve the data security of the database while improving the query performance of the database. The following describes an exemplary application of the database construction system provided in the embodiments of the present application.

[0074] See also Figure 1 , Figure 1 It is a schematic diagram of the architecture of a database construction system 100 provided in an embodiment of the present application. A terminal (terminal 400 is shown as an example) is connected to a server 200 via a network 300. The network 300 may be a wide area network or a local area network, or a combination of the two.

[0075] The terminal 400 is used for the user to use the client 410 to display the data to be stored on the graphical interface 410 - 1 (the graphical interface 410 - 1 is shown as an example). The terminal 400 and the server 200 are connected to each other via a wired or wireless network.

[0076] In some embodiments, the server 200 may be an independent physical server, or a server cluster or business system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms. The terminal 400 may be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart TV, a smart watch, a car terminal, etc., but is not limited to this. The electronic device provided in the embodiment of the present application may be implemented as a terminal or as a server. The terminal and the server may be directly or indirectly connected via wired or wireless communication, which is not limited in the embodiment of the present application.

[0077] In some embodiments, the server 200 identifies the associations of multiple data to be stored and obtains an identification result. Based on the identification result, the server 200 integrates the target data to be stored that do not have data association to obtain integrated data, and sends the integrated data to the terminal 400. The terminal 400 builds a target database based on the sub-integrated data.

[0078] In other embodiments, the terminal 400 identifies the associations of multiple data to be stored and obtains an identification result. Based on the identification result, the terminal 400 integrates the target data to be stored that do not have data association to obtain integrated data, and sends the integrated data to the server 200. The server 200 builds a target database based on the sub-integrated data.

[0079] See also Figure 2 , Figure 2 is a schematic diagram of the structure of an electronic device 500 for building a database provided in an embodiment of the present application, wherein: Figure 2 The electronic device 500 shown may be Figure 1 The server 200 or the terminal 400 in Figure 2 The electronic device 500 shown includes: at least one processor 430, a memory 450, and at least one network interface 420. The various components in the electronic device 500 are coupled together via a bus system 440. It is understood that the bus system 440 is used to achieve connection and communication between these components. In addition to the data bus, the bus system 440 also includes a power bus, a control bus, and a status signal bus. However, for the sake of clarity, the bus system 440 is not described in detail. Figure 2 Various buses are labeled as bus system 440 .

[0080] The processor 430 may be an integrated circuit chip having signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc., wherein the general-purpose processor may be a microprocessor or any conventional processor, etc.

[0081] The memory 450 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state memory, hard drives, optical drives, etc. The memory 450 may optionally include one or more storage devices that are physically remote from the processor 430.

[0082] The memory 450 includes a volatile memory or a non-volatile memory, and may also include both volatile and non-volatile memories. The non-volatile memory may be a read-only memory (ROM), and the volatile memory may be a random access memory (RAM). The memory 450 described in the embodiments of the present application is intended to include any suitable type of memory.

[0083] In some embodiments, memory 450 can store data to support various operations, examples of which include programs, modules, and data structures, or a subset or superset thereof, as exemplarily described below.

[0084] Operating system 451, including system programs for processing various basic system services and performing hardware-related tasks, such as a framework layer, a core library layer, a driver layer, etc., for implementing various basic services and processing hardware-based tasks;

[0085] The network communication module 452 is used to reach other electronic devices via one or more (wired or wireless) network interfaces 420. Exemplary network interfaces 420 include: Bluetooth, Wireless Fidelity (WiFi), and Universal Serial Bus (USB).

[0086] In some embodiments, the database construction device provided in the embodiments of the present application can be implemented in a software manner. Figure 2 The construction device 455 of the database stored in the memory 450 is shown, which can be software in the form of a program and a plug-in, etc., including the following software modules: an identification module 4551, an integration module 4552 and a construction module 4553. These modules are logical, so they can be arbitrarily combined or further split according to the functions implemented. The functions of each module will be explained below.

[0087] In other embodiments, the database construction device provided in the embodiments of the present application can be implemented in hardware. As an example, the database construction device provided in the embodiments of the present application can be a processor in the form of a hardware decoding processor, which is programmed to execute the database construction method provided in the embodiments of the present application. For example, the processor in the form of a hardware decoding processor can adopt one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field programmable gate arrays (FPGAs) or other electronic components.

[0088] In some embodiments, the terminal or server can implement the database construction method provided in the embodiments of the present application by running a computer program or a computer executable instruction. For example, the computer program can be a native program (for example, a dedicated database construction program) or a software module in the operating system, for example, a database construction module that can be embedded in any program (such as an instant messaging client, a photo album program, an electronic map client, a navigation client); for example, it can be a native application (APP, Application), that is, a program that needs to be installed in the operating system to run. In short, the above-mentioned computer program can be an application, module or plug-in in any form.

[0089] The method for constructing a database provided in the embodiment of the present application will be explained in combination with the exemplary application and implementation of the server or terminal provided in the embodiment of the present application.

[0090] See also Figure 3 , Figure 3 This is a schematic diagram of the process of constructing a database provided in the embodiment of the present application. Figure 1 , will combine Figure 3 Steps 101 to 103 are shown for illustration. The database construction method provided in the embodiment of the present application can be implemented by a server or a terminal alone, or by a server and a terminal in collaboration. The following description will be made using the server alone as an example.

[0091] In step 101, a plurality of data to be stored is obtained, and correlation identification is performed on the plurality of data to be stored to obtain an identification result.

[0092] In some embodiments, the data sources of different data to be stored may be the same or different. The data to be stored originating from the same data source have data associations, while the data to be stored originating from different data sources do not have data associations. The above-mentioned data to be stored may be text data, code data, audio data, video data, etc. The specific form of the above-mentioned data to be stored does not constitute a limitation on the embodiments not applied for.

[0093] In some embodiments, see Figure 4 , Figure 4 This is a schematic diagram of the process of constructing a database provided in the embodiment of the present application. Figure 2 , Figure 3 The step 101 shown may be performed by Figure 4 Steps 1011A to 1012A are implemented as shown.

[0094] In step 1011A, the data source of each data to be stored is obtained.

[0095] In some embodiments, the data sources of the above-mentioned data to be stored may include various information systems, devices, application components and other data sources.

[0096] As an example, the data to be stored can be sensor data, and the data source can be a weather station: the source of environmental data such as temperature, humidity, and air pressure. Smart devices: user activity data recorded by smart watches. Industrial equipment: production line operation data collected by industrial robots and sensors.

[0097] As an example, the data to be stored may be social media data, and the data source may be a social media platform: such as user-generated posts, comments, and pictures. Social network analysis: Analyze the social network structure and information dissemination of users.

[0098] As an example, the data to be stored may be financial data, and the data source may be a banking system: transaction records, account balances, credit scores, etc. Stock market: stock prices, trading volumes, market trends, etc.

[0099] In step 1012A, for any two data to be stored among the multiple data to be stored, when the data sources of the any two data to be stored are the same, it is determined that there is data correlation between the any two data to be stored; when the data sources of the any two data to be stored are different, it is determined that there is no data correlation between the any two data to be stored.

[0100] In some embodiments, the data to be stored having the same data source have data association, while the data to be stored having different data sources do not have data association.

[0101] As an example, for data to be stored with the same data source, different systems within the same enterprise, in a company, different departments may use different databases or data warehouses to store similar data, such as customer information of the sales department and customer satisfaction survey data of the marketing department. Even if they come from different internal systems, since they are all about customers, it can be determined that there is a correlation between them.

[0102] As an example, for data to be stored with the same data source, data of the same time series, such as temperature and humidity data from the same weather station, although they are stored in different tables or files, they are related because they are all about the environmental conditions at the same time point.

[0103] As an example, for data to be stored that come from different sources, different enterprises or organizations, for example, patient data from different hospitals, although they all contain the patient's medical information, there may be no direct data correlation between them because they come from different organizations and there is no clear data integration strategy.

[0104] As an example, for data to be stored from different sources, time series data from different sources, such as temperature data from different weather stations, although they are all about weather, they come from different geographical locations and may be affected by different climate and environmental factors, so there is no direct data correlation between them.

[0105] In this way, clarifying data relevance can help identify and eliminate duplicate data and ensure data consistency and accuracy. Relevance analysis helps to discover and correct inconsistencies and errors in data and improve the overall quality of data. Data relevance analysis helps to integrate data from different data sources and build a unified data view. The integrated data can be more easily analyzed and supports cross-data source queries and reports. By determining data relevance, effective relationships can be established between multiple data sources, thereby improving the efficiency and accuracy of data processing. Specifically, when the data sources are the same, the relevance between data can be identified, which helps to eliminate data redundancy and improve data quality and consistency. At the same time, when the data sources are different, by determining that there is no relevance between data, unnecessary data integration and processing can be avoided, saving time and resources. In addition, by identifying and utilizing data relevance, data analysis and decision making can be better supported, helping organizations make more accurate predictions and decisions. In general, by determining data relevance, the efficiency of data processing can be improved, the data storage structure can be optimized, and the accuracy of data analysis and decision making can be improved, thereby bringing significant business value.

[0106] In some embodiments, when the data to be stored is data in an application program, see Figure 5 , Figure 5 This is a schematic diagram of the process of constructing a database provided in the embodiment of the present application. Figure 3 , Figure 3 Step 101 shown may be performed by Figure 5 Steps 1011B to 1012B are implemented as shown.

[0107] In step 1011B, the generation time of each of the data to be stored in the application is obtained.

[0108] In some embodiments, when the data to be stored is data inherent in the application, the generation time of the data to be stored is before the startup time of the application; when the data to be stored is data generated during the operation of the application, the generation time of the data to be stored is after the startup time of the application.

[0109] In some embodiments, the generation moment refers to the point in time when data is generated or created in an application. This point in time is critical for data management and analysis because it provides the context and timestamp of data generation, which helps to understand and process the data.

[0110] In step 1012B, for any two data to be stored among the multiple data to be stored, when the generation times of the any two data to be stored are respectively before and after the startup time of the application, it is determined that there is no data correlation between the any two data to be stored; when the generation times of the any two data to be stored are both before the startup time of the application, it is determined that there is data correlation between the any two data to be stored.

[0111] In some embodiments, when the generation time of any two data to be stored is respectively before and after the startup time of the application, it means that the data to be stored is data inherent to the application and data generated by the application during the operation. Since the data inherent to the application and the data generated by the application during the operation do not have data correlation, it is determined that there is no data correlation between the any two data to be stored.

[0112] In some embodiments, when the generation times of any two data to be stored are both before the startup time of the application, it means that the data to be stored are data inherent to the application, and therefore, it is determined that there is data correlation between the any two data to be stored.

[0113] In some embodiments, when the generation time of any two data to be stored is after the startup time of the application, the data sources of any two data to be stored are obtained; when the data sources of any two data to be stored are the same, it means that any two data to be stored are generated from the same component of the application, and it is determined that there is data correlation between the any two data to be stored; when the data sources of any two data to be stored are different, it means that any two data to be stored are generated from different components of the application, and it is determined that there is no data correlation between the any two data to be stored.

[0114] As an example, in the application scenario of a social media application, the data to be stored can be a record of user-generated posts for the social media application. Data to be stored 1: User A posted a post within 30 minutes before the application was launched. Data to be stored 2: User B posted a post within 5 minutes after the application was launched. Correlation analysis: Since the generation time of data 1 to be stored is before the application is launched, and the generation time of data 2 to be stored is after the application is launched, it can be determined that there is no data correlation between the two pieces of data.

[0115] As an example, in the application scenario of an e-commerce application, the e-commerce application records the user's purchase behavior. Data to be stored 1: User A submitted an order 1 hour before the application was launched. Data to be stored 2: User B submitted an order within 10 minutes after the application was launched. Correlation analysis: Although the two order data are both about purchases, they were generated before and after the application was launched, respectively, so there is no data correlation between them.

[0116] In this way, by determining whether there is a correlation between the data, the efficiency and accuracy of data processing can be effectively improved. Specifically, when the generation time of two data to be stored is before and after the startup time of the application, it can be determined that there is no direct causal relationship or correlation between the two sets of data, thereby avoiding unnecessary data integration and analysis, saving time and resources. When the generation time of the two data to be stored is before the startup time of the application, it can be judged that there is a certain correlation between the two sets of data, which helps to analyze and explore the intrinsic connection between the data and provide support for more in-depth data analysis and decision-making. In general, by analyzing and utilizing the information at the time of data generation, the efficiency of data processing can be effectively improved, the data storage structure can be optimized, and more accurate data analysis and decision-making can be supported, thereby bringing significant business value.

[0117] In some embodiments, see Figure 6 , Figure 6 This is a schematic diagram of the process of constructing a database provided in the embodiment of the present application. Figure 4 , Figure 3 The step 101 shown can be performed for any two of the plurality of data to be stored. Figure 6 Steps 1011C to 1013C are implemented as shown.

[0118] In step 1011C, the correlation recognition model is called to predict the correlation degree between any two data to be stored, so as to obtain the correlation degree between any two data to be stored.

[0119] In some embodiments, the relevance identification model is a machine learning model that aims to predict the degree of association between any two data points. The model estimates the possible correlation or causal relationship between the data by analyzing the statistical characteristics, relationship patterns and interactions between the data. In practical applications, the relevance identification model may involve a variety of algorithms and techniques, such as distance-based algorithms (such as Euclidean distance, cosine similarity, etc.), graph-based algorithms (such as co-occurrence analysis, network analysis, etc.), and machine learning-based methods (such as support vector machines, decision trees, neural networks, etc.). These models usually require a large number of data samples for training to improve their prediction accuracy and generalization ability. By calling the relevance identification model, the degree of association between multiple data to be stored can be predicted, thereby helping data analysts and decision makers to better understand and utilize the intrinsic connections between data. This analysis helps to discover new business insights, optimize data structures, and improve the efficiency of data processing. In short, the relevance identification model is a powerful tool that can identify valuable information and patterns from large amounts of data.

[0120] In some embodiments, the degree of association refers to a measure of the degree of mutual association between two data to be stored. In data analysis, the degree of association is used to measure the correlation between two variables, that is, their statistical trend of change and the strength of the relationship. The degree of association can be positive, negative, or there is no association at all. In a positive association, an increase in one variable is usually accompanied by an increase in another variable (for example, the relationship between temperature and humidity, they usually increase or decrease at the same time). In a negative association, an increase in one variable is usually accompanied by a decrease in another variable (for example, the relationship between price and demand, demand usually decreases when prices rise). If there is no obvious statistical relationship between two variables, then there is no association between them. There are many ways to measure the degree of association, for example: Pearson correlation coefficient: It is a common method to measure the linear correlation between two continuous variables. Its value ranges from -1 to 1, where 1 indicates a perfect positive correlation, -1 indicates a perfect negative correlation, and 0 indicates no correlation. Spearman rank correlation coefficient: It is used to measure the monotonic relationship between two variables, usually used for non-normally distributed data. Kendall rank correlation coefficient: It is similar to Spearman rank correlation coefficient, but the calculation method is slightly different. Chi-square test: used to measure the association between two categorical variables, such as the relationship between gender and purchasing behavior. The association identification model predicts and estimates the association between data points by analyzing the statistical characteristics, relationship patterns, and interactions between data.

[0121] In step 1012C, when the correlation degree is less than the correlation degree threshold, it is determined that there is no data correlation between any two data to be stored.

[0122] In some embodiments, in data analysis, the correlation threshold is a pre-set value used to determine whether the correlation between two data points is significant enough. When the correlation is less than this threshold, it can be considered that there is no significant correlation between the two data points.

[0123] In step 1013C, when the correlation degree reaches the correlation degree threshold, it is determined that there is data correlation between any two data to be stored.

[0124] In some embodiments, the correlation threshold is a pre-set value used to determine whether the correlation between two data to be stored is significant enough. When the correlation reaches the correlation threshold, it can be considered that there is no significant correlation between the two data to be stored.

[0125] In this way, by calling the correlation identification model, the correlation between multiple data to be stored can be effectively predicted and evaluated, providing strong support for data analysis and decision-making. Specifically, when the correlation is less than the preset correlation threshold, it can be determined that there is no significant correlation between these data, thereby avoiding in-depth analysis and processing of these irrelevant data, saving time and resources. When the correlation reaches the correlation threshold, it indicates that there is a significant correlation between the data, and the inherent connection between these data can be further explored and utilized to provide strong support for decision-making, help discover new business opportunities and optimize existing processes. Through the application of the correlation identification model, the efficiency of data processing can be improved, the data storage structure can be optimized, and more accurate and in-depth data analysis can be supported.

[0126] In some embodiments, after the above step 101, the following processing can also be performed: in response to the identification result indicating that none of the multiple data to be stored are the target data to be stored, the data to be stored that have the data association among the multiple data to be stored are merged to obtain at least one merged data; when the number of the merged data is one, a first database is constructed based on each of the data to be stored in the merged data; when the number of the merged data is multiple, each of the merged data is integrated to obtain merged integrated data, and a second database is constructed based on each sub-merged integrated data in the merged integrated data.

[0127] In some embodiments, the above-mentioned target data to be stored refers to data to be stored that has no data association with any data to be stored among multiple data to be stored. For example, multiple data to be stored include data to be stored A, data to be stored B, data to be stored C, data to be stored D and data to be stored E. Data to be stored A and data to be stored B have data association, data to be stored C and data to be stored D have data association, and data to be stored E has no data association with any data to be stored. Then data to be stored E is the target data to be stored.

[0128] As an example, the data to be stored that have the data association among the multiple data to be stored are merged to obtain at least one merged data. The data to be stored A and the data to be stored B have data association, and the data to be stored C, the data to be stored D and the data to be stored E have data association. Then the data to be stored A and the data to be stored B are merged to obtain the merged data {data to be stored A, data to be stored B}, and the data to be stored C, the data to be stored D and the data to be stored E are merged to obtain the merged data {data to be stored C, data to be stored D, data to be stored E}.

[0129] In some embodiments, when the number of the merged data is one, constructing a first database based on each of the data to be stored in the merged data can be achieved in the following manner: performing feature extraction on the data to be stored to obtain a first data feature of the data to be stored; performing data encryption on the data to be stored to obtain first encrypted data of the data to be stored; establishing a mapping relationship between the first encrypted data and the first data feature, and constructing a first database including the first encrypted data and the first data feature based on the mapping relationship.

[0130] In some embodiments, the merged data includes at least one sub-merged data, and different sub-merged data in the merged data correspond to different data types. The above-mentioned integration of each merged data to obtain merged integrated data can be achieved in the following way: for each data type, obtaining indication information corresponding to the data type, when the indication information indicates that sub-merged data corresponding to the data type exists in at least part of the merged data, determining the sub-integrated data corresponding to the data type based on the sub-merged data corresponding to the data type; and constructing the merged integrated data based on the sub-integrated data.

[0131] In some embodiments, the above-mentioned acquisition of indication information corresponding to the data type can be implemented in the following manner: the data type is determined as a target data type, and for each merged data, the target data type is compared with the data type of each sub-merged data in the merged data to obtain a first comparison result corresponding to the merged data; when the first comparison result indicates that sub-merged data of the target data type exists in the merged data, the first comparison result is determined as the target comparison result; when the target comparison result exists, the indication information is determined as the sub-merged data corresponding to the data type exists in at least part of the merged data.

[0132] In some embodiments, the above-mentioned determination of the sub-integrated data corresponding to the data type based on the sub-data corresponding to the data type can be achieved in the following manner: when the number of sub-data corresponding to the data type is one, the sub-data corresponding to the data type is determined as the sub-integrated data corresponding to the data type; when the number of sub-data corresponding to the data type is multiple, the sub-merged data corresponding to the data type are spliced ​​to obtain the sub-integrated data corresponding to the data type.

[0133] In some embodiments, the above-mentioned construction of the integrated data based on the sub-integration data can be achieved in the following way: when the number of the sub-integration data is one, the sub-integration data is determined as the integrated data; when the number of the sub-integration data is multiple, the integrated data is constructed based on each of the sub-integration data.

[0134] In this way, multiple data to be stored are screened through the identification results, and non-target data to be stored are effectively excluded. This process greatly improves the efficiency and accuracy of data processing. Secondly, the data to be stored with data correlation are merged, which not only optimizes the data structure, but also reduces data redundancy, helps save storage space and improves data processing speed. Different strategies are used to build the database according to the number of merged data. When the number of merged data is one, the first database is directly based on the data to be stored, which simplifies the construction process and reduces the complexity of the system. When the number of merged data is multiple, the merged integrated data is obtained by integrating each merged data, and then the second database is constructed, which helps to integrate diverse data resources and improve the comprehensive utilization of data.

[0135] In some embodiments, the above-mentioned construction of the first database based on each of the data to be stored in the merged data can be achieved in the following way: performing feature extraction on each of the data to be stored to obtain data features of each of the data to be stored, and performing data encryption on each of the data to be stored to obtain encrypted data of each of the data to be stored; constructing the first database based on the data features of the data to be stored and the encrypted data of the data to be stored.

[0136] In some embodiments, the first database includes a first mapping relationship corresponding to each of the data to be stored, and the first mapping relationship uses the data feature of the data to be stored as an index and the encrypted data of the data to be stored as the data content.

[0137] In this way, by extracting features from the data to be stored, the first data features corresponding to the data to be stored are obtained, and the data to be stored is encrypted to obtain the first encrypted data corresponding to the data to be stored, and a mapping relationship between the first encrypted data and the first data features is established, and a first database including the first encrypted data and the first data features is constructed based on the mapping relationship. Since the first data features corresponding to the data to be stored can accurately reflect the data features of the data to be stored, and the first data features corresponding to the data to be stored are in vector form, the data features in vector form are not readable, and the data to be stored are usually readable, therefore, the readable data to be stored can be encrypted, and the first data features that are not readable are not encrypted, and a mapping relationship between the first encrypted data and the first data features is established, and a first database including the first encrypted data and the first data features is constructed based on the mapping relationship, thereby effectively improving the data security of data storage. At the same time, since the data features in vector form are not readable and can accurately reflect the data features of the data to be stored, and the data features in the first database are not encrypted, therefore, the first data features in the first database can be used as indexes to perform data operations on the data to be stored, and accurate operations on the data to be stored can be achieved, thereby improving the data operation performance of the stored data. In this way, the data security of the database can be effectively improved while the data operation performance of the database can be improved.

[0138] In step 102, based on the recognition result, target data to be stored that do not have data association among a plurality of data to be stored are integrated to obtain integrated data.

[0139] In some embodiments, the integrated data includes at least one sub-integrated data, and the sub-integrated data includes at least part of the target data to be stored. The so-called at least part refers to at least part of the target data to be stored, that is, all or part of the target data to be stored.

[0140] As an example, the sub-integration data includes at least part of each target data to be stored, and may be that the sub-integration data includes target data A to be stored, part of target data B to be stored, part of target data C to be stored, and target data D to be stored.

[0141] In some embodiments, the identification result is used to indicate whether each of the multiple data to be stored is the target data to be stored.

[0142] In some embodiments, the above-mentioned target data to be stored refers to data to be stored that has no data association with any data to be stored among multiple data to be stored. For example, multiple data to be stored include data to be stored A, data to be stored B, data to be stored C, data to be stored D and data to be stored E. Data to be stored A and data to be stored B have data association, data to be stored C and data to be stored D have data association, and data to be stored E has no data association with any data to be stored. Then data to be stored E is the target data to be stored.

[0143] In some embodiments, see Figure 7 , Figure 7 This is a schematic diagram of the process of constructing a database provided in the embodiment of the present application. Figure 5 , Figure 3 Step 102 shown may be performed by executing Figure 7 Steps 1021 to 1023 are implemented as shown.

[0144] In step 1021, in response to the identification result indicating that each of the plurality of data to be stored is the target data to be stored, each of the target data to be stored is integrated to obtain the integrated data.

[0145] As an example, when the identification result indicates that each of the multiple data to be stored is the target data to be stored, for example, the multiple data to be stored include data to be stored A, data to be stored B, data to be stored C, data to be stored D and data to be stored E, then the data to be stored A, data to be stored B, data to be stored C, data to be stored D and data to be stored E have no data correlation with each other.

[0146] In some embodiments, the data to be stored includes at least one sub-data, and different sub-data in the data to be stored correspond to different data types.

[0147] In some embodiments, the above-mentioned integration of each target data to be stored to obtain integrated data can be achieved in the following manner: for each data type, obtaining indication information corresponding to the data type; when the indication information indicates that sub-data corresponding to the data type exists in at least part of the target data to be stored, determining sub-integrated data corresponding to the data type based on the sub-data corresponding to the data type; and constructing the integrated data based on the sub-integrated data.

[0148] As an example, the target data to be stored includes target data A to be stored, target data B to be stored, and target data C to be stored. The target data A to be stored includes sub-data A1 and sub-data A2, the target data B to be stored includes sub-data B1, sub-data B2, and sub-data B3, and the target data C to be stored includes sub-data C1, sub-data C2, and sub-data C3. For data type 1, indication information corresponding to the data type 1 is obtained. When the indication information indicates that sub-data corresponding to the data type exists in at least part of the target data to be stored, for example, sub-data A1 corresponding to data type 1 exists in the target data A to be stored, sub-data B1 corresponding to data type 1 exists in the target data B to be stored, and sub-data C1 corresponding to data type 1 exists in the target data C to be stored, based on sub-data B1, sub-data A1, and sub-data C1 corresponding to data type 1, the sub-data corresponding to data type 1 is determined.

[0149] Continuing with the above example, for data type 2, indication information corresponding to the data type 2 is obtained. When the indication information indicates that sub-data corresponding to the data type exists in at least part of the target data to be stored, for example, sub-data A2 corresponding to data type 2 exists in the target data A to be stored, sub-data B2 corresponding to data type 2 exists in the target data B to be stored, and sub-data C2 corresponding to data type 2 exists in the target data C to be stored, based on sub-data B2, sub-data A2 and sub-data C2 corresponding to data type 1, the sub-integrated data corresponding to data type 2 is determined.

[0150] Continuing with the above example, for data type 3, indication information corresponding to the data type 3 is obtained. When the indication information indicates that sub-data corresponding to the data type exists in at least part of the target data to be stored, for example, sub-data B3 corresponding to data type 3 exists in the target data B to be stored and sub-data C3 corresponding to data type 3 exists in the target data C to be stored, based on the sub-data B3 and sub-data C3 corresponding to the data type 3, the sub-integrated data corresponding to the data type 3 is determined.

[0151] In this way, by obtaining corresponding indication information for different data types, the system can accurately identify and locate relevant sub-data in the target data to be stored, which greatly improves the pertinence and efficiency of data processing. Constructing sub-integrated data based on these sub-data helps to clearly identify and organize data, reduce data redundancy, and ensure the structuring and consistency of data. When there are multiple sub-integrated data, the system can effectively integrate them into integrated data, which not only promotes information fusion between different data types, but also provides a comprehensive data perspective, which helps to deeply analyze and mine data value. Such a data integration process can provide high-quality data resources, support decision-making, optimize business processes, and improve operational efficiency.

[0152] In some embodiments, the above-mentioned acquisition of indication information corresponding to the data type can be implemented in the following manner: the data type is determined as the target data type, and for each target data to be stored, the target data type is compared with the data type of each sub-data in the target data to be stored to obtain a first comparison result corresponding to the target data to be stored; when the first comparison result indicates that sub-data of the target data type exists in the target data to be stored, the first comparison result is determined as the target comparison result; when the target comparison result exists, the indication information is determined as the sub-data corresponding to the data type exists in at least part of the target data to be stored.

[0153] In some embodiments, the first comparison result is used to indicate whether sub-data of the target data type exists in the target data to be stored.

[0154] Continuing with the above example, data type 3 is determined as the target data type. For the target data B to be stored, the target data type 3 is compared with the data types of each sub-data (sub-data B1, sub-data B2, and sub-data B3) in the target data B to be stored, and a first comparison result corresponding to the target data to be stored is obtained. When the first comparison result indicates that the sub-data (sub-data B3) of the target data type exists in the target data to be stored, the first comparison result is determined as the target comparison result. For the target data C to be stored, the target data type 3 is compared with the data types of each sub-data (sub-data C1, sub-data C2, and sub-data C3) in the target data C to be stored, and a first comparison result corresponding to the target data C to be stored is obtained. When the first comparison result indicates that the sub-data (sub-data C3) of the target data type exists in the target data to be stored, the first comparison result is determined as the target comparison result.

[0155] Continuing with the above example, when the target comparison result exists, the indication information is determined as sub-data corresponding to the data type existing in at least part of the target data to be stored.

[0156] In this way, the targeted data processing is ensured, and by accurately matching the target data type and sub-data type, relevant data can be effectively identified and screened out, thereby improving the accuracy and efficiency of data processing; through the first comparison result obtained, it is possible to quickly determine whether sub-data of the target data type exists in the target data to be stored, thereby reducing unnecessary data processing steps and saving computing resources and time; when there is a target comparison result, the process of confirming the indication information provides a reliable basis for data analysis and subsequent processing, making data integration and utilization more efficient and purposeful.

[0157] In some embodiments, the above-mentioned determination of the subset integrated data corresponding to the data type based on the sub-data corresponding to the data type can be achieved in the following manner: when the number of sub-data corresponding to the data type is one, the sub-data corresponding to the data type is determined as the subset integrated data corresponding to the data type; when the number of sub-data corresponding to the data type is multiple, the sub-data corresponding to the data type are spliced ​​to obtain the subset integrated data corresponding to the data type.

[0158] In some embodiments, in data processing and integration, stitching refers to combining multiple data elements or data sets into a coherent whole.

[0159] Continuing with the above example, when the number of sub-data corresponding to the data type 1 is multiple, the sub-data A1, sub-data A2 and sub-data A3 corresponding to the data type are concatenated to obtain the subset data corresponding to the data type {sub-data A1, sub-data A2, sub-data A3}.

[0160] In this way, when the number of sub-data corresponding to a data type is one, it is directly determined as sub-integrated data, which simplifies the data processing process, reduces the complexity of integration, and ensures the consistency and integrity of the data. When the number of sub-data is multiple, they are merged into sub-integrated data through splicing. This approach not only improves the comprehensive utilization efficiency of data, but also helps to eliminate data islands, enhance the correlation between data, and provide more abundant and comprehensive information support for subsequent data analysis and decision-making.

[0161] In some embodiments, the above-mentioned construction of the integrated data based on the sub-integration data can be achieved in the following way: when the number of the sub-integration data is one, the sub-integration data is determined as the integrated data; when the number of the sub-integration data is multiple, the integrated data is constructed based on each of the sub-integration data.

[0162] Continuing with the above example, when there are multiple subsets of the integrated data, the integrated data is constructed based on each subset of the integrated data. The integrated data can be obtained by splicing each subset of data. For example, subset data 1, subset data 2 and subset data 3 can be spliced ​​to obtain the integrated data {sub-integration data 1, subset data 2, subset data 3}.

[0163] In this way, when the number of sub-integrated data is one, it is directly determined as integrated data. This processing method is beneficial to reduce data processing steps, reduce system complexity and maintenance costs, and ensure the directness and efficiency of data. When the number of sub-integrated data is multiple, the final integrated data is formed by constructing and integrating these sub-integrated data, which helps to achieve comprehensive data integration, improve the overall value and utilization efficiency of data, and provide a richer and more accurate information basis for data analysis and decision-making.

[0164] In other embodiments, when the data to be stored is data generated during the running of the application, the data to be stored includes a plurality of sub-data, and the sub-data in the data to be stored corresponds one-to-one to the running time of the application.

[0165] In some embodiments, the runtime of an application generally refers to the time when a software program is actually running on a computer. During this stage, the program executes code, interacts with the user, processes data, and performs various other operations. During the runtime of the program, the operating system is responsible for managing the execution of the program, including memory allocation, process scheduling, and error handling. During runtime, the application may need to rely on various runtime libraries, which provide functions that are not included at compile time, such as mathematical calculations, network communications, or file operations.

[0166] In some embodiments, when the data to be stored is data generated during the running of an application, the data to be stored includes sub-data W1, sub-data W2 and sub-data W3, the sub-data W1 in the data to be stored corresponds to the running time T1 of the application, the sub-data W2 in the data to be stored corresponds to the running time T2 of the application, and the sub-data W3 in the data to be stored corresponds to the running time T3 of the application.

[0167] In other embodiments, the above-mentioned integration of each of the target data to be stored to obtain integrated data can be achieved in the following manner: for each of the running moments of the application, the sub-data corresponding to the running moment in each of the target data to be stored are spliced ​​to obtain the sub-integrated data corresponding to the running moment; the sub-integrated data corresponding to each of the running moments are constructed as the integrated data.

[0168] As an example, the target data to be stored includes target data to be stored A, target data to be stored B and target data to be stored C, the target data to be stored A includes sub-data A1, sub-data A2 and sub-data A3, the target data to be stored B includes sub-data B1, sub-data B2 and sub-data B3, and the target data to be stored C includes sub-data C1, sub-data C2 and sub-data C3.

[0169] Continuing with the above example, for each running moment 1 of the application, the sub-data A1, sub-data B1 and sub-data C1 corresponding to the running moment 1 in each target data to be stored are concatenated to obtain the sub-integrated data corresponding to the running moment 1.

[0170] Continuing with the above example, for each running moment 2 of the application, the sub-data A2, sub-data B2 and sub-data C2 corresponding to the running moment 2 in each target data to be stored are spliced ​​to obtain the sub-integrated data corresponding to the running moment 2.

[0171] In this way, by processing the target data to be stored at the runtime of the application, the data subsets at each moment can be spliced ​​together to form complete runtime subset integrated data. Such operations help to more deeply understand and analyze the operation status of the program, including performance, resource usage, etc. In this way, comprehensive integrated data can be constructed to provide strong support for the optimization, detection and maintenance of the application. This not only helps to improve the quality and stability of the application.

[0172] In other embodiments, the above-mentioned integration of each of the target data to be stored to obtain the integrated data can be achieved in the following ways: performing format identification on each of the target data to be stored to obtain the data format of each of the target data to be stored; for each of the target data to be stored, based on the data format of the target data to be stored and a preset data format, determining candidate data corresponding to the target data to be stored, the candidate data being in the preset data format; integrating each of the candidate data to obtain the integrated data.

[0173] In some embodiments, format recognition refers to the analysis and processing of a set of data to determine the specific format or structure they follow. In computer science and information processing, format recognition generally involves parsing the structure of data, extracting its key information, and mapping it to a standard or predefined data format. When performing format recognition, the system checks the consistency and specific patterns of the data, such as field delimiters, data types, data lengths, etc. This process can be done manually or through automated algorithms and tools. For example, when processing text data, format recognition may involve determining the line structure in the text, field delimiters, the order in which data units are arranged, etc.

[0174] In some embodiments, the candidate data includes at least one sub-data, and different sub-data in the candidate data correspond to different data types. The above-mentioned integration of the candidate data to obtain the integrated data can be achieved in the following manner: for each data type, obtaining indication information corresponding to the data type, when the indication information indicates that sub-data corresponding to the data type exists in at least part of the target candidate data, determining sub-integrated data corresponding to the data type based on the sub-data corresponding to the data type; and constructing the integrated data based on the sub-integrated data.

[0175] In some embodiments, the above-mentioned acquisition of indication information corresponding to the data type can be implemented in the following manner: the data type is determined as a target data type, and for each candidate data, the target data type is compared with the data type of each sub-data in the candidate data to obtain a first comparison result corresponding to the candidate data; when the first comparison result indicates that sub-data of the target data type exists in the candidate data, the first comparison result is determined as the target comparison result; when the target comparison result exists, the indication information is determined as the sub-data corresponding to the data type exists in at least part of the candidate data.

[0176] In this way, by identifying the format of the target data to be stored, it is possible to ensure accurate understanding and processing of the data, so that when processing data from different sources and formats, it can be uniformly converted into a preset data format. The data can be analyzed and integrated more conveniently to obtain complete integrated data. On this basis, for each target data to be stored, candidate data can be determined according to its original data format and preset data format. These candidate data are data fragments converted according to the preset data format, which will be used to construct integrated data. Finally, unified integrated data can be obtained by integrating the candidate data. This integrated data set will contain key information of all target data to be stored for further analysis and processing. In this way, not only the efficiency of data processing is improved, but also the integrity and accuracy of the data are ensured, providing strong support for the optimization, detection and maintenance of applications.

[0177] In some embodiments, the above-mentioned determination of candidate data corresponding to the target data to be stored based on the data format of the target data to be stored and the preset data format can be achieved in the following manner: comparing the data format of the target data to be stored with the preset data format to obtain a second comparison result; when the second comparison result indicates that the data format of the target data to be stored is different from the preset data format, based on the preset data format, performing format conversion on the target data to be stored to obtain the candidate data; when the second comparison result indicates that the data format of the target data to be stored is the same as the preset data format, determining the target data to be stored as the candidate data.

[0178] In some embodiments, by comparing the data format of the target data to be stored with the preset data format, the difference in data format can be quickly identified, and the need for data conversion can be determined, thereby ensuring the consistency and compatibility of the data. When the comparison result shows that the data format of the target data to be stored is different from the preset data format, the target data can be format converted based on the preset data format, so that the original data can be uniformly converted into the preset data format, thereby simplifying the subsequent data integration and processing work. On the other hand, when the comparison result shows that the data format of the target data to be stored is the same as the preset data format, the target data can be directly used as candidate data, which saves time and resources for data conversion while ensuring the integrity and accuracy of the data. In short, by comparing the target data format with the preset data format, and performing corresponding data conversion or directly using the target data according to the comparison result, the efficiency of data processing can be improved, the compatibility and consistency of the data can be ensured, and strong support can be provided for subsequent data analysis and application.

[0179] In some embodiments, the preset data format is a text data format. The above-mentioned format conversion of the target data to be stored based on the preset data format to obtain the candidate data can be achieved in the following ways: performing feature extraction on the target data to be stored to obtain data features of the target data to be stored; calling a text conversion model, and based on the data features of the target data to be stored, performing format conversion on the target data to be stored to obtain the candidate data.

[0180] In some embodiments, feature extraction refers to the process of identifying and selecting useful information for a specific task from raw data. In natural language processing (NLP) and other fields, feature extraction is a key step in converting information collected from text data into a form that the model can process. These features can be words, phrases, sentence structures, or deeper semantic information, which are used to train machine learning models for tasks such as text classification, sentiment analysis, and machine translation.

[0181] In some embodiments, a text transformation model is used to transform text data into a form that can be processed by a machine learning algorithm. This transformation may include encoding text data into a numerical vector, extracting key semantic information, or converting text data into a structured data format. Text transformation models are often used in feature extraction processes to ensure that text data is processed and analyzed efficiently. This can be a simple statistical model, such as TF-IDF, or a complex deep learning model, such as a word embedding model or a sequence-to-sequence model, which can capture complex relationships and patterns in text data.

[0182] As an example, when the preset data format is a text data format and the data format of the target data to be stored is an image data format, feature extraction is performed on the target data to be stored in the image data format to obtain data features of the target data to be stored; a text conversion model is called, and based on the data features of the target data to be stored in the image data format, format conversion is performed on the target data to be stored in the image data format to obtain the candidate data in the text data format.

[0183] In this way, by extracting the features of the target data to be stored, the key features of these data can be obtained. These features are crucial for subsequent data analysis and processing, and can help reduce the dimension of the data, remove redundant information, and retain key information that is useful for specific tasks. In this way, data can be used more efficiently and the performance and accuracy of the model can be improved. After extracting the data features of the target data to be stored, the text conversion model is called to convert the format of these data, which can convert the text data into a unified format suitable for machine learning algorithms, thereby simplifying the complexity of data processing and analysis. The use of the text conversion model ensures the consistency and compatibility of the data, so that the data can be effectively used for various text analysis tasks. Through the application of feature extraction and text conversion models, candidate data is obtained. The candidate data not only contains the key features of the original data, but also is presented in a format suitable for machine learning algorithm processing. This provides strong support for subsequent data analysis and application, improves the efficiency and accuracy of data processing, and also provides a reliable basis for data-driven decision-making.

[0184] In step 1022, in response to the identification result indicating that some of the multiple data to be stored are the target data to be stored, the data to be stored with the data association among the multiple data to be stored are merged to obtain at least one merged data.

[0185] In some embodiments, the multiple data to be stored include data A to be stored, data B to be stored, and data C to be stored, and data A to be stored and data B to be stored can be target data to be stored. Then the identification result indicates that some of the data to be stored among the multiple data to be stored are the target data to be stored.

[0186] Continuing with the above example, the data to be stored that have the data association among the multiple data to be stored are merged to obtain at least one merged data (data to be stored C).

[0187] In step 1023, the target data to be stored and the merged data are integrated to obtain the integrated data.

[0188] In some embodiments, the target data to be stored includes at least one sub-data, and the merged data includes at least one sub-data. Different sub-data in the target data to be stored correspond to different data types, and different sub-data in the merged data correspond to different data types. The above-mentioned integration of the target data to be stored and the merged data to obtain the integrated data can be implemented in the following manner: for each data type, obtaining indication information corresponding to the data type, when the indication information indicates that the sub-data corresponding to the data type exists in at least part of the target data to be stored, or the sub-data corresponding to the data type exists in at least part of the merged data, determining the sub-integrated data corresponding to the data type based on the sub-data corresponding to the data type; and constructing the integrated data based on the sub-integrated data.

[0189] In some embodiments, obtaining the indication information corresponding to the data type can be achieved in the following manner: determining the data type as the target data type, comparing the target data type with the data type of each sub-data in the target data to be stored for each target data to be stored and each merged data, obtaining a first comparison result corresponding to the target data to be stored, and when the first comparison result corresponding to the target data to be stored indicates that sub-data of the target data type exists in the target data to be stored, determining the first comparison result as the target comparison result; comparing the target data type with the data type of each sub-data in the merged data, obtaining a first comparison result corresponding to the merged data, and when the first comparison result corresponding to the merged data indicates that sub-data of the target data type exists in the merged data, determining the first comparison result as the target comparison result. When the target comparison result exists, determining the indication information as the sub-data corresponding to the data type exists in at least part of the target data to be stored or at least part of the merged data.

[0190] In some embodiments, based on the sub-data corresponding to the data type, determining the sub-integrated data corresponding to the data type can be achieved in the following manner: when the number of sub-data corresponding to the data type is one, determining the sub-data corresponding to the data type as the sub-integrated data corresponding to the data type; when the number of sub-data corresponding to the data type is multiple, concatenating the sub-data corresponding to the data type to obtain the sub-integrated data corresponding to the data type.

[0191] In this way, when processing multiple data to be stored, the recognition results are used to determine whether the data contain the target data to be stored. It can help distinguish which data is the target data and which is not, so as to carry out subsequent processing in a targeted manner. When the recognition results show that all the data to be stored are target data, these data can be directly integrated to obtain complete integrated data, saving time and resources for data processing, while also ensuring the integrity and consistency of the data. When the recognition results show that the data to be stored contains non-target data, those data to be stored with data association can be merged. It can help us integrate related data so that we can better use these data for analysis and processing. Integrate the target data to be stored and the merged data to obtain the final integrated data. It realizes the unified management and analysis of data from different sources and improves the utilization and value of data.

[0192] In step 103, a target database is constructed based on the data of each sub-integration.

[0193] In some embodiments, the target database includes mapping relationships corresponding to each sub-integration data, the mapping relationships use data features of the sub-integration data as indexes, and use encrypted data of each sub-integration data as data content.

[0194] In some embodiments, the above-mentioned construction of the target database based on each sub-integration data can be achieved in the following manner: performing feature extraction on the sub-integration data to obtain the first data feature of the sub-integration data; performing data encryption on the sub-integration data to obtain the first encrypted data of the sub-integration data; establishing a mapping relationship between the first encrypted data and the first data feature, and constructing a target database including the first encrypted data and the first data feature based on the mapping relationship.

[0195] In some embodiments, after step 103 above, the data processing request can also be responded to in the following manner: in response to a data processing request carrying preset data, feature extraction is performed on the preset data to obtain data features of the preset data; from the target database, a first data feature whose similarity with the data feature of the preset data reaches a similarity threshold, and first encrypted data that has a mapping relationship with the first data feature is determined from the mapping relationship; based on the first encrypted data, the data processing request is responded to.

[0196] In some embodiments, the data processing request is used to request generation of associated data of the target data, and responding to the data processing request based on the first encrypted data can be achieved in the following ways: decrypting the first encrypted data to obtain decrypted data; fusing the target data and the decrypted data to obtain fused data; and based on the fused data, calling a machine learning model to generate associated data of the target data.

[0197] In some embodiments, the target database includes multiple sub-target databases, each of which corresponds to an object, and the encryption keys used for the encrypted data in different sub-target databases are different; decrypting the first encrypted data to obtain decrypted data can be achieved in the following way: determining the object corresponding to the data processing request, and obtaining the target key of the sub-target database corresponding to the object; based on the target key, decrypting the first encrypted data to obtain decrypted data.

[0198] In some embodiments, decrypting the first encrypted data to obtain decrypted data can be achieved in the following manner: obtaining key authorization information of the object corresponding to the data processing request for the first encrypted data, the key authorization information being used to indicate the authority to obtain the key of the first encrypted data; obtaining the key of the first encrypted data from a key manager based on the key authorization information of the first encrypted data; decrypting the first encrypted data based on the key of the first encrypted data to obtain decrypted data.

[0199] In some embodiments, the target database includes multiple sub-target databases, and determining the first data feature from the target database whose similarity with the data feature of the target data reaches a similarity threshold can be achieved in the following manner: obtaining the target identifier carried in the data processing request, and comparing the target identifier with the database identifiers of each of the sub-target databases to obtain an identifier comparison result; when the identifier comparison result indicates that the database identifier is the same as the target identifier, determining the corresponding sub-target database as the response database corresponding to the data processing request; determining the feature similarity between the data feature of the target data and each of the data features in the response database, and determining the data feature whose feature similarity is greater than the similarity threshold as the first data feature.

[0200] In some embodiments, responding to the data processing request based on the first encrypted data can be achieved in the following manner: when the number of the first data features exceeds the target number, evaluating the quality of the first encrypted data corresponding to each of the first data features to obtain a quality evaluation result; based on the quality evaluation result, screening out a target number of first encrypted data from the first encrypted data corresponding to each of the first data features; and responding to the data processing request based on the target number of first encrypted data.

[0201] In some embodiments, the above-mentioned acquisition of multiple data to be stored can be achieved in the following manner: obtain second encrypted data, the second encrypted data including an encrypted data name and encrypted data content; obtain a first name key, and based on the first name key, decrypt the encrypted data name to obtain a first content key carried in the encrypted data name; based on the first content key, decrypt the encrypted data content to obtain the data to be stored.

[0202] In some embodiments, the above-mentioned acquisition of the second encrypted data can be implemented in the following manner: acquiring a queue message transmitted in an end-to-end manner; parsing the queue message to obtain the second encrypted data.

[0203] In some embodiments, the above-mentioned feature extraction of the subset data to obtain the first data feature of the subset data can be achieved by the following method: data segmentation of the data to be stored to obtain multiple unit data; feature extraction of each unit data in parallel to obtain unit data features corresponding to each unit data; feature fusion of each unit data feature to obtain the first data feature of the data to be stored.

[0204] In some embodiments, the above-mentioned data encryption of the sub-integration data to obtain the first encrypted data of the sub-integration data can be achieved in the following manner: obtaining a second content key for encrypting the sub-integration data, encrypting the sub-integration data based on the second content key, and obtaining the encrypted data content corresponding to the sub-integration data; obtaining a second name key for encrypting the content key, encrypting the content key based on the second name key, and obtaining the encrypted data name corresponding to the sub-integration data; data merging the encrypted data name and the encrypted data content to obtain the first encrypted data corresponding to the sub-integration data.

[0205] Thus, by acquiring multiple data to be stored, the multiple data to be stored are identified for correlation, and the identification results are obtained. Based on the identification results, the target data to be stored that do not have data correlation among the multiple data to be stored are integrated to obtain integrated data. Based on the sub-integrated data, a target database is constructed, and the target database includes mapping relationships corresponding to each sub-integrated data. The mapping relationship uses the data features of the sub-integrated data as indexes and the encrypted data of each sub-integrated data as data content. Thus, by identifying the correlation of multiple data to be stored, the identification results are obtained. Based on the identification results, the target data to be stored that do not have data correlation among the multiple data to be stored are integrated to obtain integrated data, so that the obtained integrated data can achieve unified management and analysis of the target data to be stored that do not have data correlation. By constructing a target database based on the sub-integrated data, the data that can be retrieved in the obtained target database is the integrated sub-integrated data. Since the sub-integrated data includes at least part of each target data to be stored, the target database can achieve rapid retrieval of at least part of each target data to be stored, thereby effectively improving the query performance of the determined target database. Since the data features corresponding to the sub-integration data can accurately reflect the data characteristics of the sub-integration data, and the data features corresponding to the sub-integration data are in vector form, the data features in vector form are not readable, while the sub-integration data are usually readable, therefore, the readable sub-integration data can be encrypted, and the unreadable data features are not encrypted, a mapping relationship between the encrypted data and the data features is established, and a target database is constructed based on the mapping relationship, thereby effectively improving the data security of the target database.

[0206] Below, an exemplary application of the embodiment of the present application in an actual application scenario of a large language model will be described.

[0207] In the application scenario of the large language model, when the user uses the large language model, a text expression content (that is, the target request data described above) is generally sent to the large language model. The target database constructed by the database construction method provided in the embodiment of the present application performs content enhancement on the content of the text expression. In fact, it uses functions such as vector retrieval and calculation of vector similarity to find the corresponding supplementary corpus information, supplement the input to the large language model for prediction, and obtain the prediction result corresponding to the content of the text expression.

[0208] The embodiment of the present application combines the process of building a large model knowledge base using private data with traditional big data processing ETL, that is, adding the core steps in the knowledge base construction process: embedding calculation, data quality assessment, data encryption storage, etc. to the data integration system or stream computing system. According to the data source type, it can be divided into two types: the data source is the source database, the entire library table is built from library to library to build a private knowledge base, and the data source is the data collection of various types of scattered data from library to library to build a private knowledge base.

[0209] The embodiment of the present application uses customized stream computing data integration software to embed the Embedding computing UDF (User Define Function), data evaluation module, and data encryption module to build a private data knowledge base, and saves the encryption and decryption public and private keys used through the KMS service to ensure the security of data use.

[0210] In some embodiments, see Figure 8 , Figure 8 This is a schematic diagram of the principle of the database construction method provided in the embodiment of the present application. Figure 1 When the data source is a library table used daily by the business party and built with existing stock, you can directly use the data integration technology (Change Data Capture) of the stream computing engine to integrate and build a large language model private data knowledge base technology (data vectorization (data embedding, achieved by calling the embedding service), data quality assessment (achieved by calling the data quality assessment service), data encryption, and data private storage (managing keys through key management)), generate a knowledge base (target database) with one click, and synchronize the added and modified information in the data source at any time.

[0211] In some embodiments, see Fig. 9 , Fig. 9 This is a schematic diagram of the principle of the database construction method provided in the embodiment of the present application. Figure 2When the data sources are scattered in various places, such as text files, web page files, log information, and temporary data tables, the data integration framework (including the collection layer, the aggregation layer (including agent-1, agent-2, and agent-3), and the cache layer (cache layer 1, cache layer 2, and cache layer 3)) can be used to collect and process the data. At the same time, the sorting layer (Sort) supports data segmentation, data quality assessment (implemented through data quality assessment services), data embedding vectorization (data embedding, implemented through embedding services), and data encryption (managing keys through key management) in the process of building a private knowledge base through custom UDF functions.

[0212] In some embodiments, the embodiments of the present application transmit user private data end-to-end encrypted to the LLM application platform data processing system through MQ. The system obtains the authorized asymmetric key through the KMS service and decrypts the transmitted encrypted data in the memory. The plaintext data then enters the data processing Pipeline for data format parsing, data evaluation and scoring, and calculation of Embedding values. The Vector DB client is then used to obtain the symmetric key for data encryption from KMS, encrypt the processed plaintext data, and finally store the encrypted data and the corresponding Embedding value in the vector database.

[0213] In some embodiments, the database construction method provided in the embodiments of the present application is used to improve the platform usability: coupled with the data integration platform, the data encryption component is adapted in the data access component used by each business, and the knowledge base construction link is seamlessly accessed. When the business party needs data access, no additional adaptation development work is required. Reduce the platform operation and maintenance costs: the entire system is coupled with the ETL function of the big data stream computing system, so that the system has complete indicator detection, data reconciliation and other functions. At the same time, the stream computing system has good back pressure and fault tolerance, etc., to ensure the stability of the system. Support massive business access: combined with the existing data integration platform, the link for building a private knowledge base is seamlessly embedded, so that massive businesses can directly use the data access function of the data integration platform to complete the construction of a private data knowledge base, so that the large model capabilities can empower each business.

[0214] The embodiments of the present application are mainly used to improve the privacy, application scale, and data quality issues generated when using RAG technology in large language model applications. They are generally integrated in large language model application platforms for use, such as LangChain, LlamaIndex, or data platforms, to provide data access and vectorized storage functions to the application platform. Or they provide the LLM-powered autonomous agent system with the function of long-term memory, that is, the storage and retrieval of corpus, as a knowledge base outside the large language model training, to provide additional information.

[0215] The embodiment of the present application can be combined with the data access platform to embed the link for building the knowledge base. When the business party needs to convert the original whole database data into a knowledge base, the Flink CDC system can be used to migrate the whole database to create a knowledge base. When the business party needs to convert the data scattered in various files into a knowledge base, the data integration platform can be used to complete the collection of various types of data using the existing data access program.

[0216] In some embodiments, see Fig.10 , Fig.10 It is a schematic diagram of the system architecture of the method for constructing a database provided in the embodiment of the present application. The system architecture of the method for constructing a database provided in the embodiment of the present application can be divided into five subsystems: Vector database: the final storage place for processed data, the library table content includes the retrieved primary key ID, meta information, encrypted storage information, and embedding value used for familiarity calculation. The vector database used in the implementation process of the embodiment of the present application can be Star rocks. Embedding service: The embedded model is deployed on the Ray distributed computing framework to provide services. The service is provided in the form of http requests, inputs one (or a group of) text data, and returns one (or a group of) Embedding values. Large language model service: The large language model is deployed on the ray distributed computing framework to provide reasoning services. When the quality assessment algorithm splices the corresponding prompt statement, it is sent to the Serving service through an http request to obtain the comparison score data. Data integration framework: The carrier of the core processing logic of building a knowledge base, wherein the data integration framework includes traditional big data processing ETL capabilities, library-to-library data migration capabilities (such as Flink CDC), and full-link data collection capabilities. The management and control services include data nodes, open application programming interfaces, unified scheduling, security permissions, metadata, and approval flows. The data integration framework used in the implementation of the present application embodiment is as follows: Fig.10The data is processed by the collection layer, the aggregation layer (agent-1, agent-2 and agent-3), the cache layer (cache layer 1, cache layer 2 and cache layer 3) and the sorting layer (sorting on Flink, independent sorting, development kit, offline / online real-time computing), and then the data is stored to obtain the target database. Key management and distribution: KMS is used to generate, distribute and authorize keys. The owner of the private number can manage the key and clearly know the caller, usage scenario, number of times the key is used. Service access includes service access A, service access B, service access C, service access D and service access E.

[0217] In some embodiments, see Figure 8 and Fig.11 , Fig.11 This is a schematic diagram of the process of constructing a database provided in the embodiment of the present application. Figure 6 , Figure 8 The processing flow shown can be Fig.11 Steps 201 to 207 are implemented as shown below. Fig.11 Steps 201 to 207 are shown for explanation.

[0218] In step 201, a source database table is obtained.

[0219] In some embodiments, a source database table that needs to be processed is determined, and a table containing rich information is selected from the source database as an object for data extraction.

[0220] In step 202, a data segmentation method is selected.

[0221] In some embodiments, a data segmentation strategy is determined based on data characteristics and requirements to optimize the efficiency of subsequent processing and analysis.

[0222] In step 203, an embedding method is selected.

[0223] In some embodiments, a suitable embedding technology is selected to convert the data into a vector form that can be understood by the model, thereby enhancing the expressiveness of the data.

[0224] In step 204, data quality assessment is performed.

[0225] In some embodiments, a user-defined function (UDF) is used to evaluate data quality to ensure data accuracy and reliability.

[0226] In step 205, the data is encrypted and decrypted.

[0227] In some embodiments, data security measures are implemented to encrypt data to ensure the security of data during migration and storage.

[0228] In step 206, a data migration configuration file is compiled.

[0229] In some embodiments, based on the selections made in the previous steps, a detailed data migration configuration file is created to guide the migration process of data from the source to the target, ensuring that the data is transferred correctly.

[0230] In step 207, a target vector database table is constructed.

[0231] In some embodiments, vectorized data tables are constructed in the target database to provide a structured and optimized knowledge base for the large model RAG, facilitating model access and learning.

[0232] In some embodiments, see Fig. 9 and Fig.12 , Fig.12 This is a schematic diagram of the process of constructing a database provided in the embodiment of the present application. Figure 7 , Fig. 9 The processing flow shown can be Fig.12 Steps 301 to 311 shown in the figure are implemented, and the entire process of data collection and collection processing is processed by the data integration platform, including data collection, data aggregation and data caching. Finally, the stream computing capability of Flink SQL is used at the sorting layer to implement the embedding client, the data quality assessment function, the call of the large language model quality assessment Serving, and the data encryption storage processing through the user-defined function (UDF). Fig.12 Steps 301 to 311 are shown for explanation.

[0233] In step 301, a data source is obtained.

[0234] In some embodiments, the collected data source is obtained, such as a log file, a text file, etc.

[0235] In step 302, it is determined whether encryption of the acquisition process is required.

[0236] In some embodiments, whether to configure data encryption during collection is to decide whether to implement encryption measures during the data collection stage to ensure the security and privacy of data during transmission.

[0237] In step 303, when encryption is required, encryption is configured.

[0238] In some embodiments, encryption is configured to perform data encryption on the collected data.

[0239] In step 304, when encryption is not required or encryption configuration has been completed, the collection layer collects.

[0240] In some embodiments, the SDK of the acquisition layer is used to efficiently extract required data from different file formats and locations.

[0241] In step 305 , the aggregation layer aggregates data from different sources.

[0242] In some embodiments, data from multiple sources are processed and integrated uniformly to form a unified data view.

[0243] In step 306, the cache layer caches data.

[0244] In some embodiments, a message queue is used as a cache layer to temporarily store collected data to balance data flow and improve the processing capacity of the system to achieve traffic peak shaving.

[0245] In step 307, the data is segmented.

[0246] In some embodiments, the aggregated data is sorted and segmented to optimize the data structure and prepare for vectorized processing.

[0247] In step 308, an embed service is selected.

[0248] In some embodiments, a suitable embedding technology is selected to convert the data into a vector form that can be understood by the model, thereby enhancing the expressiveness of the data.

[0249] In step 309, data quality assessment is performed.

[0250] In some embodiments, a user-defined function (UDF) is used to evaluate data quality to ensure data accuracy and reliability.

[0251] In step 310, the data is encrypted and decrypted.

[0252] In some embodiments, data security measures are implemented to encrypt data to ensure the security of data during migration and storage.

[0253] In step 311, a target vector database table is constructed.

[0254] In some embodiments, vectorized data tables are constructed in the target database to provide a structured and optimized knowledge base for the large model RAG, facilitating model access and learning.

[0255] In some embodiments, see Fig.13 , Fig.13It is a configuration file diagram of the database construction method provided in the embodiment of the present application. It is a configuration file that needs to be written when migrating the entire database data to build a private data knowledge base. The entire configuration file includes four parts: Source, Sink, Transform, and Pipeline. The Source represents the source database table, and the fields therein are the information for linking the source database and the source data table that needs to be processed; the Sink represents the configuration information of the target vector database table, and the fields therein are the information for linking the target vector database and the target data table to be accessed; the Transform represents the configuration of each stage of private data processing in the stream computing process, such as the data segmentation method, the type of Embedding algorithm used, the LLM serving service called during data quality assessment, and whether encryption is required during data flow; the Pipeline represents the configuration of the entire stream computing, such as metadata configuration, resource number configuration, parallelism configuration, etc.

[0256] In some embodiments, the stream computing engine used in the embodiments of the present application may be Flink Cdc, or other types of stream computing engines such as Spark Streaming, Storm, Blink, etc. The overall data integration framework used in the embodiments of the present application may be Apache InLong, and alternatives may use other types of data integration frameworks, or message middleware components (Message queue) with custom UDFs. The distributed computing engine used in the pipeline for processing data can use not only Ray, but also other distributed computing frameworks such as MPI (Message Passing Interface), Spark, Flink, etc.

[0257] In this way, the database construction method provided in the embodiment of the present application combines the construction process of the private knowledge base with the traditional big data processing ETL technology, and integrates the entire data link: data access, data segmentation, data vectorization processing (embedding) and data scoring into the stream computing engine. At the same time, by adopting a distributed computing framework to support the Embedding service and LLM service, the amount of data processed is increased to the TB level, and the scale of business parties accessed by the platform is increased by an order of magnitude, which significantly improves the ease of use and improves the stability of the system by more than 90%. Use the DataEvaluator Service to rate and score the data and evaluate the data quality, so that the quality of the retrieved corpus content is improved by more than 20%.

[0258] Thus, by acquiring multiple data to be stored, the multiple data to be stored are identified for correlation, and the identification results are obtained. Based on the identification results, the target data to be stored that do not have data correlation among the multiple data to be stored are integrated to obtain integrated data. Based on the sub-integrated data, a target database is constructed, and the target database includes mapping relationships corresponding to each sub-integrated data. The mapping relationship uses the data features of the sub-integrated data as indexes and the encrypted data of each sub-integrated data as data content. Thus, by identifying the correlation of multiple data to be stored, the identification results are obtained. Based on the identification results, the target data to be stored that do not have data correlation among the multiple data to be stored are integrated to obtain integrated data, so that the obtained integrated data can achieve unified management and analysis of the target data to be stored that do not have data correlation. By constructing a target database based on the sub-integrated data, the data that can be retrieved in the obtained target database is the integrated sub-integrated data. Since the sub-integrated data includes at least part of each target data to be stored, the target database can achieve rapid retrieval of at least part of each target data to be stored, thereby effectively improving the query performance of the determined target database. Since the data features corresponding to the sub-integration data can accurately reflect the data characteristics of the sub-integration data, and the data features corresponding to the sub-integration data are in vector form, the data features in vector form are not readable, while the sub-integration data are usually readable, therefore, the readable sub-integration data can be encrypted, and the unreadable data features are not encrypted, a mapping relationship between the encrypted data and the data features is established, and a target database is constructed based on the mapping relationship, thereby effectively improving the data security of the target database.

[0259] It is understandable that in the embodiments of the present application, when it involves relevant data such as data to be stored, when the embodiments of the present application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of relevant data need to comply with relevant laws, regulations and standards of relevant countries and regions.

[0260] The following is a description of an exemplary structure of a database construction device 455 provided in an embodiment of the present application implemented as a software module. In some embodiments, for example, Figure 2As shown, the software modules in the database construction device 455 stored in the memory 450 may include: an identification module, used to obtain multiple data to be stored, identify the associations of the multiple data to be stored, and obtain an identification result; an integration module, used to integrate the target data to be stored that does not have data association among the multiple data to be stored based on the identification result, to obtain integrated data, wherein the integrated data includes at least one sub-integration data, and the sub-integration data includes at least part of the data of each target data to be stored; a construction module, used to construct a target database based on each sub-integration data, wherein the target database includes a mapping relationship corresponding to each sub-integration data, wherein the mapping relationship uses the data features of the sub-integration data as an index and uses the encrypted data of each sub-integration data as data content.

[0261] In some embodiments, the above-mentioned identification module is also used to obtain the data source of each of the data to be stored; for any two of the multiple data to be stored, when the data sources of the any two data to be stored are the same, it is determined that there is data correlation between the any two data to be stored; when the data sources of the any two data to be stored are different, it is determined that there is no data correlation between the any two data to be stored.

[0262] In some embodiments, when the data to be stored is data in an application, the above-mentioned identification module is also used to obtain the generation time of each data to be stored in the application; for any two data to be stored among the multiple data to be stored, when the generation times of the any two data to be stored are respectively before and after the startup time of the application, it is determined that there is no data correlation between the any two data to be stored; when the generation times of the any two data to be stored are both before the startup time of the application, it is determined that there is data correlation between the any two data to be stored.

[0263] In some embodiments, the above-mentioned identification module is also used to perform the following processing for any two data to be stored among the multiple data to be stored: calling the correlation identification model to predict the correlation between the any two data to be stored, and obtain the correlation between the any two data to be stored; when the correlation is less than the correlation threshold, determining that there is no data correlation between the any two data to be stored; when the correlation reaches the correlation threshold, determining that there is data correlation between the any two data to be stored.

[0264] In some embodiments, the above-mentioned construction module is also used to merge the data to be stored that have the data association among the multiple data to be stored in response to the identification result indicating that none of the multiple data to be stored are the target data to be stored, so as to obtain at least one merged data; when the number of the merged data is one, construct a first database based on each of the data to be stored in the merged data; when the number of the merged data is multiple, integrate each of the merged data to obtain merged integrated data, and construct a second database based on each sub-merged integrated data in the merged integrated data.

[0265] In some embodiments, the above-mentioned construction module is also used to extract features of each of the data to be stored to obtain data features of each of the data to be stored, and to encrypt each of the data to be stored to obtain encrypted data of each of the data to be stored; based on the data features of the data to be stored and the encrypted data of the data to be stored, a first database is constructed; the first database includes a first mapping relationship corresponding to each of the data to be stored, and the first mapping relationship uses the data features of the data to be stored as an index and the encrypted data of the data to be stored as data content.

[0266] In some embodiments, the above-mentioned integration module is also used to, in response to the recognition result indicating that each of the multiple data to be stored is the target data to be stored, integrate the target data to be stored to obtain the integrated data; in response to the recognition result indicating that some of the multiple data to be stored are the target data to be stored, merge the multiple data to be stored that have the data association to obtain at least one merged data; integrate the target data to be stored and the merged data to obtain the integrated data.

[0267] In some embodiments, the data to be stored includes at least one sub-data, and different sub-data in the data to be stored correspond to different data types. The above-mentioned integration module is also used to obtain indication information corresponding to the data type for each data type. When the indication information indicates that sub-data corresponding to the data type exists in at least part of the target data to be stored, the sub-integration data corresponding to the data type is determined based on the sub-data corresponding to the data type; and the integrated data is constructed based on the sub-integration data.

[0268] In some embodiments, the above-mentioned integrated module is also used to determine the data type as a target data type, and for each target data to be stored, compare the target data type with the data type of each sub-data in the target data to be stored to obtain a first comparison result corresponding to the target data to be stored; when the first comparison result indicates that sub-data of the target data type exists in the target data to be stored, the first comparison result is determined as the target comparison result; when the target comparison result exists, the indication information is determined as the sub-data corresponding to the data type exists in at least part of the target data to be stored.

[0269] In some embodiments, the above-mentioned integration module is also used to, when the number of sub-data corresponding to the data type is one, determine the sub-data corresponding to the data type as the sub-integrated data corresponding to the data type; when the number of sub-data corresponding to the data type is multiple, concatenate the sub-data corresponding to the data type to obtain the sub-integrated data corresponding to the data type.

[0270] In some embodiments, the above-mentioned integration module is also used to determine the sub-integration data as the integrated data when the number of the sub-integration data is one; when the number of the sub-integration data is multiple, construct the integrated data based on each of the sub-integration data.

[0271] In some embodiments, when the data to be stored is data generated during the operation of an application, the data to be stored includes multiple sub-data, and the sub-data in the data to be stored corresponds one-to-one to the running time of the application; the above-mentioned integration module is also used to splice the sub-data corresponding to the running time in each target data to be stored for each running time of the application, to obtain sub-integrated data corresponding to the running time; and the sub-integrated data corresponding to each running time is constructed as the integrated data.

[0272] In some embodiments, the above-mentioned integration module is also used to perform format identification on each of the target data to be stored to obtain the data format of each of the target data to be stored; for each of the target data to be stored, based on the data format of the target data to be stored and a preset data format, determine the candidate data corresponding to the target data to be stored, and the candidate data is in the preset data format; integrate each of the candidate data to obtain the integrated data.

[0273] In some embodiments, the above-mentioned integrated module is also used to compare the data format of the target data to be stored with the preset data format to obtain a second comparison result; when the second comparison result indicates that the data format of the target data to be stored is different from the preset data format, based on the preset data format, the target data to be stored is format converted to obtain the candidate data; when the second comparison result indicates that the data format of the target data to be stored is the same as the preset data format, the target data to be stored is determined as the candidate data.

[0274] In some embodiments, the preset data format is a text data format, and the above-mentioned integrated module is also used to extract features of the target data to be stored to obtain data features of the target data to be stored; call the text conversion model, and based on the data features of the target data to be stored, perform format conversion on the target data to be stored to obtain the candidate data.

[0275] In some embodiments, the above-mentioned database construction device also includes: a response module, which is used to respond to a data processing request carrying preset data, extract features from the preset data, and obtain data features of the preset data; determine from the target database a first data feature whose similarity with the data feature of the preset data reaches a similarity threshold, and determine from the mapping relationship a first encrypted data that has a mapping relationship with the first data feature; and respond to the data processing request based on the first encrypted data.

[0276] The embodiment of the present application provides a computer program product, which includes a computer program or a computer executable instruction, and the computer program or the computer executable instruction is stored in a computer-readable storage medium. The processor of the electronic device reads the computer executable instruction from the computer-readable storage medium, and the processor executes the computer executable instruction, so that the electronic device executes the database construction method described in the embodiment of the present application.

[0277] The embodiment of the present application provides a computer-readable storage medium storing computer-executable instructions, wherein the computer-executable instructions are stored. When the computer-executable instructions are executed by a processor, the processor will execute the method for constructing a database provided in the embodiment of the present application, for example, Figure 3 The method of constructing the database is shown.

[0278] In some embodiments, the computer-readable storage medium may be a memory such as FRAM, ROM, PROM, EPROM, EEPROM, flash memory, magnetic surface storage, optical disk, or CD-ROM; or it may be various electronic devices including one or any combination of the above memories.

[0279] In some embodiments, computer executable instructions may be in the form of a program, software, software module, script or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a stand-alone program or as a module, component, subroutine or other unit suitable for use in a computing environment.

[0280] As an example, computer-executable instructions may, but need not, correspond to a file in a file system, may be stored as part of a file storing other programs or data, such as in one or more scripts in a HyperText Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files storing one or more modules, subroutines, or code portions).

[0281] As an example, computer executable instructions may be deployed to be executed on one electronic device, or on multiple electronic devices located at one site, or on multiple electronic devices distributed at multiple sites and interconnected by a communication network.

[0282] In summary, the embodiments of the present application have the following beneficial effects:

[0283] (1) By acquiring multiple data to be stored, performing correlation identification on the multiple data to be stored, obtaining identification results, integrating the target data to be stored that do not have data correlation among the multiple data to be stored based on the identification results, obtaining integrated data, and constructing a target database based on the sub-integrated data. The target database includes mapping relationships corresponding to each sub-integrated data. The mapping relationships use the data features of the sub-integrated data as indexes and the encrypted data of each sub-integrated data as data content. In this way, by performing correlation identification on the multiple data to be stored, obtaining identification results, and integrating the target data to be stored that do not have data correlation among the multiple data to be stored based on the identification results, obtaining integrated data, the obtained integrated data can achieve unified management and analysis of the target data to be stored that do not have data correlation. By constructing a target database based on the sub-integrated data, the data that can be retrieved in the obtained target database is the integrated sub-integrated data. Since the sub-integrated data includes at least part of the data in each target data to be stored, the target database can achieve rapid retrieval of at least part of the data in each target data to be stored, thereby effectively improving the query performance of the determined target database. Since the data features corresponding to the sub-integration data can accurately reflect the data characteristics of the sub-integration data, and the data features corresponding to the sub-integration data are in vector form, the data features in vector form are not readable, while the sub-integration data are usually readable, therefore, the readable sub-integration data can be encrypted, and the unreadable data features are not encrypted, a mapping relationship between the encrypted data and the data features is established, and a target database is constructed based on the mapping relationship, thereby effectively improving the data security of the target database.

[0284] (2) Clarifying data relevance can help identify and eliminate duplicate data and ensure data consistency and accuracy. Relevance analysis helps to discover and correct inconsistencies and errors in data and improve the overall quality of data. Data relevance analysis helps to integrate data from different data sources and build a unified data view. The integrated data can be more easily analyzed and supports cross-data source queries and reports. By determining data relevance, effective relationships can be established between multiple data sources, thereby improving the efficiency and accuracy of data processing. Specifically, when the data sources are the same, the relevance between data can be identified, which helps to eliminate data redundancy and improve data quality and consistency. At the same time, when the data sources are different, by determining that there is no relevance between data, unnecessary data integration and processing can be avoided, saving time and resources. In addition, by identifying and utilizing data relevance, data analysis and decision making can be better supported, helping organizations make more accurate predictions and decisions. In general, by determining data relevance, the efficiency of data processing can be improved, the data storage structure can be optimized, and the accuracy of data analysis and decision making can be improved, thereby bringing significant business value.

[0285] (3) By determining whether there is a correlation between the data, the efficiency and accuracy of data processing can be effectively improved. Specifically, when the generation time of two sets of data to be stored is before and after the startup time of the application, it can be determined that there is no direct causal relationship or correlation between the two sets of data, thereby avoiding unnecessary data integration and analysis, saving time and resources. When the generation time of the two sets of data to be stored is both before the startup time of the application, it can be judged that there is a certain correlation between the two sets of data, which helps to analyze and explore the internal connection between the data and provide support for more in-depth data analysis and decision-making. In general, by analyzing and utilizing the information at the time of data generation, the efficiency of data processing can be effectively improved, the data storage structure can be optimized, and more accurate data analysis and decision-making can be supported.

[0286] (4) By calling the correlation identification model, the correlation between multiple data to be stored can be effectively predicted and evaluated, providing strong support for data analysis and decision-making. Specifically, when the correlation is less than the preset correlation threshold, it can be determined that there is no significant correlation between these data, thereby avoiding in-depth analysis and processing of these irrelevant data, saving time and resources. When the correlation reaches the correlation threshold, it indicates that there is a significant correlation between the data, and the inherent connection between these data can be further explored and utilized to provide strong support for decision-making, help discover new business opportunities and optimize existing processes. Through the application of the correlation identification model, the efficiency of data processing can be improved, the data storage structure can be optimized, and more accurate and in-depth data analysis can be supported.

[0287] (5) By calling the correlation identification model, the correlation between multiple data to be stored can be effectively predicted and evaluated, providing strong support for data analysis and decision-making. Specifically, when the correlation is less than the preset correlation threshold, it can be determined that there is no significant correlation between these data, thereby avoiding in-depth analysis and processing of these irrelevant data, saving time and resources. When the correlation reaches the correlation threshold, it indicates that there is a significant correlation between the data, and the inherent connection between these data can be further explored and utilized to provide strong support for decision-making, help discover new business opportunities and optimize existing processes. Through the application of the correlation identification model, the efficiency of data processing can be improved, the data storage structure can be optimized, and more accurate and in-depth data analysis can be supported.

[0288] (6) By screening multiple data to be stored through identification results, non-target data to be stored is effectively excluded. This process greatly improves the efficiency and accuracy of data processing. Secondly, the data to be stored with data association is merged, which not only optimizes the data structure, but also reduces data redundancy, helps save storage space and improves data processing speed. Different strategies are used to build the database according to the number of merged data. When the number of merged data is one, the first database is directly based on the data to be stored, which simplifies the construction process and reduces the complexity of the system. When the number of merged data is multiple, the merged integrated data is obtained by integrating each merged data, and then the second database is constructed, which helps to integrate diverse data resources and improve the comprehensive utilization of data.

[0289] (7) By extracting features from the data to be stored, a first data feature corresponding to the data to be stored is obtained, and data encryption is performed on the data to be stored to obtain first encrypted data corresponding to the data to be stored, and a mapping relationship between the first encrypted data and the first data feature is established, and a first database including the first encrypted data and the first data feature is constructed based on the mapping relationship. Since the first data feature corresponding to the data to be stored can accurately reflect the data characteristics of the data to be stored, and the first data feature corresponding to the data to be stored is in vector form, the data feature in vector form is not readable, while the data to be stored is usually readable, therefore, the readable data to be stored can be encrypted, and the unreadable first data feature is not encrypted, and a mapping relationship between the first encrypted data and the first data feature is established, and a first database including the first encrypted data and the first data feature is constructed based on the mapping relationship, thereby effectively improving the data security of data storage. At the same time, since the data feature in vector form is not readable and can accurately reflect the data characteristics of the data to be stored, and the data features in the first database are not encrypted, therefore, the first data feature in the first database can be used as an index to perform data operations on the data to be stored, which can achieve accurate operations on the data to be stored, thereby improving the data operation performance of the stored data. In this way, the data security of the database can be effectively improved while the data operation performance of the database can be improved.

[0290] (8) By obtaining corresponding indication information for different data types, the system can accurately identify and locate relevant sub-data in the target data to be stored, which greatly improves the pertinence and efficiency of data processing. Constructing sub-integrated data based on these sub-data helps to clearly identify and organize data, reduce data redundancy, and ensure the structuring and consistency of data. When there are multiple sub-integrated data, the system can effectively integrate them into integrated data, which not only promotes information fusion between different data types, but also provides a comprehensive data perspective, which helps to deeply analyze and mine data value. Such a data integration process can provide high-quality data resources, support decision-making, optimize business processes, and improve operational efficiency.

[0291] (9) It ensures the targeted nature of data processing. By accurately matching the target data type and sub-data type, it effectively identifies and screens out relevant data, thereby improving the accuracy and efficiency of data processing. Through the first comparison result obtained, it can quickly determine whether there is sub-data of the target data type in the target data to be stored, thereby reducing unnecessary data processing steps and saving computing resources and time. When there is a target comparison result, the process of confirming the indication information provides a reliable basis for data analysis and subsequent processing, making data integration and utilization more efficient and purposeful.

[0292] (10) When the number of sub-data corresponding to a data type is one, it is directly determined as sub-integrated data. The beneficial effect is that it simplifies the data processing process, reduces the complexity of integration, and ensures the consistency and integrity of the data. When the number of sub-data is multiple, they are merged into sub-integrated data by splicing. This approach not only improves the comprehensive utilization efficiency of data, but also helps to eliminate data islands, enhance the correlation between data, and provide more abundant and comprehensive information support for subsequent data analysis and decision-making.

[0293] (11) When there is only one sub-integrated data, it is directly identified as integrated data. This processing method is beneficial to reduce data processing steps, reduce system complexity and maintenance costs, and ensure the directness and efficiency of data. When there are multiple sub-integrated data, the final integrated data is formed by constructing and integrating these sub-integrated data, which helps to achieve comprehensive data integration, improve the overall value and utilization efficiency of data, and provide a richer and more accurate information basis for data analysis and decision-making.

[0294] (12) By processing the target data to be stored at the runtime of the application, the data subsets at each moment can be spliced ​​together to form complete runtime subset integrated data. Such operations help to more deeply understand and analyze the running status of the program, including performance, resource usage, etc. In this way, comprehensive integrated data can be constructed to provide strong support for the optimization, detection and maintenance of the application. This not only helps to improve the quality and stability of the application.

[0295] (13) By identifying the format of the target data to be stored, it is possible to ensure accurate understanding and processing of the data, so that when processing data from different sources and formats, they can be uniformly converted into the preset data format. The data can be analyzed and integrated more conveniently to obtain complete integrated data. On this basis, for each target data to be stored, candidate data can be determined based on its original data format and the preset data format. These candidate data are data fragments converted according to the preset data format, which will be used to construct integrated data. Finally, by integrating the candidate data, unified integrated data can be obtained. This integrated data set will contain the key information of all target data to be stored for further analysis and processing. In this way, not only the efficiency of data processing is improved, but also the integrity and accuracy of the data are ensured, providing strong support for the optimization, detection and maintenance of applications.

[0296] (14) By extracting the features of the target data to be stored, the key features of the data can be obtained. These features are crucial for subsequent data analysis and processing, and can help reduce the dimension of the data, remove redundant information, and retain key information that is useful for specific tasks. In this way, data can be used more efficiently and the performance and accuracy of the model can be improved. After extracting the data features of the target data to be stored, the text conversion model is called to convert the format of the data. The text data can be converted into a unified format suitable for machine learning algorithms, thereby simplifying the complexity of data processing and analysis. The use of the text conversion model ensures the consistency and compatibility of the data, so that the data can be effectively used for various text analysis tasks. Through the application of feature extraction and text conversion models, candidate data is obtained. The candidate data not only contains the key features of the original data, but also is presented in a format suitable for machine learning algorithm processing. This provides strong support for subsequent data analysis and application, improves the efficiency and accuracy of data processing, and also provides a reliable foundation for data-driven decision-making.

[0297] (15) When processing multiple data to be stored, the recognition results are used to determine whether the data contain the target data to be stored. This can help distinguish which data are target data and which are not, so as to carry out subsequent processing in a targeted manner. When the recognition results show that all the data to be stored are target data, these data can be directly integrated to obtain complete integrated data, saving time and resources for data processing, while also ensuring the integrity and consistency of the data. When the recognition results show that the data to be stored contain non-target data, those data to be stored that have data associations can be merged. This can help us integrate related data so that we can better use these data for analysis and processing. Integrate the target data to be stored and the merged data to obtain the final integrated data. This achieves unified management and analysis of data from different sources, improving data utilization and value.

[0298] (16) The database construction method provided in the embodiment of the present application combines the construction process of the private knowledge base with the traditional big data processing ETL technology, and integrates the entire data link: data access, data segmentation, data vectorization processing (Embedding) and data scoring into the stream computing engine. At the same time, by adopting a distributed computing framework to support the Embedding service and LLM service, the amount of data processed is increased to the TB level, and the scale of business parties connected to the platform is increased by an order of magnitude, which significantly improves the ease of use and improves the stability of the system by more than 90%. Use the DataEvaluator Service to rate and score the data and evaluate the data quality, so that the quality of the retrieved corpus content is improved by more than 20%.

[0299] The above is only an embodiment of the present application and is not intended to limit the protection scope of the present application. Any modifications, equivalent substitutions and improvements made within the spirit and scope of the present application are included in the protection scope of the present application.

Claims

1. A method for constructing a database, characterized in that: The method comprises: Acquire multiple data to be stored, perform correlation identification on the multiple data to be stored, and obtain an identification result; In response to the identification result indicating that some of the data to be stored in the plurality of data to be stored are target data to be stored without data association, merging the data to be stored with the data association in the plurality of data to be stored to obtain at least one merged data, the target data to be stored includes at least one sub-data, the merged data includes at least one sub-data, and the target data to be stored and the different sub-data in the merged data correspond to different data types; For each of the data types, obtaining indication information corresponding to the data type, and when the indication information indicates that sub-data corresponding to the data type exists in at least part of the target data to be stored, or sub-data corresponding to the data type exists in at least part of the merged data, determining the sub-integrated data corresponding to the data type based on the sub-data corresponding to the data type; Feature extraction is performed on each of the sub-integration data to obtain data features of each of the sub-integration data, and a target database is constructed based on each of the sub-integration data and the data features. The target database includes mapping relationships corresponding to each of the sub-integration data, and the mapping relationships use the data features of the sub-integration data as indexes and the encrypted data of each of the sub-integration data as data content.

2. The method according to claim 1, characterized in that The step of performing correlation identification on the plurality of data to be stored to obtain an identification result includes: Obtaining the data source of each of the data to be stored; For any two data to be stored among the multiple data to be stored, when the data sources of the any two data to be stored are the same, it is determined that there is data correlation between the any two data to be stored; when the data sources of the any two data to be stored are different, it is determined that there is no data correlation between the any two data to be stored.

3. The method according to claim 1, characterized in that When the data to be stored is data in an application, the performing of correlation identification on the plurality of data to be stored to obtain an identification result includes: Obtaining the generation time of each of the data to be stored in the application program; For any two of the multiple data to be stored, when the generation times of the any two data to be stored are respectively before and after the startup time of the application, it is determined that there is no data correlation between the any two data to be stored; when the generation times of the any two data to be stored are both before the startup time of the application, it is determined that there is data correlation between the any two data to be stored.

4. The method according to claim 1, characterized in that: The step of performing correlation identification on the plurality of data to be stored to obtain an identification result includes: The following processing is performed for any two data to be stored among the plurality of data to be stored: Calling a correlation recognition model to predict the correlation between any two data to be stored, and obtaining the correlation between any two data to be stored; When the correlation degree is less than the correlation degree threshold, determining that there is no data correlation between the two data to be stored; When the correlation degree reaches the correlation degree threshold, it is determined that there is data correlation between the any two data to be stored.

5. The method according to claim 1, characterized in that After the association of the plurality of data to be stored is identified and an identification result is obtained, the method further includes: In response to the identification result indicating that none of the plurality of data to be stored is the target data to be stored, merging the data to be stored that have the data association among the plurality of data to be stored to obtain at least one merged data; When the number of the merged data is one, constructing a first database based on each of the to-be-stored data in the merged data; When there are multiple merged data, each merged data is integrated to obtain merged integrated data, and a second database is constructed based on each sub-merged integrated data in the merged integrated data.

6. The method according to claim 5, characterized in that The step of constructing a first database based on each of the to-be-stored data in the merged data includes: Extracting features of each of the data to be stored to obtain data features of each of the data to be stored, and encrypting each of the data to be stored to obtain encrypted data of each of the data to be stored; constructing the first database based on the data characteristics of the data to be stored and the encrypted data of the data to be stored; The first database includes a first mapping relationship corresponding to each of the data to be stored, wherein the first mapping relationship uses the data feature of the data to be stored as an index and uses the encrypted data of the data to be stored as data content.

7. The method according to claim 1, characterized in that After the association of the plurality of data to be stored is identified and an identification result is obtained, the method further includes: In response to the identification result indicating that each of the plurality of data to be stored is the target data to be stored, each of the target data to be stored is integrated to obtain integrated data.

8. The method according to claim 7, characterized in that The data to be stored includes at least one sub-data, different sub-data in the data to be stored correspond to different data types, and the integrating the target data to be stored to obtain the integrated data includes: For each of the data types, obtaining indication information corresponding to the data type, and when the indication information indicates that sub-data corresponding to the data type exists in at least part of the target data to be stored, determining sub-integrated data corresponding to the data type based on the sub-data corresponding to the data type; The integrated data is constructed based on the sub-integrated data.

9. The method according to claim 8, characterized in that The obtaining indication information corresponding to the data type includes: Determine the data type as a target data type, and for each target data to be stored, compare the target data type with the data type of each sub-data in the target data to be stored to obtain a first comparison result corresponding to the target data to be stored, and when the first comparison result indicates that sub-data of the target data type exists in the target data to be stored, determine the first comparison result as a target comparison result; When the target comparison result exists, the indication information is determined as sub-data corresponding to the data type existing in at least part of the target data to be stored.

10. The method according to claim 8, characterized in that The determining, based on the sub-data corresponding to the data type, the subset data corresponding to the data type comprises: When the number of sub-data corresponding to the data type is one, determining the sub-data corresponding to the data type as the subset data corresponding to the data type; When the number of sub-data corresponding to the data type is multiple, the sub-data corresponding to the data type are concatenated to obtain subset data corresponding to the data type.

11. The method according to claim 8, characterized in that The step of constructing the integrated data based on the sub-integrated data includes: When the number of the sub-integration data is one, determining the sub-integration data as the integration data; When there are multiple sub-integration data, the integration data is constructed based on each of the sub-integration data.

12. The method according to claim 7, characterized in that When the data to be stored is data generated during the running of the application, the data to be stored includes a plurality of sub-data, and the sub-data in the data to be stored corresponds one-to-one to the running time of the application; The step of integrating the target data to be stored to obtain the integrated data includes: For each of the running moments of the application, concatenate the sub-data corresponding to the running moment in each of the target data to be stored to obtain the sub-integrated data corresponding to the running moment; The sub-integrated data corresponding to each of the running moments are constructed into the integrated data.

13. The method according to claim 7, characterized in that The step of integrating the target data to be stored to obtain integrated data includes: Performing format identification on each of the target data to be stored to obtain the data format of each of the target data to be stored; For each of the target data to be stored, based on the data format of the target data to be stored and a preset data format, determining candidate data corresponding to the target data to be stored, the candidate data being in the preset data format; The candidate data are integrated to obtain the integrated data.

14. The method according to claim 13, characterized in that The determining, based on the data format of the target data to be stored and the preset data format, candidate data corresponding to the target data to be stored includes: Comparing the data format of the target data to be stored with the preset data format to obtain a second comparison result; When the second comparison result indicates that the data format of the target data to be stored is different from the preset data format, performing format conversion on the target data to be stored based on the preset data format to obtain the candidate data; When the second comparison result indicates that the data format of the target data to be stored is the same as the preset data format, the target data to be stored is determined as the candidate data.

15. The method according to claim 14, characterized in that The preset data format is a text data format, and based on the preset data format, the target data to be stored is converted into a format to obtain the candidate data, including: Extracting features of the target data to be stored to obtain data features of the target data to be stored; The text conversion model is called to convert the format of the target data to be stored based on the data features of the target data to be stored, so as to obtain the candidate data.

16. The method according to claim 1, characterized in that After constructing the target database based on each of the sub-integration data and the data features, the method further includes: In response to a data processing request carrying preset data, extracting features of the preset data to obtain data features of the preset data; Determine, from the target database, a first data feature whose similarity with the data feature of the preset data reaches a similarity threshold, and determine, from the mapping relationship, first encrypted data that has a mapping relationship with the first data feature; The data processing request is responded to based on the first encrypted data.

17. A database construction device, characterized in that: The device comprises: An identification module is used to obtain a plurality of data to be stored, perform correlation identification on the plurality of data to be stored, and obtain an identification result; An integration module is used for merging the data to be stored that have the data association among the multiple data to be stored in response to the identification result indicating that some of the data to be stored are target data to be stored without data association, so as to obtain at least one merged data, wherein the target data to be stored includes at least one sub-data, and the merged data includes at least one sub-data, and the target data to be stored and the merged data have different data types corresponding to the sub-data; for each data type, obtaining indication information corresponding to the data type, and when the indication information indicates that at least some of the target data to be stored have sub-data corresponding to the data type, or at least some of the merged data have sub-data corresponding to the data type, determining the sub-integrated data corresponding to the data type based on the sub-data corresponding to the data type; A construction module is used to extract features from each of the sub-integration data to obtain data features of each of the sub-integration data, and to construct a target database based on each of the sub-integration data and the data features. The target database includes mapping relationships corresponding to each of the sub-integration data, and the mapping relationships use the data features of the sub-integration data as indexes and the encrypted data of each of the sub-integration data as data content.

18. An electronic device, characterized in that: The electronic device comprises: A memory for storing computer executable instructions or computer programs; A processor, configured to implement the method for constructing a database according to any one of claims 1 to 16 when executing computer executable instructions or computer programs stored in the memory.

19. A computer-readable storage medium storing computer-executable instructions or a computer program, characterized in that: When the computer executable instructions or the computer program are executed by a processor, the method for constructing a database according to any one of claims 1 to 16 is implemented.

20. A computer program product comprising a computer program or computer executable instructions, characterized in that When the computer program or computer executable instructions are executed by a processor, the method for constructing a database according to any one of claims 1 to 16 is implemented.

Images