Multi-source heterogeneous data processing method and device, storage medium and program product

Abstract

The embodiment of the invention discloses a multi-source heterogeneous data processing method and device, a storage medium and a program product, and belongs to the field of data processing. The multi-source heterogeneous data processing method comprises the following steps: acquiring to-be-stored data; according to the data structure type of the to-be-stored data, selecting a corresponding preset model to perform key information extraction on the to-be-stored data to obtain key data of multiple dimensions; enabling the dimensions of all the key data to form a label group; and associating and inputting the tag group and the key data into a preset retrieval table, and storing the to-be-stored data. According to the method, the to-be-stored data can be retrieved only when the tag group and the key data are consistent in the subsequent retrieval process, so that the retrieval precision of the to-be-stored data is improved, and the retrieval difficulty is reduced.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of data processing, in particular to a multi-source heterogeneous data processing method, device, storage medium and program product. BACKGROUND

[0002] With the explosive growth of global Internet information, information of various industries will appear on the Internet in different structural forms, such as text, image, table and the like. When saving data of a certain industry, if not sorted, it will greatly increase the difficulty of subsequent data retrieval. SUMMARY

[0003] The embodiments of the present application provide a multi-source heterogeneous data processing method, device, storage medium and program product to at least solve the technical problem of great data retrieval difficulty.

[0004] According to a first aspect of the embodiments of the present application, a multi-source heterogeneous data processing method is provided, which comprises: obtaining to-be-stored data; selecting a corresponding preset model according to a data structure type of the to-be-stored data to extract key information of the to-be-stored data, to obtain key data of multiple dimensions, wherein the data structure type comprises at least one of text, image and table; forming a label group from dimensions of all the key data; associating the label group and the key data to enter a preset retrieval table and saving the to-be-stored data.

[0005] In the present embodiment, when saving to-be-stored data, key data of the to-be-stored data is extracted and dimensions of the key data are obtained, so as to generate a label group for multiple dimensions. In this way, the to-be-stored data can be constrained from two aspects of dimensions and data, so that in the subsequent retrieval process, only when the label group and the key data are both met, the to-be-stored data can be retrieved, the retrieval accuracy of the to-be-stored data is improved, and the retrieval difficulty is reduced.

[0006] In combination with the first aspect, in an optional implementation manner of the embodiments of the present application, the selecting a corresponding preset model according to a data structure type of the to-be-stored data to extract key information of the to-be-stored data, to obtain key data of multiple dimensions comprises: if the data structure type of the to-be-stored data supports direct extraction of the key data, a large language model is used for extraction, otherwise an image recognition model is used for extraction.

[0007] According to the implementation manner, the to-be-stored data of the large language model can be directly used for key information extraction using the large language model, which is beneficial to improve the accuracy of key information extraction. The to-be-stored data that cannot use or is not suitable for using the large language model is extracted using the image recognition model, which is beneficial to ensure the stable acquisition of key data.

[0008] With reference to the first aspect, in an optional implementation manner of the embodiments of the present application, the method further includes: determining whether the search table already has a storage area identical to the label group; if yes, storing the key data into the corresponding storage area; if no, generating a new storage area in the search table according to the label group and storing the key data into the new storage area; wherein the search table includes a plurality of storage areas, and different storage areas correspond to different label groups.

[0009] According to the implementation manner, when the label group and the key data are associated and stored, the storage area is determined according to the label group. For the case that the existing storage area cannot be found, a new storage area is directly generated, which is beneficial to the automatic update and automatic expansion of the label group, improves the accuracy and convenience of subsequent search, and avoids the case that the search result cannot be obtained because the label group cannot be found.

[0010] With reference to the first aspect, in an optional implementation manner of the embodiments of the present application, the method further includes: if the to-be-stored data belongs to structured data, storing the to-be-stored data into a relational database, otherwise, performing object storage on the to-be-stored data.

[0011] According to the implementation manner, storing the structured data into the relational database can improve the search convenience and search accuracy of the structured data. The to-be-stored data of unstructured data is stored by object storage, which is also beneficial to the subsequent search convenience and can reduce the search difficulty.

[0012] With reference to the first aspect, in an optional implementation manner of the embodiments of the present application, the method further includes: in response to a search request, extracting a search keyword in the search request and a dimension of the search keyword; determining a target storage area from the search table according to the dimension of the search keyword; determining the key data from the target storage area according to the search keyword; returning the data associated with the key data as a search result.

[0013] According to the implementation manner, the retrieval is performed according to the dimensions and the keywords simultaneously in the retrieval process, and the retrieval result is determined, so that the accuracy of the retrieval result is improved.

[0014] With reference to the first aspect, in an optional implementation manner of the embodiments of the present application, the retrieval result further includes generation information of the retrieval keyword.

[0015] According to the implementation manner, the generation information of the retrieval keyword is convenient for verifying the generation accuracy of the retrieval keyword. Only when the retrieval keyword is correctly generated, the accuracy of the retrieval result is ensured. When the retrieval result deviates from the user demand, the accuracy of the retrieval keyword can be checked according to the generation information.

[0016] With reference to the first aspect, in an optional implementation manner of the embodiments of the present application, the method further includes: According to the retrieval frequency of each data, the data with a retrieval frequency exceeding a preset frequency threshold is stored in a cache space.

[0017] According to the implementation manner, the data with a greater retrieval frequency is stored separately, which is beneficial to improving the retrieval efficiency.

[0018] According to the second aspect of the embodiments of the present application, an electronic device is provided, and the electronic device includes a memory and a processor. The memory is configured to store a computer program. The processor is configured to execute the computer program to implement the steps of the method.

[0019] According to the third aspect of the embodiments of the present application, a computer readable storage medium is provided, and the computer readable storage medium stores a computer program. When the computer program runs on a computer or a processor, the computer or the processor executes the steps of the method.

[0020] According to the fourth aspect of the embodiments of the present application, a computer program product is provided, and the computer program product includes computer instructions. When the computer instructions are executed by a computer or a processor, the steps of the method are executed.

[0021] The technical effects obtained by the second aspect to the fourth aspect are similar to the technical effects obtained by the corresponding technical means in the first aspect, and will not be described here. BRIEF DESCRIPTION OF DRAWINGS

[0022] Figure 1 is a flowchart of a multi-source heterogeneous data processing method provided by the embodiments of the present application; Figure 2is an architecture diagram of a multi-source heterogeneous data processing system in a specific application provided by an embodiment of the present application. Figure 3 is a retrieval flowchart of a multi-source heterogeneous data processing system in a specific application provided by an embodiment of the present application. Figure 4 is a structural block diagram of an electronic device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0023] In order to enable persons skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by persons skilled in the art without creative labor should fall within the scope of protection of the present application.

[0024] It should be understood that "multiple" referred to herein means two or more. In the description of the embodiments of the present application, unless otherwise specified, " / " represents the meaning of or, for example, A / B can represent A or B; "and / or" herein is only a description of the association relationship of the associated objects, which means that there can be three relationships, for example, A and / or B can represent the three cases of A alone, A and B together, and B alone. In addition, in order to clearly describe the technical solutions of the embodiments of the present application, in the embodiments of the present application, the same items or similar items with basically the same functions and effects are distinguished by using "first", "second", etc. The person skilled in the art can understand that "first", "second", etc. do not limit the quantity and execution order, and "first", "second", etc. also do not necessarily mean different.

[0025] In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device containing a series of steps or units does not have to be limited to only those steps or units clearly listed, but can include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0026] First, the terms related to the embodiments of the present application are introduced.

[0027] Object storage, also known as object-based storage, is a general term used to describe a method for solving and processing discrete units, which are referred to as objects.

[0028] Like files, objects contain data, but unlike files, objects do not have a hierarchy in a tiered structure. Each object is at the same level in a flat address space called a storage pool, and one object does not belong to the next level of another object.

[0029] Both files and objects have metadata associated with the data they contain, but objects are characterized by extended metadata. Each object is assigned a unique identifier that allows a server or end user to retrieve the object without having to know the physical address of the data. This approach helps automate and simplify data storage in a cloud computing environment.

[0030] With the explosive growth of global Internet information, information in various industries will appear on the Internet in different structural forms. For example, financial information, current financial institutions need to process data from multiple channels, such as: Financial report data (unstructured formats such as PDF, WORD), news information (HTML or text), real-time market data (structured data such as CSV, JSON), and emerging multi-modal data such as ESG, public opinion, and on-chain data.

[0031] These data have the following characteristics: Strong data heterogeneity: difficult to structure and tag with uniform rules; Static and lagging tag system: unable to automatically expand to cope with new data types and index dimensions; Low storage and access efficiency: mixed storage of structured and unstructured data, high query delay, and difficult to efficiently retrieve by tags and dimensions.

[0032] Some current financial companies have certain data processing capabilities, but mostly rely on rule engines or manual tagging, and lack a full-process intelligent tagging and adaptive storage system for heterogeneous data.

[0033] Therefore, the embodiments of the present application provide a multi-source heterogeneous data processing method, which has at least one of the following characteristics: 1. Automatic label extraction for multi-source financial data.

[0034] 2. Dynamic label system expansion capability.

[0035] 3. Layered storage structure design driven by labels.

[0036] 4. Semantic enhanced retrieval capability with mixed natural language and structured labels.

[0037] Next, the multi-source heterogeneous data processing method provided by the embodiments will be further described with reference to Figure 1The flowchart shown is a multi-source heterogeneous data processing method, which includes the following processing steps.

[0038] S100, Obtain the data to be stored.

[0039] The data to be stored refers to the data that needs to be saved to the database. The source of the data to be stored is not specifically limited in this embodiment. It can be obtained by web crawlers or actively transmitted by humans or smart devices.

[0040] It should be noted that there are many sources and ways to obtain the data to be stored, so the data structure types of different data to be stored may be different. In order to facilitate subsequent retrieval, the data to be stored can be preprocessed before saving, such as cleaning, parsing and format standardization.

[0041] S102. Select the corresponding preset model according to the data structure type of the data to be stored to extract key information from the data to be stored, and obtain key data in multiple dimensions.

[0042] The data structure type includes at least one of text, image, and table.

[0043] In one embodiment, different models can handle different objects. That is, when the data structure type of the data to be stored is different, it is necessary to select a model that can process it in order to correctly obtain the key data. For example, a large language model (GPT, BERT, etc.) can be used to extract key information from text-based data to be stored, resulting in key data in multiple dimensions.

[0044] For ease of understanding, text, images, and tables are all carriers of information, or rather, forms of information representation. Therefore, after processing by a pre-defined model, the information that the data to be stored reflects can be obtained. Within this information lies key information; extracting this key information yields key data, and different key data reflect different dimensions. For example, after using an image recognition model such as CNN to analyze an image, the image expresses the message: Company A will be delisted in 5 years. The key data includes Company A, 5 years later, and delisting. The dimension for Company A is its name, the dimension for 5 years later is time, and the dimension for delisting is the event.

[0045] Based on the above explanation, it can be understood that different data to be stored contain different key data and the dimensions of the key data are different. In order to improve the convenience and accuracy of subsequent retrieval, when training the preset model, the preset model can generate new dimensions on the basis of the existing dimensions. The specific training method can refer to the training process of existing neural network models. For example, the training data containing new dimensions can be labeled and then fed into the training of the preset model, so that the preset model can identify and generate new dimensions.

[0046] S104. Form a label group for all the dimensions of the key data.

[0047] In this context, a tag group is essentially a combination of dimensions. For example, if the dimensions of all the key data of a certain data to be stored include name, time, and event, then the tag group of the data to be stored is: name + time + event. It should be noted that the number of key data that can be extracted from different data to be stored is different. Therefore, the dimensions of the tag groups of different data to be stored can be different, and the number of dimensions may also be different. This results in some tag groups including two dimensions, while some tag groups include more or fewer dimensions.

[0048] S106. Associate the tag group and key data into a preset search table and save the data to be stored.

[0049] The search table is a preset table used to store tag groups and key data. The corresponding data (saved data to be stored) can be directly retrieved by searching the tag groups and key data in the search table.

[0050] In this embodiment, when saving the data to be stored, the key data of the data to be stored is extracted and the dimensions of the key data are obtained. Tag groups are then generated for multiple dimensions. This constrains the data to be stored from both the dimensions and the data, so that in the subsequent retrieval process, the data to be stored can only be retrieved when both the tag groups and the key data are consistent. This improves the retrieval accuracy of the data to be stored and reduces the retrieval difficulty.

[0051] In one possible embodiment of this application, the step of selecting a corresponding preset model based on the data structure type of the data to be stored to extract key information from the data to be stored, thereby obtaining key data in multiple dimensions, includes: If the data structure type of the data to be stored supports the direct extraction of key data, then a large language model is used for extraction; otherwise, an image recognition model is used for extraction.

[0052] Whether the data structure type of the data to be stored supports the direct extraction of key data can be determined based on the data structure type that the large language model used can handle. For example, if the large language model used is the GPT model, which can directly extract key data from text, PDF, WORD and PPT, then the data to be stored whose data structure type meets the above type is processed by the large language model, while the data to be stored whose data structure type is other is processed by the image recognition model.

[0053] It should be noted that image recognition models refer to models capable of performing recognition and other processing on images, aiming to extract key data from images. One scenario involves images containing identifiable text; in this case, the image recognition model is used to obtain the text, and then key data is extracted. Another scenario involves images without identifiable text. In this case, the image can be discarded. However, if a piece of text can be extracted from the image's content, it can be retained; this decision depends on the capabilities of the available preset models.

[0054] This implementation method allows for the direct extraction of key information from data to be stored using a large language model, which improves the accuracy of key information extraction. For data to be stored that cannot be stored using a large language model or is not suitable for using a large language model, an image recognition model is used for extraction, which helps ensure the stable acquisition of key data.

[0055] Optionally, in one implementation of this embodiment, the step of associating the tag group and key data into a preset search table includes: Determine whether the search table already contains a storage area identical to the tag group; If so, the key data will be stored in the corresponding storage area; If not, a new storage area is generated in the search table according to the tag group, and the key data is stored in the new storage area; The search table includes multiple storage areas, and different storage areas correspond to different tag groups.

[0056] In one embodiment, the retrieval table has multiple storage areas, and different storage areas correspond to different tag groups. When the tag group of the data to be stored is obtained, the same tag group is searched in the retrieval table. If it is found, the key data of the data to be stored is saved in the storage area. If it is not found, a new storage area is generated.

[0057] For ease of understanding, let's consider a table where storage area A has the tag group "Name + Event," and storage area B has the tag group "Name + Event + Event." If the tag group for the data to be stored is "Name + Event," then the key data of the data to be stored is saved in storage area A. This ensures that during subsequent searches, if the search request only contains the dimensions "Name" and "Event," the key data can be retrieved directly from storage area A, making it convenient, fast, and accurate. If the tag group for the data to be stored is "Name + Sentiment + Event," then a new storage area C is generated to store the key data of the data to be stored.

[0058] Using this implementation method, when storing tag groups and key data in association, the storage area is first determined based on the tag group. If no existing storage area is found, a new storage area will be generated directly. This facilitates the automatic updating and expansion of tag groups, improves the accuracy and convenience of subsequent searches, and avoids situations where search results cannot be obtained due to the inability to find tag groups.

[0059] Optionally, in one implementation of this embodiment, saving the data to be stored includes: If the data to be stored is structured data, it is stored in a relational database; otherwise, it is stored as an object.

[0060] In one embodiment, structured data refers to data with structures such as CSV and JSON.

[0061] This implementation method improves the ease and accuracy of retrieval of structured data by storing it in a relational database. Unstructured data is stored as objects, which also enhances retrieval convenience and reduces search difficulty.

[0062] Optionally, in one implementation of this embodiment, the method further includes: In response to a search request, extract the search keywords and dimensions of the search keywords from the search request; The target storage area is determined from the search table based on the dimensions of the search keywords; The key data is determined from the target storage area based on the search keywords; The data associated with the key data will be returned as search results.

[0063] It should be noted that, in addition to using dimensions and keywords for retrieval, semantic vector retrieval can also be combined to make the retrieval results more accurate.

[0064] By adopting this implementation method, the search will be performed simultaneously based on dimensions and keywords during the retrieval process to determine the search results, which can improve the accuracy of the search results.

[0065] Optionally, in one implementation of this embodiment, the search results may further include information on the generation of search keywords.

[0066] By adopting this implementation method, the generated information of search keywords can be used to verify the accuracy of the generated search keywords. Only when the search keywords can be generated correctly can the accuracy of the search results be guaranteed. This also makes it easier to check the accuracy of the search keywords based on the generated information when there is a discrepancy between the search results and the user's needs.

[0067] Optionally, in one implementation of this embodiment, the method further includes: Based on the retrieval frequency of each data item, data whose retrieval frequency exceeds a preset frequency threshold are stored in the cache space.

[0068] The frequency threshold can be set by the user, and this embodiment does not impose any limitations.

[0069] By using this implementation method, data with higher retrieval frequency is stored separately, which helps to improve retrieval efficiency.

[0070] In the above embodiments of this application, the descriptions of each embodiment have their own emphasis. Parts not described in detail in a certain embodiment can be referred to in the relevant descriptions of other embodiments. The steps illustrated in the related flowcharts can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowcharts, in some cases, the steps shown or described may be performed in a different order than that shown here. In other words, the order of steps described in the foregoing embodiments is merely an example. Reasonable adjustments to the order of steps based on the content of the embodiments of this application are also within the protection scope of the embodiments of this application.

[0071] In one specific implementation of the embodiments of this application, such as Figures 2-3 As shown, the method of this embodiment is based on the following system: The system includes the following core modules: (1) Data preprocessing and standardization module, Used for: Unified access to both structured and unstructured data sources; Perform basic cleaning, parsing, and format standardization on text, images, and tables; (2) Large model-driven label extraction and feature recognition module, Used for: Utilize large language models (such as ChatGPT, BERT) or self-developed models to perform semantic understanding of text content; Automatically generate tags that include dimensions such as domain concepts, sentiment tendencies, and financial indicators; The tagging system is "scalable" and can be dynamically iterated through a feedback mechanism; (3) Tag-driven hierarchical storage module, Used for: Categorize and index the data according to tag dimensions; Configure a caching layer (such as Redis) or hot data partitioning for high-frequency or high-value tags; It supports storing structured data in relational databases (such as PostgreSQL) and unstructured text / PDFs in object storage (such as MinIO). The stored metadata includes a tag index, which improves query speed; (4) Enhanced search engine (Intelligent Semantic Retrieval Engine, ISRE). Used for: Supports Boolean tag combination search (e.g., "Industry=New Energy" AND "Sentiment=Bullish"); Connect to semantic vector search engines (such as FAISS or ES vector extension plugins) to support natural language queries; Multimodal retrieval support (simultaneous retrieval of text, images, and charts); Real-time tag recommendations and query builder assist users in combining complex queries; (5) Index adaptive update and interpretability mechanism, Used for: New data is automatically updated and indexed upon being entered into the database. The search results provide the tag generation path and tagging basis, improving auditability.

[0072] Specifically, such as Figure 3 As shown, during use, the system first performs a natural language or structured query on the user's query request, and then extracts keywords, structured data, and time data through the query parsing module. If tags can be directly determined from the query request, the tag retrieval module will perform tag tree and inverted index. If semantic vectors are obtained from the query request, the semantic vector retrieval module will perform indexing through an embedded vector library (FAISS, etc.). If both tags and semantic vectors are obtained, or if indexing cannot be performed based on tags or semantic vectors, a multimodal result fusion index will be performed, and finally, the results will be displayed.

[0073] Example 1: Tagging and Storage of Multi-Source Financial Data Access a batch of financial report PDFs and market data CSV files; The system automatically extracts text, table, and image information; Generate tags such as sentiment, industry, and indicators for text, and identify fields such as net profit and growth rate for tables; Based on tags, hierarchical storage and index construction are performed; It can be queried by downstream risk control and investment research systems; Example 2: Natural Language Retrieval of Sentiment Analysis Articles: The user entered: "Find positive news about AI chips in the past month"; The system analyzes the query intent and time range; By combining vector recall with tag filtering, news items labeled "AI chip", "bullish", and "last 30 days" are accurately returned; Example 3: Integrated image and text retrieval: User search: "Negative reports containing Apple's earnings chart"; The system matches PDF or image / text data with labels such as "Apple Inc.", "Revenue" charts, and "Sentiment = Negative" and returns them uniformly.

[0074] The system provided in this embodiment has the following characteristics: An automated label generation mechanism based on a large model; The tag system evolves dynamically (supports automatic modeling and summarization of newly added tags). Tag-driven hierarchical storage structure (caching, hot and cold partitions, etc.); Unified labeling and semantic retrieval capabilities for multimodal data; Enhanced retrieval capabilities through the fusion of tag and vector recall; Automatic maintenance and self-updating mechanism for tag index.

[0075] It can improve marking efficiency: It automatically extracts multi-layer labels, which is more than 10 times more efficient than traditional manual marking; Enhance system scalability: Support seamless integration of new data and concepts through an evolving tag tree; Improve query efficiency: Query latency reduced from seconds to hundreds of milliseconds; Supports semantic search: Natural language input can retrieve relevant heterogeneous data, improving user experience; Improve the utilization rate of data value: The tagging structure enhances the data reuse capability and can be used by downstream systems such as investment research, risk control, and public opinion systems.

[0076] The above examples illustrate the method embodiments according to this application. The present invention also provides an electronic device, which includes a memory and a processor. The memory is used to store computer programs; The processor is used to execute the computer program to implement the steps of the method described above.

[0077] This invention also provides a computer-readable storage medium storing a computer program that, when run on a computer or processor, causes the computer or processor to perform the steps of the method described above.

[0078] This invention also provides a computer program product comprising computer instructions that, when executed by a computer or processor, cause the steps of the method described above to be performed.

[0079] Specifically, such as Figure 4 As shown, the electronic device includes a processor 100, at least one communication bus 200, a user interface 300, at least one external communication interface 400, and a memory 500. The communication bus 200 is configured to enable communication between these components. The user interface 300 may include a display screen, and the external communication interface 400 may include standard wired and wireless interfaces. The memory 500 stores multi-source heterogeneous data processing methods. The processor 100 is used to employ these methods when executing the multi-source heterogeneous data processing methods stored in the memory 500.

[0080] The descriptions of the above computer program products, computer-readable storage media, and electronic devices are similar to those of the above method embodiments, and have similar beneficial effects. For any technical details not disclosed in the computer program products, computer-readable storage media, and electronic devices of this application, please refer to the descriptions of the method embodiments of this application for understanding.

[0081] The sequence numbers or order of description of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0082] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units can be a logical functional division, and in actual implementation, there may be other division methods. For instance, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual coupling, direct coupling, or communication connection may be through some interfaces; the indirect coupling or communication connection between units or modules may be electrical or other forms.

[0083] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0084] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0085] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions. When the computer instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium accessible to a computer, or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., digital versatile disc (DVD)), or a semiconductor medium (e.g., solid state disk (SSD)). It is worth noting that the computer-readable storage medium mentioned in the embodiments of this application can be a non-volatile storage medium; in other words, it can be a non-transient storage medium. It should be noted that the information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, stored data, displayed data, etc.), and signals involved in the embodiments of this application are all authorized by the user or fully authorized by all parties, and the collection, use, and processing of related data must comply with the relevant laws, regulations, and standards of the relevant countries and regions. For example, the scene data of the current frame in the 3D virtual scene involved in the embodiments of this application, the client's device information, and the scene interaction information are all obtained with full authorization.

[0086] The above description is only a preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. A method for processing multi-source heterogeneous data, characterized in that, The method includes: Retrieve the data to be stored; Based on the data structure type of the data to be stored, a corresponding preset model is selected to extract key information from the data to be stored, thereby obtaining key data in multiple dimensions. The data structure type includes at least one of text, image, and table. Group all dimensions of the key data into label groups; The tag group and key data are associated and entered into a preset search table, and the data to be stored is saved.

2. The multi-source heterogeneous data processing method according to claim 1, characterized in that, The process involves selecting a corresponding preset model based on the data structure type of the data to be stored to extract key information from the data, resulting in key data across multiple dimensions, including: If the data structure type of the data to be stored supports the direct extraction of key data, then a large language model is used for extraction; otherwise, an image recognition model is used for extraction.

3. The multi-source heterogeneous data processing method according to claim 1, characterized in that, The step of associating the tag group and key data and entering them into a preset search table includes: Determine whether the search table already contains a storage area identical to the tag group; If so, the key data will be stored in the corresponding storage area; If not, a new storage area is generated in the search table according to the tag group, and the key data is stored in the new storage area; The search table includes multiple storage areas, and different storage areas correspond to different tag groups.

4. The multi-source heterogeneous data processing method according to claim 3, characterized in that, Saving the data to be stored includes: If the data to be stored is structured data, it is stored in a relational database; otherwise, it is stored as an object.

5. The multi-source heterogeneous data processing method according to claim 3, characterized in that, The method further includes: In response to a search request, extract the search keywords and dimensions of the search keywords from the search request; The target storage area is determined from the search table based on the dimensions of the search keywords; The key data is determined from the target storage area based on the search keywords; The data associated with the key data will be returned as search results.

6. The multi-source heterogeneous data processing method according to claim 5, characterized in that, The search results also include information on the generation of search keywords.

7. The multi-source heterogeneous data processing method according to claim 5, characterized in that, The method further includes: Based on the retrieval frequency of each data item, data whose retrieval frequency exceeds a preset frequency threshold are stored in the cache space.

8. An electronic device, characterized in that, The electronic device includes a memory and a processor; The memory is used to store computer programs; The processor is configured to execute the computer program to implement the steps of the method according to any one of claims 1-7.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed on a computer or processor, causes the computer or processor to perform the steps of the method according to any one of claims 1-7.

10. A computer program product, characterized in that, The computer program product includes computer instructions that, when executed by a computer or processor, cause the steps of the method as described in any one of claims 1-7 to be performed.

Images