Data recall method and device and related product
By using the structured query language SQL template and vectorization processing combined with the multiple recall mechanism, the retrieval and recall problems of structured and unstructured data in a diversified data environment are solved, and more efficient and accurate data recall is achieved.
Patent Information
- Application Number
- CN202510547157.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-04-28
- Publication Date
- 2025-08-08
AI Technical Summary
In the prior art, it is impossible to efficiently search and recall structured and unstructured data in a diversified data environment, resulting in insufficient accuracy of data recall.
The structured query language SQL template is used to process structured data, and the unstructured data is vectorized. Combined with the multiple recall mechanism, different types of data are retrieved and recalled respectively.
It improves the accuracy of data recall, and searches and recalls different types of data through a multiple recall mechanism, which improves the efficiency and accuracy of information retrieval.
Smart Images
Figure CN120448403A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of data processing technology, and in particular to a data recall method, device and related products. Background Art
[0002] With the continuous development of artificial intelligence (AI) technology, information retrieval and automatic answer generation have been widely used in numerous application scenarios, particularly in intelligent search engines, intelligent question-answering systems, and recommender systems. With the increasing diversity of data sources and types, more and more applications need to process multimodal data, encompassing both structured and unstructured data. Structured data, typically in tabular form, is easy to store, query, and process; whereas unstructured data, including text, images, and video, presents complex information and is more challenging to process. Efficiently retrieving information and generating high-quality answers in this diverse data environment has become a key challenge for AI technology.
[0003] In the existing technology, it mainly relies on the processing of a single data type. When the data to be retrieved includes structured data and unstructured data, it is impossible to accurately retrieve and recall the data.
[0004] In summary, how to improve the accuracy of data recall is an urgent problem that those skilled in the art need to solve. Summary of the Invention
[0005] In view of this, the present application provides a data recall method, device and related products, aiming to improve the accuracy of data recall.
[0006] In a first aspect, the present application provides a data recall method, comprising:
[0007] Acquire data to be retrieved; the data to be retrieved includes structured data and unstructured data;
[0008] Processing the structured data using a structured query language SQL template;
[0009] performing vectorization processing on the unstructured data;
[0010] The multi-way recall mechanism is used to retrieve and recall the processed structured data and the vectorized unstructured data to obtain multiple recall results of the data to be retrieved.
[0011] Optionally, the unstructured data includes text data and image data, and the vectorizing the unstructured data includes:
[0012] Encoding the text data to obtain a semantic vector corresponding to the text data;
[0013] The image data is embedded into a shared vector space.
[0014] Optionally, before processing the structured data using a structured query language SQL template, the method further includes:
[0015] Constructing the SQL template;
[0016] The processing of the structured data using a structured query language SQL template includes:
[0017] Fill the structured data into the SQL template.
[0018] Optionally, after using a multi-way recall mechanism to retrieve and recall the processed structured data and the vectorized unstructured data, and obtaining a plurality of recall results of the data to be retrieved, the method further includes:
[0019] Obtaining confidence levels and context relevance corresponding to the plurality of recall results;
[0020] The plurality of recall results are sorted using the confidence level and the context relevance.
[0021] Optionally, after using a multi-way recall mechanism to retrieve and recall the processed structured data and the vectorized unstructured data, and obtaining a plurality of recall results of the data to be retrieved, the method further includes:
[0022] Recording the user's interactive behavior on the recall result to obtain a behavior dataset;
[0023] The parameters of the retrieval and recall are adjusted according to the behavioral dataset.
[0024] In a second aspect, the present application provides a data recall device, comprising:
[0025] A first acquisition module is used to acquire data to be retrieved; the data to be retrieved includes structured data and unstructured data;
[0026] A first processing module is used to process the structured data using a structured query language SQL template;
[0027] A second processing module, configured to perform vectorization processing on the unstructured data;
[0028] The recall module is used to retrieve and recall the processed structured data and the vectorized unstructured data using a multi-channel recall mechanism to obtain a plurality of recall results of the data to be retrieved.
[0029] Optionally, the unstructured data includes text data and image data, and the second processing module includes:
[0030] An encoding unit, configured to encode the text data to obtain a semantic vector corresponding to the text data;
[0031] An embedding module is used to embed the image data into a shared vector space.
[0032] Optionally, the device further comprises:
[0033] A construction module, used to construct the SQL template;
[0034] The first processing module includes:
[0035] A filling unit is used to fill the structured data into the SQL template.
[0036] Optionally, the device further comprises:
[0037] A second acquisition module is used to obtain the confidence and context relevance corresponding to the multiple recall results respectively;
[0038] A sorting module is used to sort the multiple recall results using the confidence level and the context relevance.
[0039] Optionally, the device further comprises:
[0040] A recording module, configured to record the user's interactive behavior on the recall result to obtain a behavior data set;
[0041] An adjustment module is used to adjust the parameters of the retrieval and recall according to the behavior data set.
[0042] In a third aspect, an embodiment of the present application provides a computer device comprising: a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the computer program, it implements the data recall method described in any one of the implementation methods in the first aspect of the embodiment of the present application.
[0043] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, in which instructions are stored. When the instructions are executed on a terminal device, the terminal device executes a data recall method as described in any one of the implementation methods in the first aspect of the embodiment of the present application.
[0044] The present application provides a data recall method. When executing the method, the data to be retrieved is first obtained, wherein the data to be retrieved includes structured data and unstructured data, and then the structured data is processed using a structured query language SQL template. Then, the unstructured data is vectorized. Finally, a multi-way recall mechanism is used to retrieve and recall the processed structured data and the vectorized unstructured data, and the recall results of multiple data to be retrieved are obtained. In this way, by using a multi-way recall mechanism and using different methods to retrieve and recall different types of data to be retrieved, the accuracy of data recall can be improved. BRIEF DESCRIPTION OF THE DRAWINGS
[0045] In order to more clearly illustrate the technical solutions in this embodiment or the prior art, the following briefly introduces the drawings required for use in the embodiment or the prior art description. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.
[0046] Figure 1 A flow chart of a data recall method provided in an embodiment of the present application;
[0047] Figure 2 A flow chart of another data recall method provided in an embodiment of the present application;
[0048] Figure 3 A schematic diagram of the structure of a data recall device provided in an embodiment of the present application;
[0049] Figure 4 A schematic diagram of the structure of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0050] The following will provide a clear and complete description of the technical solutions in the embodiments of this application, in conjunction with the accompanying drawings. This application provides a data recall method, device, and related equipment for use in the field of data processing technology. The above is merely an example and does not limit the application areas of the methods and devices provided in this application.
[0051] With the continuous development of artificial intelligence (AI) technology, information retrieval and automatic answer generation have been widely used in numerous application scenarios, particularly in intelligent search engines, intelligent question-answering systems, and recommender systems. With the increasing diversity of data sources and types, more and more applications need to process multimodal data, encompassing both structured and unstructured data. Structured data, typically in tabular form, is easy to store, query, and process; whereas unstructured data, including text, images, and video, presents complex information and is more challenging to process. Efficiently retrieving information and generating high-quality answers in this diverse data environment has become a key challenge for AI technology.
[0052] In the existing technology, it mainly relies on the processing of a single data type. When the data to be retrieved includes structured data and unstructured data, it is impossible to accurately retrieve and recall the data.
[0053] After research, the inventors proposed the technical solution of this application. First, the data to be retrieved is obtained, where the data to be retrieved includes structured data and unstructured data. Then, the structured data is processed using a structured query language SQL template. Then, the unstructured data is vectorized. Finally, a multi-way recall mechanism is used to retrieve and recall the processed structured data and the vectorized unstructured data, obtaining recall results for multiple data to be retrieved. In this way, using the multi-way recall mechanism and using different methods to retrieve and recall different types of data to be retrieved can improve the accuracy of data recall.
[0054] In order to enable those skilled in the art to better understand the present application, the present application is further described in detail below in conjunction with the accompanying drawings and specific embodiments. Obviously, the embodiments described are only a part of the embodiments of the present application, rather than all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without making creative work are within the scope of protection of the present application. It should be noted that, for ease of description, only the parts related to the relevant invention are shown in the accompanying drawings. In the absence of conflict, the embodiments in the present application and the features in the embodiments can be combined with each other.
[0055] See also Figure 1 , Figure 1 A method flow chart of a data recall method provided in an embodiment of the present application includes:
[0056] S101: Obtain data to be retrieved.
[0057] First, the data to be retrieved is obtained. This data consists of two types: structured data and unstructured data. Structured data refers to data with a clear organization and format, typically stored in tables and records, and can be efficiently retrieved using standardized query languages such as SQL. Unstructured data, on the other hand, refers to data that has no fixed format and is difficult to manage and query directly using traditional databases, such as text, graphics, video, and audio.
[0058] S102: Processing the structured data using a structured query language SQL template.
[0059] Next, you need to pre-build multiple Structured Query Language (SQL) templates to adapt to different scenarios. For example: SELECT name, price FROM products WHERE category = 'electronics' AND price < 1000. The SQL template predefines the query logic and supports dynamic parameter filling. Set up B+ tree indexes for frequently queried fields, such as primary keys or filter conditions, to improve retrieval efficiency. Finally, fill the SQL template with the structured data from the data to be retrieved. This allows the database to execute the query and return the results.
[0060] S103: Perform vectorization processing on unstructured data.
[0061] Vectorized representation of unstructured data, that is, vectorized encoding of unstructured data. Vectorized encoding refers to the process of converting unstructured data into numerical vectors. Through this conversion, the semantics and characteristics of the data can be captured, thereby facilitating similarity calculation and retrieval, which is mainly accomplished through deep learning models.
[0062] First, for text data, models such as BERT and RoBERTa are used to encode sentences and generate semantic vectors. This model is based on the Transformer architecture and uses a self-attention mechanism to capture contextual semantics. Specifically, a pre-trained language model, such as BERT, is used to extract the [CLS] vector of the text, which represents its global semantics. When using this vector for retrieval, vector databases such as Milvus and FAISS are used to store the semantic vectors, and the cosine similarity between the input query vector and the stored vector is calculated, returning the result with the highest similarity. Similarity matching refers to evaluating the similarity between data by calculating the similarity between them in terms of content. It is used for retrieval of unstructured data, such as calculating the similarity of text or images.
[0063] For image data, deep learning models such as ResNet and CLIP are used to extract feature vectors. During retrieval, images and text are matched using a shared embedding space. Specifically, cross-modal models such as CLIP are used to embed images into a shared vector space and perform comparison retrieval with text vectors.
[0064] S104: Using a multi-way recall mechanism, the processed structured data and the vectorized unstructured data are retrieved and recalled to obtain recall results of multiple data to be retrieved.
[0065] A multi-channel recall mechanism refers to a system that uses multiple parallel recall channels to process different types of data, such as structured data and unstructured data. This parallel processing can improve retrieval efficiency and information comprehensiveness. The core of the multi-channel recall mechanism is to design independent recall channels for different types of data. Each recall channel includes specific data preprocessing methods and retrieval algorithms. For example, the structured data channel is based on SQL index retrieval; the text data channel is based on vector retrieval; and the image data channel is based on visual feature matching and cross-modal retrieval. In this way, the multi-channel recall mechanism combines task decomposition theory, improves retrieval efficiency through parallel processing, and reduces interference between different data types. As a result, different retrieval results corresponding to different types of data can be obtained.
[0066] In an embodiment of the present application, the data to be retrieved is first obtained, where the data to be retrieved includes structured data and unstructured data. The structured data is then processed using a structured query language SQL template. The unstructured data is then vectorized. Finally, a multi-way recall mechanism is used to retrieve and recall the processed structured data and the vectorized unstructured data, obtaining recall results for multiple data to be retrieved. In this way, by using a multi-way recall mechanism and using different methods to retrieve and recall different types of data to be retrieved, the accuracy of data recall can be improved.
[0067] In addition, if Figure 2 As shown, Figure 2 A flowchart of another data recall method provided in an embodiment of the present application includes:
[0068] The implementation of steps S101-S104 is the same as that of steps S201-S204, and will not be repeated here.
[0069] S205: Obtain confidence levels and context relevance corresponding to the multiple recall results.
[0070] The weight of each channel is dynamically calculated based on the confidence and contextual relevance of the recall result. Confidence is the relevance score of each recall result. Contextual relevance is the contextual matching degree calculated by a deep learning model based on user input and historical behavior.
[0071] S206: Sort the multiple recall results using the confidence level and the context relevance.
[0072] Use weighted averaging or attention mechanism to rank the results and generate the final candidate set.
[0073] In addition, embodiments of the present application can also optimize feedback. This requires first collecting feedback data. Specifically, users' interactions with generated results, such as clicks, likes, and redirects, can be recorded to form a behavioral dataset. Then, based on user feedback, a reinforcement learning model, such as the REINFORCE algorithm, is used to update recall and fusion parameters. For example, the weight of recall channels with high user satisfaction can be increased, reducing the probability of generating low-relevance results.
[0074] The above are some specific implementations of the data recall method provided in the embodiment of the present application. Based on this, the present application also provides a corresponding device. The device provided in the embodiment of the present application will be introduced from the perspective of functional modularization.
[0075] See also Figure 3 , Figure 3 This is a schematic diagram of the structure of a data recall device provided in an embodiment of the present application. The data recall device 300 includes:
[0076] A first acquisition module 310 is configured to acquire data to be retrieved; the data to be retrieved includes structured data and unstructured data;
[0077] A first processing module 320 is configured to process the structured data using a structured query language SQL template;
[0078] A second processing module 330 is configured to perform vectorization processing on the unstructured data;
[0079] The recall module 340 is used to retrieve and recall the processed structured data and the vectorized unstructured data using a multi-way recall mechanism to obtain a plurality of recall results of the data to be retrieved.
[0080] Optionally, the unstructured data includes text data and image data, and the second processing module 330 includes:
[0081] An encoding unit, configured to encode the text data to obtain a semantic vector corresponding to the text data;
[0082] An embedding module is used to embed the image data into a shared vector space.
[0083] Optionally, the apparatus 300 further includes:
[0084] A construction module, used to construct the SQL template;
[0085] The first processing module 320 includes:
[0086] A filling unit is used to fill the structured data into the SQL template.
[0087] Optionally, the apparatus 300 further includes:
[0088] A second acquisition module is used to obtain the confidence and context relevance corresponding to the multiple recall results respectively;
[0089] A sorting module is used to sort the multiple recall results using the confidence level and the context relevance.
[0090] Optionally, the apparatus 300 further includes:
[0091] A recording module, configured to record the user's interactive behavior on the recall result to obtain a behavior data set;
[0092] An adjustment module is used to adjust the parameters of the retrieval and recall according to the behavior data set.
[0093] The embodiments of the present application also provide corresponding devices and computer storage media for implementing the solutions provided by the embodiments of the present application.
[0094] like Figure 4 As shown, computer device 01 is a general-purpose computing device. Components of computer device 01 may include, but are not limited to, one or more processors or processor units 03, system memory 08, and bus 04 connecting various system components (including system memory 08 and processor unit 03).
[0095] Bus 04 represents one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, a processor, or a local bus using any of a variety of bus architectures. Examples of these architectures include, but are not limited to, the Industry Standard Architecture (ISA) bus, the Micro Channel Architecture (MAC) bus, the Enhanced ISA bus, the Video Electronics Standards Association (VESA) local bus, and the Peripheral Component Interconnect (PCI) bus.
[0096] The computer device 01 typically includes a variety of computer system readable media, which can be any available media that can be accessed by the computer device 01, including volatile and non-volatile media, removable and non-removable media.
[0097] System memory 08 may include computer system readable media in the form of volatile memory, such as random access memory (RAM) 09 and / or cache memory 10. Computer device 01 may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, storage system 11 may be used to read and write non-removable, non-volatile magnetic media ( Figure 4 Not shown, often called a "hard drive"). Although Figure 4 Not shown, a disk drive for reading and writing to a removable non-volatile disk (e.g., a "floppy disk"), and an optical disk drive for reading and writing to a removable non-volatile optical disk (e.g., a CD-ROM, DVD-ROM, or other optical media) may be provided. In these cases, each drive may be connected to the bus 04 via one or more data medium interfaces. The system memory 08 may include at least one program product having a set (e.g., at least one) of program modules configured to perform the functions of various embodiments of the present invention.
[0098] A program / utility 12 having a set (at least one) of program modules 13 may be stored, for example, in system memory 08. Such program modules 13 include, but are not limited to, an operating system, one or more application programs, other program modules, and program data, each of which, or some combination thereof, may include an implementation of a network environment. Program modules 13 generally perform the functions and / or methods of the embodiments described herein.
[0099] The computer device 01 may also communicate with one or more external devices 02 (e.g., a keyboard, a pointing device, a display 07, etc.), one or more devices that enable a user to interact with the computer device 01, and / or any device that enables the computer device 01 to communicate with one or more other computing devices (e.g., a network card, a modem, etc.). Such communication may be performed via an input / output (I / O) interface 06. Furthermore, the computer device 01 may also communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) via a network adapter 05. Figure 4 As shown, the network adapter 05 communicates with other modules of the computer device 01 via the bus 04. Figure 4Not shown, other hardware and / or software modules may be used in conjunction with the computer device 01, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.
[0100] The processor unit 03 executes various functional applications and data processing by running programs stored in the system memory 08, such as implementing a data recall method provided in an embodiment of the present application.
[0101] It should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply the existence of any such actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or device comprising the element.
[0102] Through the description of the above embodiments, it can be known that those skilled in the art can clearly understand that all or part of the steps in the above embodiment methods can be implemented by means of software plus a general hardware platform. Based on this understanding, the technical solution of the present application can be embodied in the form of a software product, which can be stored in a storage medium, such as a read-only memory (ROM) / RAM, a magnetic disk, an optical disk, etc., and includes a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network communication device such as a router) to execute the methods described in each embodiment or certain parts of the embodiments of the present application.
[0103] Each embodiment in this specification is described in a progressive manner. The same or similar parts between the embodiments can be referred to each other. Each embodiment focuses on the differences from other embodiments. In particular, for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple. For the relevant parts, refer to the partial description of the method embodiment. Some or all of the modules can be selected according to actual needs to achieve the purpose of the embodiment. Those of ordinary skill in the art can understand and implement it without paying any creative work.
[0104] The above description is merely an exemplary embodiment of the present application and is not intended to limit the scope of protection of the present application.
Claims
1. A data recall method, characterized in that: include: Get the data to be retrieved; The data to be retrieved includes structured data and unstructured data; Processing the structured data using a structured query language SQL template; performing vectorization processing on the unstructured data; The multi-way recall mechanism is used to retrieve and recall the processed structured data and the vectorized unstructured data to obtain multiple recall results of the data to be retrieved.
2. The method according to claim 1, characterized in that The unstructured data includes text data and image data, and the vectorizing the unstructured data includes: Encoding the text data to obtain a semantic vector corresponding to the text data; The image data is embedded into a shared vector space.
3. The method according to claim 1, characterized in that Before processing the structured data using the structured query language SQL template, the method further includes: Constructing the SQL template; The processing of the structured data using a structured query language SQL template includes: Fill the structured data into the SQL template.
4. The method according to claim 1, wherein After the multi-channel recall mechanism is used to retrieve and recall the processed structured data and the vectorized unstructured data, and a plurality of recall results of the data to be retrieved are obtained, the method further includes: Obtaining confidence levels and context relevance corresponding to the plurality of recall results; The plurality of recall results are sorted using the confidence level and the context relevance.
5. The method according to claim 1, wherein After the multi-channel recall mechanism is used to retrieve and recall the processed structured data and the vectorized unstructured data, and a plurality of recall results of the data to be retrieved are obtained, the method further includes: Recording the user's interactive behavior on the recall result to obtain a behavior dataset; The parameters of the retrieval and recall are adjusted according to the behavioral dataset.
6. A data recall device, characterized in that: include: A first acquisition module is used to acquire data to be retrieved; The data to be retrieved includes structured data and unstructured data; A first processing module is used to process the structured data using a structured query language SQL template; A second processing module, configured to perform vectorization processing on the unstructured data; The recall module is used to retrieve and recall the processed structured data and the vectorized unstructured data using a multi-channel recall mechanism to obtain a plurality of recall results of the data to be retrieved.
7. The device according to claim 6, characterized in that The unstructured data includes text data and image data, and the second processing module includes: An encoding unit, configured to encode the text data to obtain a semantic vector corresponding to the text data; An embedding module is used to embed the image data into a shared vector space.
8. The device according to claim 6, characterized in that The device further comprises: A construction module, used to construct the SQL template; The first processing module includes: A filling unit is used to fill the structured data into the SQL template.
9. A computer device, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the data recall method according to any one of claims 1 to 5 is implemented.
10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores instructions, and when the instructions are executed on the terminal device, the terminal device executes the data recall method according to any one of claims 1 to 5.