Unstructured data retrieval method, device and equipment and readable medium

Abstract

The invention provides an unstructured data retrieval method, device and equipment and a readable medium, and is applied to an unstructured data retrieval system, the unstructured data retrieval system at least comprises a gateway layer, a business service layer and a data processing layer, and the data processing layer is utilized to obtain unstructured data; converting the unstructured data into vectors by using a data processing layer, and storing the vectors into a vector storage library; acquiring a retrieval request of the unstructured data by utilizing the gateway layer, and sending the retrieval request to the business service layer; sending the retrieval request to a data processing layer by using a business service layer; based on the vector storage library, retrieving the to-be-retrieved data in the retrieval request by using the data processing layer to obtain a retrieval result; the retrieval results are screened through the business service layer by using the similarity script to obtain a target retrieval result, and the target retrieval result is sent to a gateway layer; and the target retrieval result is output by utilizing the gateway layer, so that the problems of relatively high cost and relatively poor effect of manual weight adjustment are avoided.

Description

Technical Field

[0001] The present invention relates to the technical field of information retrieval, and in particular to a method for retrieving unstructured data, a device for retrieving unstructured data, an electronic device and a computer-readable medium. Background Art

[0002] In the current era of big data, there is a large amount of unstructured data from multiple channels and sources. These unstructured data cover various forms such as text, images, audio and video. Traditional data retrieval methods face many challenges in dealing with the rapidly growing and diverse unstructured data.

[0003] When searching for unstructured data, if the method of manual weighting is used, that is, manually adjusting the weights of different data items, it usually takes a lot of time and human resources, and the cost is high. In addition, due to factors such as human subjectivity, fatigue, and knowledge limitations, manual judgment may not be accurate or consistent, which may lead to inaccurate search results, unable to meet user needs or accurately reflect the true relevance of the data, resulting in poor results of manual weighting. Summary of the invention

[0004] The embodiments of the present invention provide a method, device, equipment and computer-readable storage medium for retrieving unstructured data to solve the problem that manual weighting requires a lot of time and human resources, is costly and has poor results when retrieving unstructured data.

[0005] The embodiment of the present invention discloses a method for retrieving unstructured data, which is applied to an unstructured data retrieval system. The unstructured data retrieval system at least includes a gateway layer, a business service layer, and a data processing layer. The method includes:

[0006] Acquiring preset unstructured data using the data processing layer;

[0007] Using the data processing layer to convert the unstructured data into vectors, and storing the vectors in a preset vector storage library;

[0008] Utilizing the gateway layer to obtain a user's retrieval request for the unstructured data, and sending the retrieval request to the business service layer;

[0009] Using the business service layer to send the search request to the data processing layer;

[0010] Based on the vector repository, using the data processing layer to retrieve the data to be retrieved in the retrieval request to obtain a retrieval result;

[0011] The business service layer uses a preset similarity script to screen the search results to obtain target search results, and sends the target search results to the gateway layer;

[0012] The target search result is outputted using the gateway layer.

[0013] Optionally, the unstructured data includes at least one or more of text data, video data, image data and audio data; the step of converting the unstructured data into vectors and storing the vectors in a preset vector storage library includes:

[0014] Convert the text data into text vectors using a preset word embedding model;

[0015] Using a preset convolutional neural network model and / or a recurrent neural network model, the video data, the image data, and the audio data are converted into a video vector, an image vector, and an audio vector, respectively;

[0016] The text vector, the video vector, the image vector and the audio vector are stored in the vector storage library.

[0017] Optionally, before the step of converting the unstructured data into a vector, the step includes:

[0018] removing noise from the unstructured data; and / or,

[0019] classifying the unstructured data; and / or,

[0020] extracting features of the unstructured data; and / or,

[0021] The unstructured data is standardized.

[0022] Optionally, the step of using the data processing layer to retrieve the data to be retrieved in the search request to obtain the search result includes:

[0023] Extracting the data to be retrieved in the search request; the data to be retrieved is unstructured data to be retrieved;

[0024] Preprocessing the unstructured data to be retrieved;

[0025] Converting the preprocessed unstructured data to be retrieved into vectors to be retrieved;

[0026] The data processing layer is used to search the vector to be searched to obtain the search result.

[0027] Optionally, the step of using the data processing layer to search the vector to be searched to obtain the search result includes:

[0028] Get the search radius of preset length;

[0029] The vectors within the search radius around the vector to be searched in the vector storage are taken as the search results.

[0030] Optionally, the similarity script involves at least cosine similarity and / or Euclidean distance.

[0031] Optionally, the step of acquiring preset unstructured data by using the data processing layer includes:

[0032] The data processing layer is used to obtain the unstructured data from a preset file server.

[0033] The embodiment of the present invention further discloses a retrieval device for unstructured data, which is applied to an unstructured data retrieval system. The unstructured data retrieval system at least includes a gateway layer, a business service layer, and a data processing layer. The device includes:

[0034] A data acquisition module, used to acquire preset unstructured data using the data processing layer;

[0035] A vector conversion module, used to convert the unstructured data into vectors using the data processing layer, and store the vectors into a preset vector storage library;

[0036] A search request acquisition module, used to use the gateway layer to acquire the user's search request for the unstructured data, and send the search request to the business service layer;

[0037] A search request sending module, used to send the search request to the data processing layer using the business service layer;

[0038] A retrieval module, configured to retrieve the data to be retrieved in the retrieval request based on the vector repository and using the data processing layer to obtain a retrieval result;

[0039] A screening module, used to screen the search results by using a preset similarity script through the business service layer to obtain a target search result, and send the target search result to the gateway layer;

[0040] An output module is used to output the target search result using the gateway layer.

[0041] Optionally, the unstructured data includes at least one or more of text data, video data, image data and audio data; and the vector conversion module includes:

[0042] A first vector conversion submodule, used to convert the text data into a text vector using a preset word embedding model;

[0043] A second vector conversion submodule, used to convert the video data, the image data and the audio data into a video vector, an image vector and an audio vector respectively by using a preset convolutional neural network model and / or a recurrent neural network model;

[0044] The vector storage submodule is used to store the text vector, the video vector, the image vector and the audio vector into the vector storage repository.

[0045] Optionally, the device comprises:

[0046] an unstructured data processing module, used to remove noise from the unstructured data; and / or,

[0047] classifying the unstructured data; and / or,

[0048] extracting features of the unstructured data; and / or,

[0049] The unstructured data is standardized.

[0050] Optionally, the retrieval module includes:

[0051] An extraction submodule, used for extracting the data to be retrieved in the search request; the data to be retrieved is unstructured data to be retrieved;

[0052] A preprocessing submodule, used for preprocessing the unstructured data to be retrieved;

[0053] A third vector conversion submodule, used for converting the preprocessed unstructured data to be retrieved into a vector to be retrieved;

[0054] The retrieval submodule is used to use the data processing layer to search the vector to be searched to obtain the search result.

[0055] Optionally, the retrieval submodule includes:

[0056] A radius acquisition unit, used to acquire a search radius of a preset length;

[0057] The retrieval result is used as a unit, and is used to take the vectors within the retrieval radius around the vector to be retrieved in the vector storage as the retrieval result.

[0058] Optionally, the similarity script involves at least cosine similarity and / or Euclidean distance.

[0059] Optionally, the data acquisition module includes:

[0060] The data acquisition submodule is used to acquire the unstructured data from a preset file server using the data processing layer.

[0061] The embodiment of the present invention further discloses an electronic device, comprising a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus;

[0062] The memory is used to store computer programs;

[0063] The processor is used to implement the method described in the embodiment of the present invention when executing the program stored in the memory.

[0064] The embodiment of the present invention further discloses one or more computer-readable media on which instructions are stored. When executed by one or more processors, the processors are enabled to execute the method described in the embodiment of the present invention.

[0065] The embodiments of the present invention include the following advantages:

[0066] In an embodiment of the present invention, an unstructured data retrieval system includes at least a gateway layer, a business service layer, and a data processing layer. The unstructured data retrieval system uses the data processing layer to obtain preset unstructured data, uses the data processing layer to convert the unstructured data into vectors, and stores the vectors in a preset vector repository. The unstructured data retrieval system uses the gateway layer to obtain the user's unstructured data retrieval request, and sends the retrieval request to the business service layer, uses the business service layer to send the retrieval request to the data processing layer, based on the vector repository, uses the data processing layer to retrieve the data to be retrieved in the retrieval request, obtains the retrieval result, and then uses the business service layer to filter the retrieval result using a preset similarity script to obtain the target retrieval result, and sends the target retrieval result to the gateway layer, and uses the gateway layer to output the target retrieval result. In an embodiment of the present invention, when converting unstructured data into vectors, the unstructured data retrieval system determines the corresponding vector dimension according to different categories of unstructured data, selects appropriate features as components of the vector, and improves the effect and performance of unstructured data retrieval. In an embodiment of the present invention, the application of the unstructured data retrieval system optimizes the query processing process of unstructured data, realizes the optimization of vector generation and similarity calculation, avoids the problem of high cost and poor effect of manual weight adjustment, and improves the retrieval speed and accuracy of unstructured data. The unstructured data retrieval system of the present invention can realize the processing and retrieval of large-scale unstructured data, and optimizes the retrieval of unstructured data. The unstructured data retrieval system of the present invention is also scalable and customizable, and can adapt to future data growth and changes in application scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0067] Figure 1 is a flowchart of a method for retrieving unstructured data provided in an embodiment of the present invention;

[0068] Figure 2 is a schematic diagram of an unstructured data retrieval system provided in an embodiment of the present invention;

[0069] Figure 3 is a flowchart of a method for retrieving unstructured data provided in an embodiment of the present invention;

[0070] Figure 4 is a structural block diagram of a retrieval device for unstructured data provided in an embodiment of the present invention;

[0071] Figure 5 is a block diagram of an electronic device provided in an embodiment of the present invention;

[0072] Figure 6 is a schematic diagram of a computer-readable medium provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0073] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0074] Reference Figure 1 , shows a flowchart of a method for retrieving unstructured data provided in an embodiment of the present invention, which is applied to an unstructured data retrieval system. The unstructured data retrieval system includes at least a gateway layer, a business service layer, and a data processing layer, and may specifically include the following steps:

[0075] Step 101, using the data processing layer to obtain preset unstructured data;

[0076] In an embodiment of the present invention, the unstructured data retrieval system can be used to retrieve unstructured data. Unstructured data includes at least one or more of text data, image data, audio data, and video data. Among them, text data can also include user operation behavior data recorded in text on the Internet and mobile applications.

[0077] In an embodiment of the present invention, the unstructured data retrieval system includes at least a gateway layer, a business service layer, and a data processing layer. The gateway layer and the business service layer are connected in communication, and the business service layer and the data processing layer are connected in communication. The unstructured data retrieval system can use the data processing layer to input unstructured data from multiple channels as a source of unstructured data retrieval. Specifically, the unstructured data retrieval system can use the data processing layer to obtain manually input data, and can also obtain unstructured data stored in a file server from a file server.

[0078] Step 102, using the data processing layer to convert the unstructured data into vectors, and storing the vectors in a preset vector storage library;

[0079] In an embodiment of the present invention, the data processing layer can not only extract unstructured data, but also preprocess the unstructured data and convert the preprocessed unstructured data into vectors. Therefore, the unstructured data retrieval system can use the data processing layer to preprocess the unstructured data and convert the preprocessed unstructured data into vectors. For example, the data processing layer can preprocess text data and convert the preprocessed text data into text vectors. The unstructured data retrieval system can also use the data processing layer to store vectors in a preset vector repository. The vector repository can be Milvus.

[0080] It should be noted that Milvus is an open source vector database that supports storage and query operations for TB-level vectors, and has the characteristics of high flexibility, stability, reliability, and high-speed query. Among them, TB-level refers to a very large amount, and storage operations include adding, deleting, and modifying vector data. Query operations can achieve near real-time query, that is, very fast and close to real-time return results.

[0081] Milvus integrates widely used vector index libraries such as Faiss, NMSLIB, and Annoy. Milvus also supports data partitioning and sharding, data persistence, incremental data ingestion, scalar-vector hybrid query, time travel, and other functions. At the same time, it greatly optimizes the performance of vector retrieval and can meet the application requirements of any vector retrieval scenario. Among them, Faiss, NMSLIB, and Annoy are all vector index libraries; data partitioning and sharding refers to allowing data to be stored in a distributed manner and supporting large-scale data processing; data persistence refers to the ability to store data in the vector index library for a long time; incremental data ingestion refers to the ability to gradually add new data to the vector index library; scalar-vector hybrid query refers to the ability to query vector and scalar data at the same time; timetravel refers to the ability of Milvus to query the historical state of data.

[0082] Milvus also provides a complete set of simple and intuitive APIs (application programming interfaces) to provide different index types for different needs. In addition, Milvus can also filter scalar data, further improving the recall rate (the accuracy of the query) and enhancing the flexibility of the search.

[0083] Step 103, using the gateway layer to obtain the user's retrieval request for the unstructured data, and sending the retrieval request to the business service layer;

[0084] In the embodiment of the present invention, the unstructured data retrieval system can use the gateway layer to obtain the unstructured data retrieval request input by the user, and send the retrieval request to the business service layer. It should be noted that the unstructured data retrieval request includes the unstructured data to be retrieved.

[0085] Step 104, using the business service layer to send the search request to the data processing layer;

[0086] In the embodiment of the present invention, the unstructured data retrieval system can use the business service layer to send the retrieval request to the data processing layer.

[0087] Step 105, based on the vector repository, using the data processing layer to search for the data to be searched in the search request to obtain a search result;

[0088] In an embodiment of the present invention, the unstructured data retrieval system can use the data processing layer to preprocess the unstructured data to be retrieved in the retrieval request, and vectorize the preprocessed unstructured data to obtain the vector to be retrieved. Then, the unstructured data retrieval system can call the vector repository to retrieve the vector to be retrieved and obtain the retrieval result.

[0089] Step 106, screening the search results by using a preset similarity script through the business service layer to obtain a target search result, and sending the target search result to the gateway layer;

[0090] In the embodiment of the present invention, the unstructured data retrieval system can use the data processing layer to send the retrieval results to the business service layer. The business service layer can use the similarity script to filter the retrieval results, obtain the target retrieval results, and send the target retrieval results to the gateway layer.

[0091] Step 107: Utilize the gateway layer to output the target search result.

[0092] In an embodiment of the present invention, the unstructured data retrieval system can use the gateway layer to output the target retrieval results to the user.

[0093] In an embodiment of the present invention, an unstructured data retrieval system includes at least a gateway layer, a business service layer, and a data processing layer. The unstructured data retrieval system uses the data processing layer to obtain preset unstructured data, uses the data processing layer to convert the unstructured data into vectors, and stores the vectors in a preset vector repository. The unstructured data retrieval system uses the gateway layer to obtain the user's unstructured data retrieval request, and sends the retrieval request to the business service layer, uses the business service layer to send the retrieval request to the data processing layer, based on the vector repository, uses the data processing layer to retrieve the data to be retrieved in the retrieval request, obtains the retrieval result, and then uses the business service layer to filter the retrieval result using a preset similarity script to obtain the target retrieval result, and sends the target retrieval result to the gateway layer, and uses the gateway layer to output the target retrieval result. In an embodiment of the present invention, when converting unstructured data into vectors, the unstructured data retrieval system determines the corresponding vector dimension according to different categories of unstructured data, selects appropriate features as components of the vector, and improves the effect and performance of unstructured data retrieval. In an embodiment of the present invention, the application of the unstructured data retrieval system optimizes the query processing process of unstructured data, realizes the optimization of vector generation and similarity calculation, avoids the problem of high cost and poor effect of manual weight adjustment, and improves the retrieval speed and accuracy of unstructured data. The unstructured data retrieval system of the present invention can realize the processing and retrieval of large-scale unstructured data, and optimizes the retrieval of unstructured data. The unstructured data retrieval system of the present invention is also scalable and customizable, and can adapt to future data growth and changes in application scenarios.

[0094] Further, in any of the above embodiments, the unstructured data includes at least one or more of text data, video data, image data, and audio data; step 102 includes:

[0095] Sub-step S11, converting the text data into a text vector using a preset word embedding model;

[0096] Sub-step S12, using a preset convolutional neural network model and / or a recurrent neural network model, converting the video data, the image data, and the audio data into a video vector, an image vector, and an audio vector, respectively;

[0097] Sub-step S13, storing the text vector, the video vector, the image vector and the audio vector into the vector storage library.

[0098] In an embodiment of the present invention, the unstructured data includes at least one or more of text data, video data, image data, and audio data. The unstructured data retrieval system can convert text data into text vectors using a preset word embedding model. In an embodiment of the present invention, the word embedding model can output a corresponding vector for an input word. For an input sentence, the word embedding model can calculate the average of all word vectors in the sentence or add all word vectors in the sentence to obtain a sentence vector corresponding to the sentence. The word embedding model can also use TF-IDF (TermFrequency-Inverse Document Frequency) weights to weight all word vectors in the sentence and then average them to obtain a sentence vector corresponding to the sentence.

[0099] In an embodiment of the present invention, the unstructured data retrieval system can also use a preset convolutional neural network model and / or a recurrent neural network model to convert video data, image data and audio data into video vectors, image vectors and audio vectors respectively.

[0100] In a specific example, the steps of converting image data into image vectors using a convolutional neural network model include: first, inputting the image data into the convolutional neural network model. Before the last convolutional layer or fully connected layer of the model, a series of feature maps are output, which contain the features extracted by the convolutional neural network model. Then, these feature maps can be converted into vectors of fixed length by global average pooling or global maximum pooling. In addition, the feature map can also be expanded and then input into the fully connected layer, which will output a fixed-length vector, which is the image vector.

[0101] In another specific example, the step of converting audio data into an audio vector using a recurrent neural network model includes: first, inputting the audio data into the recurrent neural network model. During the model processing, the output of the last time step of the recurrent neural network model usually contains summary information of the entire sequence of audio data. Next, the output of the last time step of the recurrent neural network model is regarded as the features extracted by the model. Then, these features are converted into a vector of fixed length, and this fixed length vector is the audio vector.

[0102] In another specific example, the steps of converting video data into video vectors using a convolutional neural network model are as follows: First, the video data is decomposed into separate video frames. Then, the convolutional neural network model is applied to each frame to extract key features. In order to more accurately capture the continuous changes in time of video frames, a 3D CNN (three-dimensional convolutional neural network model) or a RNN / LSTM network (recursive neural network / long short-term memory network) can be used to process the sequence of these video frames to obtain the temporal features of the video data. Next, the features of all frames are integrated through techniques such as average pooling, maximum pooling or attention mechanisms. In order to incorporate information in the time dimension, the features of all frames and the temporal features of the video data can be encoded through the time encoding layer of the convolutional neural network model or by using the final hidden state of the RNN / LSTM to obtain the target features. Finally, the target features are normalized to obtain the video vector.

[0103] In an embodiment of the present invention, the unstructured data retrieval system can utilize the data processing layer to store text vectors, video vectors, image vectors, and audio vectors into a vector storage repository.

[0104] In an embodiment of the present invention, unstructured data is converted into vectors through a word embedding model, a convolutional neural network model, and a recurrent neural network model to facilitate semantic analysis and information retrieval of unstructured data, thereby realizing vector-based retrieval of unstructured data, improving the retrieval efficiency of unstructured data, and providing support for more accurate decision-making and business application development. In an embodiment of the present invention, vector-based retrieval of unstructured data can be applied to different types of unstructured data, including text data, image data, audio data, etc., and has strong flexibility and versatility, solving the problem that when retrieving unstructured data, the unstructured data is data without a fixed format or organizational structure, which makes it impossible to assemble data conditions.

[0105] Further, in any of the above embodiments, before step 102, the steps include:

[0106] Sub-step S21, removing noise from the unstructured data; and / or,

[0107] classifying the unstructured data; and / or,

[0108] extracting features of the unstructured data; and / or,

[0109] The unstructured data is standardized.

[0110] In the embodiment of the present invention, before converting the unstructured data into vectors, the unstructured data needs to be preprocessed so as to perform vectorization processing on the unstructured data.

[0111] If the unstructured data is text data, the preprocessing of the text data includes: 1) Cleaning: removing the noise in the text data, including irrelevant characters and stop words. Irrelevant characters include punctuation marks, numbers, etc., and stop words include common but low-information words such as "de" (的), "he" (和), etc. 2) Word segmentation: splitting the text into words. 3) Standardization: converting the text into a unified format and performing stemming or lemmatization to restore the words to their basic forms. 4) Labeling: classifying the text data and adding corresponding labels to different categories. 5) Feature extraction: extracting the features in the text data.

[0112] If the unstructured data is image data, the preprocessing of the image data includes: 1) Cleaning: removing the noise in the image data. 2) Normalizing pixel values: scaling the image pixel values to the range of 0 - 1. 3) Feature extraction: extracting image features using algorithms such as color histograms, edge detection, and feature maps in deep learning. 4) Labeling: classifying the image data and adding corresponding labels to different categories.

[0113] If the unstructured data is audio data, the preprocessing of the audio data includes: 1) Cleaning: removing the noise in the audio data, including silent parts, etc., and performing noise reduction on the audio. 2) Framing: splitting the audio into short-time frames. 3) Feature extraction: extracting spectral features, acoustic features, etc. in the audio data. 4) Labeling: classifying the audio data and adding corresponding labels to different categories.

[0114] If the unstructured data is video data, the preprocessing of the video data includes: 1) Cleaning: removing the noise in the video data. 2) Framing: decomposing the video into video frames. 3) Feature extraction: extracting image features from each video frame and time series features from the video stream. 4) Labeling: classifying the video data and adding corresponding labels to different categories.

[0115] Furthermore, in any of the above embodiments, step 105 includes:

[0116] Sub-step S31, extracting the data to be retrieved in the retrieval request; the data to be retrieved is unstructured data to be retrieved;

[0117] Sub-step S32, preprocessing the unstructured data to be retrieved;

[0118] Sub-step S33, converting the preprocessed unstructured data to be retrieved into a retrieval vector;

[0119] Sub-step S34, using the data processing layer to retrieve the retrieval vector to obtain the retrieval result.

[0120] In an embodiment of the present invention, in the process of the unstructured data retrieval system using the data processing layer to retrieve the data to be retrieved in the retrieval request, the unstructured data retrieval system can use the data processing layer to extract the data to be retrieved in the retrieval request. It should be noted that the data to be retrieved is the unstructured data to be retrieved. The unstructured data retrieval system can use the data processing layer to preprocess the unstructured data to be retrieved, and convert the preprocessed unstructured data to be retrieved into a vector to be retrieved. Finally, the data processing layer is used to retrieve the vector to be retrieved to obtain a retrieval result.

[0121] Further, in any of the above embodiments, sub-step S34 includes:

[0122] Sub-step S41, obtaining a search radius of a preset length;

[0123] Sub-step S42: taking the vectors within the search radius around the to-be-searched vector in the vector storage as the search results.

[0124] In an embodiment of the present invention, in the process of the unstructured data retrieval system using the data processing layer to search the vector to be searched to obtain the search results, the unstructured data retrieval system can obtain a search radius of a preset length and take the vectors within the search radius around the vector to be searched in the vector storage library as the search results.

[0125] Further, in any of the above embodiments, the similarity script at least involves cosine similarity and / or Euclidean distance.

[0126] In an embodiment of the present invention, the business service layer of the unstructured data retrieval system can use cosine similarity and / or Euclidean distance to filter the retrieval results of the data processing layer to obtain target retrieval results. Specifically, the business service layer can use cosine similarity to calculate the similarity between the vector in the retrieval result and the vector to be retrieved, and use the vector with a similarity higher than a preset similarity threshold as the target retrieval result. The business service layer can also calculate the Euclidean distance between the vector in the retrieval result and the vector to be retrieved, and use the vector with a Euclidean distance lower than a preset Euclidean distance threshold as the target retrieval result.

[0127] In the embodiment of the present invention, the similarity script at least involves cosine similarity and / or Euclidean distance, and the business service layer of the unstructured data retrieval system selects a suitable similarity calculation method to measure the similarity between a vector and a vector to be retrieved according to the retrieval needs of the unstructured data. In the embodiment of the present invention, the retrieval of unstructured data is performed through the index structure and the similarity script, which greatly improves the retrieval efficiency of unstructured data with multiple data formats and large data volumes.

[0128] Further, in any of the above embodiments, step 101 includes:

[0129] Sub-step S51: using the data processing layer to obtain the unstructured data from a preset file server.

[0130] In the embodiment of the present invention, the data processing layer of the unstructured data retrieval system can input unstructured data from the file server.

[0131] Reference Figure 2 , showing a schematic diagram of an unstructured data retrieval system provided in an embodiment of the present invention. The unstructured data retrieval system includes a gateway layer, a business service layer, a data processing layer, and a deployment environment layer. The gateway layer is used to receive a user's unstructured data retrieval request and forward the retrieval request to the business service layer. The gateway layer is also used to output the target retrieval result of the retrieval request sent by the business service layer to the user. Therefore, the gateway layer is used to interact with the user. The gateway layer includes nginx, REST API, and Tomcat. Among them, nginx is used to receive and forward retrieval requests for unstructured data, and REST API defines the interaction protocol between the user's client and the gateway layer. REST API usually runs on Tomcat, processes requests from nginx, and returns responses. Tomcat is used to implement REST API, process business logic, and provide services to the user's client through nginx.

[0132] The business service layer provides retrieval services, unstructured data retrieval APIs, and similarity calculation scripts. The business service layer is used to receive retrieval requests sent by the gateway layer through the unstructured data retrieval API and send the retrieval requests to the data processing layer. The business service layer is also used to use the similarity calculation script to filter the retrieval results of the data processing layer, obtain the target retrieval results, and send the target retrieval results to the gateway layer.

[0133] The data processing layer includes unstructured data extraction service, data preprocessing service, vectorized training service and Milvus. Among them, Milvus is a repository of vector data. The data processing layer can extract the data to be retrieved in the retrieval request, and the data to be retrieved is the unstructured data to be retrieved. Then, the data processing layer preprocesses the unstructured data to be retrieved and converts the preprocessed unstructured data to be retrieved into the vector to be retrieved. The data processing layer can call Milvus to retrieve the vector to be retrieved, obtain the retrieval results, and send the retrieval results to the business service layer.

[0134] The data processing layer can also obtain manually entered unstructured data or unstructured data stored in the file server. Then, the unstructured data is preprocessed and the preprocessed unstructured data is vectorized to obtain vectors. The data processing layer can also store the vectors in Milvus.

[0135] The unstructured data retrieval system also includes a deployment environment layer. The deployment environment layer includes physical machines, VPS docker, and centos7. Among them, VPS docker is a Docker container environment running on a virtual private server, providing application containerized hosting services. Docker is an open source application container engine. Centos7 can be used as a server operating system, providing a stable and reliable environment, suitable for server deployment.

[0136] Reference Figure 3 , showing a flow chart of a method for retrieving unstructured data provided in an embodiment of the present invention. The unstructured data retrieval system can use the data processing layer to periodically pull unstructured data from the file video server. Unstructured data includes one or more of text data, video data, image data, behavior data and audio data. The unstructured data retrieval system can use the data processing layer to preprocess the unstructured data, and the preprocessing includes: removing noise from the unstructured data; and / or, classifying the unstructured data and adding labels; and / or, extracting features of the unstructured data; and / or, standardizing the unstructured data. The unstructured data retrieval system uses the data processing layer to perform vectorization training on the preprocessed unstructured data, converts the preprocessed unstructured data into vectors, and stores the vectors in a preset Milvus vector database.

[0137] The unstructured data retrieval system can use the gateway layer to obtain the user's unstructured data retrieval request, which is the user query. The unstructured data retrieval system sends the retrieval request to the business service layer, and uses the business service layer to send the retrieval request to the data processing layer. The unstructured data retrieval system uses the data processing layer to extract the data to be retrieved in the retrieval request, and the data to be retrieved is the unstructured data to be retrieved. Then, the data processing layer is used to preprocess the unstructured data to be retrieved, and the preprocessed unstructured data to be retrieved is converted into a vector to be retrieved. Based on the Milvus vector database, the data processing layer is used to retrieve the vector to be retrieved to obtain the retrieval result, which is the result set. The unstructured data retrieval system uses the data processing layer to send the result set to the business service layer, and the business service layer uses a preset similarity script to filter the retrieval results to obtain the target retrieval results, and sends the target retrieval results to the gateway layer, and uses the gateway layer to output the target retrieval results.

[0138] It should be noted that, for the sake of simplicity, the method embodiments are described as a series of action combinations, but those skilled in the art should be aware that the embodiments of the present invention are not limited by the order of the actions described, because according to the embodiments of the present invention, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required by the embodiments of the present invention.

[0139] Reference Figure 4 , shows a structural block diagram of an unstructured data retrieval device provided in an embodiment of the present invention, which is applied to an unstructured data retrieval system. The unstructured data retrieval system includes at least a gateway layer, a business service layer, and a data processing layer; specifically, it may include the following modules:

[0140] The data acquisition module 401 is used to acquire preset unstructured data using the data processing layer;

[0141] A vector conversion module 402, configured to convert the unstructured data into vectors using the data processing layer, and store the vectors into a preset vector storage library;

[0142] A search request acquisition module 403 is used to acquire the user's search request for the unstructured data by using the gateway layer, and send the search request to the business service layer;

[0143] A search request sending module 404, used to send the search request to the data processing layer using the business service layer;

[0144] A retrieval module 405 is used to retrieve the data to be retrieved in the retrieval request based on the vector repository and using the data processing layer to obtain a retrieval result;

[0145] A screening module 406 is used to screen the search results by using a preset similarity script through the business service layer to obtain a target search result, and send the target search result to the gateway layer;

[0146] The output module 407 is used to output the target search result using the gateway layer.

[0147] In an optional embodiment of the present invention, the unstructured data includes at least one or more of text data, video data, image data and audio data; and the vector conversion module includes:

[0148] A first vector conversion submodule, used to convert the text data into a text vector using a preset word embedding model;

[0149] A second vector conversion submodule, used to convert the video data, the image data and the audio data into a video vector, an image vector and an audio vector respectively by using a preset convolutional neural network model and / or a recurrent neural network model;

[0150] The vector storage submodule is used to store the text vector, the video vector, the image vector and the audio vector into the vector storage repository.

[0151] In an optional embodiment of the present invention, the device comprises:

[0152] an unstructured data processing module, used to remove noise from the unstructured data; and / or,

[0153] classifying the unstructured data; and / or,

[0154] extracting features of the unstructured data; and / or,

[0155] The unstructured data is standardized.

[0156] In an optional embodiment of the present invention, the retrieval module includes:

[0157] An extraction submodule, used for extracting the data to be retrieved in the search request; the data to be retrieved is unstructured data to be retrieved;

[0158] A preprocessing submodule, used for preprocessing the unstructured data to be retrieved;

[0159] A third vector conversion submodule, used for converting the preprocessed unstructured data to be retrieved into a vector to be retrieved;

[0160] The retrieval submodule is used to use the data processing layer to search the vector to be searched to obtain the search result.

[0161] In an optional embodiment of the present invention, the retrieval submodule includes:

[0162] A radius acquisition unit, used to acquire a search radius of a preset length;

[0163] The retrieval result is used as a unit, and is used to take the vectors within the retrieval radius around the vector to be retrieved in the vector storage as the retrieval result.

[0164] In an optional embodiment of the present invention, the similarity script at least involves cosine similarity and / or Euclidean distance.

[0165] In an optional embodiment of the present invention, the data acquisition module includes:

[0166] The data acquisition submodule is used to acquire the unstructured data from a preset file server using the data processing layer.

[0167] As for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0168] In addition, an embodiment of the present invention further provides an electronic device, such as Figure 5 As shown, it includes a processor 501, a communication interface 502, a memory 503 and a communication bus 504, wherein the processor 501, the communication interface 502, and the memory 503 communicate with each other through the communication bus 504.

[0169] Memory 503, used for storing computer programs;

[0170] The processor 501 is used to execute the program stored in the memory 503, and implements the following steps:

[0171] Acquiring preset unstructured data using the data processing layer;

[0172] Using the data processing layer to convert the unstructured data into vectors, and storing the vectors in a preset vector storage library;

[0173] Utilizing the gateway layer to obtain a user's retrieval request for the unstructured data, and sending the retrieval request to the business service layer;

[0174] Using the business service layer to send the search request to the data processing layer;

[0175] Based on the vector repository, using the data processing layer to retrieve the data to be retrieved in the retrieval request to obtain a retrieval result;

[0176] The business service layer uses a preset similarity script to screen the search results to obtain target search results, and sends the target search results to the gateway layer;

[0177] The target search result is outputted using the gateway layer.

[0178] In an optional embodiment of the present invention, the unstructured data includes at least one or more of text data, video data, image data and audio data; the step of converting the unstructured data into a vector and storing the vector in a preset vector storage library includes:

[0179] Convert the text data into text vectors using a preset word embedding model;

[0180] Using a preset convolutional neural network model and / or a recurrent neural network model, the video data, the image data, and the audio data are converted into a video vector, an image vector, and an audio vector, respectively;

[0181] The text vector, the video vector, the image vector and the audio vector are stored in the vector storage library.

[0182] In an optional embodiment of the present invention, the step of converting the unstructured data into a vector includes:

[0183] removing noise from the unstructured data; and / or,

[0184] classifying the unstructured data; and / or,

[0185] extracting features of the unstructured data; and / or,

[0186] The unstructured data is standardized.

[0187] In an optional embodiment of the present invention, the step of using the data processing layer to retrieve the data to be retrieved in the search request to obtain the search result includes:

[0188] Extracting the data to be retrieved in the search request; the data to be retrieved is unstructured data to be retrieved;

[0189] Preprocessing the unstructured data to be retrieved;

[0190] Converting the preprocessed unstructured data to be retrieved into vectors to be retrieved;

[0191] The data processing layer is used to search the vector to be searched to obtain the search result.

[0192] In an optional embodiment of the present invention, the step of using the data processing layer to search the vector to be searched to obtain the search result includes:

[0193] Get the search radius of preset length;

[0194] The vectors within the search radius around the vector to be searched in the vector storage are taken as the search results.

[0195] In an optional embodiment of the present invention, the similarity script at least involves cosine similarity and / or Euclidean distance.

[0196] In an optional embodiment of the present invention, the step of acquiring preset unstructured data by using the data processing layer includes:

[0197] The data processing layer is used to obtain the unstructured data from a preset file server.

[0198] The communication bus mentioned in the above terminal can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0199] The communication interface is used for communication between the above terminal and other devices.

[0200] The memory may include a random access memory (RAM) or a non-volatile memory, such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0201] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0202] like Figure 6 As shown, in another embodiment provided by the present invention, a computer-readable storage medium 601 is also provided, in which instructions are stored. When the computer-readable storage medium is run on a computer, the computer executes a method for retrieving unstructured data described in the above embodiment.

[0203] In another embodiment of the present invention, a computer program product including instructions is provided. When the computer program product is run on a computer, the computer executes the unstructured data retrieval method described in the above embodiment.

[0204] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented by software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present invention is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state hard disk Solid State Disk (SSD)), etc.

[0205] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0206] Each embodiment in this specification is described in a related manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0207] The above description is only a preferred embodiment of the present invention and is not intended to limit the protection scope of the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention are included in the protection scope of the present invention.

Claims

1. A method for retrieving unstructured data, characterized in that: Applied to an unstructured data retrieval system, the unstructured data retrieval system at least includes a gateway layer, a business service layer and a data processing layer, and the method includes: Acquiring preset unstructured data using the data processing layer; Using the data processing layer to convert the unstructured data into vectors, and storing the vectors in a preset vector storage library; Utilizing the gateway layer to obtain a user's retrieval request for the unstructured data, and sending the retrieval request to the business service layer; Using the business service layer to send the search request to the data processing layer; Based on the vector repository, using the data processing layer to retrieve the data to be retrieved in the retrieval request to obtain a retrieval result; The business service layer uses a preset similarity script to screen the search results to obtain target search results, and sends the target search results to the gateway layer; The target search result is outputted using the gateway layer.

2. The method according to claim 1, characterized in that The unstructured data at least includes one or more of text data, video data, image data and audio data; the step of converting the unstructured data into vectors and storing the vectors in a preset vector storage library includes: Convert the text data into text vectors using a preset word embedding model; Using a preset convolutional neural network model and / or a recurrent neural network model, the video data, the image data, and the audio data are converted into a video vector, an image vector, and an audio vector, respectively; The text vector, the video vector, the image vector and the audio vector are stored in the vector storage library.

3. The method according to claim 1, characterized in that The step of converting the unstructured data into a vector includes: removing noise from the unstructured data; and / or, classifying the unstructured data; and / or, extracting features of the unstructured data; and / or, The unstructured data is standardized.

4. The method according to claim 1, characterized in that: The step of using the data processing layer to retrieve the data to be retrieved in the retrieval request to obtain the retrieval result includes: Extracting the data to be retrieved in the search request; the data to be retrieved is unstructured data to be retrieved; Preprocessing the unstructured data to be retrieved; Converting the preprocessed unstructured data to be retrieved into vectors to be retrieved; The data processing layer is used to search the vector to be searched to obtain the search result.

5. The method according to claim 4, characterized in that The step of using the data processing layer to search the vector to be searched to obtain the search result comprises: Get the search radius of the preset length; The vectors within the search radius around the vector to be searched in the vector storage are taken as the search results.

6. The method according to claim 1, characterized in that The similarity script involves at least cosine similarity and / or Euclidean distance.

7. The method according to claim 1, characterized in that The step of using the data processing layer to obtain preset unstructured data includes: The data processing layer is used to obtain the unstructured data from a preset file server.

8. A retrieval device for unstructured data, characterized in that: Applied to an unstructured data retrieval system, the unstructured data retrieval system at least includes a gateway layer, a business service layer and a data processing layer; the device includes: A data acquisition module, used to acquire preset unstructured data using the data processing layer; A vector conversion module, used to convert the unstructured data into vectors using the data processing layer, and store the vectors into a preset vector storage library; A retrieval request acquisition module, used to use the gateway layer to acquire the user's retrieval request for the unstructured data, and send the retrieval request to the business service layer; A search request sending module, used to send the search request to the data processing layer using the business service layer; A retrieval module, configured to retrieve the data to be retrieved in the retrieval request based on the vector repository and using the data processing layer to obtain a retrieval result; A screening module, used to screen the search results by using a preset similarity script through the business service layer to obtain a target search result, and send the target search result to the gateway layer; An output module is used to output the target search result using the gateway layer.

9. An electronic device, characterized in that: It includes a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other through the communication bus; The memory is used to store computer programs; The processor is used to implement the method according to any one of claims 1 to 7 when executing the program stored in the memory.

10. One or more computer-readable media having instructions stored thereon, which when executed by one or more processors cause the processors to perform the method according to any one of claims 1 to 7.

Images