Data processing method and device, equipment, storage medium and product
Through the method of feature extraction and model selection of target text data, the problem of low efficiency in text data vectorization in the prior art is solved, and more efficient text data vectorization and better adaptability are achieved.
Patent Information
- Application Number
- CN202411573506.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-11-05
- Publication Date
- 2025-02-18
AI Technical Summary
The existing data vectorization efficiency through fixed text embedding models is not high, resulting in low adaptability between the vectorized data and text data, and low text vectorization efficiency.
By extracting the target text data features, basic features and fine-grained features are obtained, the target text embedding model is determined based on these features, and the text data is vectorized based on the model.
Dynamic selection of text embedding models improves the efficiency of text data vectorization and enhances the adaptability of vectorized data and text data.
Smart Images
Figure CN119474251A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of data processing technology, and in particular to data processing methods, devices, equipment, storage media and products. Background Art
[0002] In machine learning and deep learning, data is usually represented in the form of vectors. Vector databases are databases specifically used to store and query vectors. The vectors they store come from the vectorization of text, speech, images, videos, etc. Compared with traditional databases, vector databases can handle more unstructured data (such as images and audio). However, the existing applications of vector databases all first vectorize text data through a single text embedding model, and then store the vectorized data in the vector database. The above method easily leads to low compatibility between the vectorized data and the text data, and low text vectorization efficiency. Summary of the invention
[0003] The main purpose of this application is to provide a data processing method, device, equipment, storage medium and product, aiming to solve the technical problem of low efficiency of data vectorization through fixed text embedding models.
[0004] To achieve the above purpose, the present application proposes a data processing method, which includes:
[0005] Perform feature extraction on the target text data to obtain feature extraction results;
[0006] Determine a target text embedding model corresponding to the target text data based on the feature extraction result;
[0007] The target text data is vectorized according to the target text embedding model to obtain vectorized data corresponding to the target text data.
[0008] Optionally, the feature extraction result includes basic features and fine-grained features;
[0009] The step of extracting features from the target text data to obtain feature extraction results includes:
[0010] Extracting basic features of the target text data to obtain basic features corresponding to the target text data;
[0011] Fine-grained features are extracted from the target text data to obtain fine-grained features corresponding to the target text data.
[0012] Optionally, the step of determining a target text embedding model corresponding to the target text data based on the feature extraction result includes:
[0013] Selecting a candidate model corresponding to the target text data according to the basic features;
[0014] A target text embedding model corresponding to the target text data is determined according to the fine-grained features and the candidate models.
[0015] Optionally, the step of determining a target text embedding model corresponding to the target text data according to the fine-grained features and the candidate model comprises:
[0016] Performing an accuracy test on the candidate model based on a preset test text set to obtain an accuracy test result;
[0017] Determining the adaptability of the candidate model according to the fine-grained features;
[0018] A target text embedding model corresponding to the target text data is determined based on the adaptability and the accuracy test results.
[0019] Optionally, the step of vectorizing the target text data according to the target text embedding model to obtain vectorized data corresponding to the target text data includes:
[0020] Determining a current state of the target text embedding model;
[0021] When the current state is the target state, the target text data is vectorized by using the target text embedding model to obtain vectorized data corresponding to the target text data.
[0022] Optionally, after the step of vectorizing the target text data according to the target text embedding model to obtain vectorized data corresponding to the target text data, the method further includes:
[0023] Determine an inverted index corresponding to the target text data;
[0024] The vectorized data is stored based on the inverted index.
[0025] In addition, to achieve the above-mentioned purpose, the present application also proposes a data processing device, the data processing device comprising:
[0026] A feature extraction module is used to extract features from target text data and obtain feature extraction results;
[0027] A determination module, used for determining a target text embedding model corresponding to the target text data based on the feature extraction result;
[0028] A vectorization module is used to vectorize the target text data according to the target text embedding model to obtain vectorized data corresponding to the target text data.
[0029] In addition, to achieve the above objectives, the present application also proposes a data processing device, which includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the data processing method described above.
[0030] In addition, to achieve the above objectives, the present application also proposes a storage medium, which is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the data processing method described above are implemented.
[0031] In addition, to achieve the above-mentioned purpose, the present application also provides a computer program product, which includes a computer program, and when the computer program is executed by a processor, the steps of the data processing method described above are implemented.
[0032] The present application performs feature extraction on target text data to obtain feature extraction results; determines a target text embedding model corresponding to the target text data based on the feature extraction results; and vectorizes the target text data according to the target text embedding model to obtain vectorized data corresponding to the target text data. Since the present application determines a target text embedding model based on the features of the target text data, and vectorizes the target text data according to the target text embedding model, compared to the existing method of vectorizing text data through a fixed text embedding model, the above method of the present application can dynamically select a text embedding model according to the features of the text data, thereby improving the efficiency of text data vectorization. BRIEF DESCRIPTION OF THE DRAWINGS
[0033] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.
[0034] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.
[0035] Figure 1 A flowchart of the first embodiment of the data processing method of the present application is provided;
[0036] Figure 2 A schematic diagram of a data processing device module provided for the first embodiment of the data processing method of the present application;
[0037] Figure 3A flowchart of the second embodiment of the data processing method of this application is provided;
[0038] Figure 4 This is a schematic diagram of the module structure of the data processing device according to an embodiment of the present application;
[0039] Figure 5 Schematic diagram of the device structure of the hardware operating environment involved in the data processing method in the embodiment of the present application.
[0040] The purpose, features and advantages of this application will be further described in conjunction with the embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION
[0041] It should be understood that the specific embodiments described herein are only used to explain the technical solutions of the present application and are not used to limit the present application.
[0042] In order to better understand the technical solution of the present application, a detailed description will be given below in conjunction with the accompanying drawings and specific implementation methods.
[0043] The main solution of the embodiment of the present application is: perform feature extraction on the target text data to obtain feature extraction results; determine the target text embedding model corresponding to the target text data based on the feature extraction results; vectorize the target text data according to the target text embedding model to obtain vectorized data corresponding to the target text data. Since the present application determines the target text embedding model according to the characteristics of the target text data, and vectorizes the target text data according to the target text embedding model, compared to the existing method of vectorizing text data through a fixed text embedding model, the above method of the present application can dynamically select a text embedding model according to the characteristics of the text data, thereby improving the efficiency of text data vectorization.
[0044] It should be noted that the execution subject of this embodiment may be a computing service device with data processing, network communication and program running functions, such as a tablet computer, a personal computer, a mobile phone, etc., or an electronic device capable of realizing the above functions, a data processing device, etc. The following takes a data processing device as an example to illustrate this embodiment and the following embodiments.
[0045] Based on this, the present application embodiment provides a data processing method, referring to Figure 1 , Figure 1 A flowchart of the first embodiment of the data processing method of the present application is provided.
[0046] In this embodiment, the data processing method includes the following steps:
[0047] Step S10, performing feature extraction on the target text data to obtain a feature extraction result;
[0048] It should be noted that the target text data may be text data that needs to be vectorized and stored in a vector database. A vector database is built in the data processing device to store vectorized data of the target text data. The feature extraction of the target text data may be performed by extracting features from the target text data using a common feature extraction algorithm. The feature extraction result of the target text data is obtained.
[0049] Furthermore, in order to better recommend a corresponding target text embedding model for the target text data and improve the vectorization efficiency of the target text data, the feature extraction result includes basic features and fine-grained features; the step S10 includes:
[0050] Extracting basic features of the target text data to obtain basic features corresponding to the target text data;
[0051] Fine-grained features are extracted from the target text data to obtain fine-grained features corresponding to the target text data.
[0052] It should be noted that the basic feature extraction of the target text data may include first performing text preprocessing on the target text data, including word segmentation, stop word removal and normalization. Then, feature extraction such as word frequency analysis, text length calculation and topic distribution analysis is performed on the target text data to obtain the basic features corresponding to the target text data. Specifically, word frequency analysis may be to calculate the frequency of occurrence of words in the target text data using the TF-IDF algorithm; text length calculation may be to count the number of characters and words in the target text data; topic distribution analysis may be to analyze the text topic distribution using the LDA topic model. The fine-grained feature extraction of the target text data to obtain the fine-grained features corresponding to the target text data may be to perform keyword extraction, named entity recognition and other processing on the target text data. Specifically, keyword extraction may be to use the TextRank algorithm to identify important keywords in the target text data; named entity recognition may be to extract entity information in the target text data using the BERT-based named entity recognition model to obtain the fine-grained features corresponding to the target text data.
[0053] Step S20, determining a target text embedding model corresponding to the target text data based on the feature extraction result;
[0054] It should be noted that the target text embedding model corresponding to the target text data determined based on the feature extraction result can be a target text embedding model that can better understand and process the target text data selected from a plurality of optional text embedding models according to the feature extraction result. For example, if the feature extraction result shows that the target text data is a short text, the target text embedding model can be FastText, and if the feature extraction result shows that the target text data is a long text, the target text embedding model can be a BERT model.
[0055] Step S30, vectorizing the target text data according to the target text embedding model to obtain vectorized data corresponding to the target text data.
[0056] It should be noted that the target text data is vectorized according to the target text embedding model to obtain the vectorized data corresponding to the target text data, which may be performed by calling the target text embedding model to vectorize the target text data to obtain the vectorized data corresponding to the target text data.
[0057] In the specific implementation, a vector database is built in the data processing device, and a vectorized model recommendation module is embedded in the vector database kernel. The vectorized model recommendation module can dynamically recommend a text embedding model based on the text features of the target text data (i.e., the feature extraction results), and then use the text embedding model to vectorize the target text data. At the same time, the vector database has also adjusted its kernel architecture and introduced direct support for text types, so that text can be directly converted into vectorized data and then stored in the vector database, without the need for users to convert the text into vectors and then store them in the vector database.
[0058] Further, in order to improve the speed of text vectorization, the step S30 may include: determining the current state of the target text embedding model;
[0059] When the current state is the target state, the target text data is vectorized by using the target text embedding model to obtain vectorized data corresponding to the target text data.
[0060] It should be noted that the current state of the target text embedding model may include an available state and an unavailable state. The unavailable state may be a state where the model is in a state to be evaluated or is executing a task. The target state may be an available state. When the current state is the target state, i.e., the available state, the target text data is vectorized by the target text embedding model to obtain the vectorized data corresponding to the target text data, and the vectorized data is stored in a vector database. When the current state is not the target state, i.e., the unavailable state, an alternative model is selected to perform vectorization of the text data (also called a query operation), which can avoid query delays and ensure the real-time performance and response speed of the data processing equipment.
[0061] In a specific implementation, the vector database kernel built in the data processing device may include a text feature extraction module, an embedding model recommendation module, and a vector database query optimization module. Figure 2 , Figure 2 A schematic diagram of a data processing device module provided for the first embodiment of the data processing method of the present application; wherein the vector database query optimization module is located after the embedding model recommendation module, and is responsible for optimizing the selection and application of the target text embedding model during the query process. The model status of the target text embedding model is obtained during the query. If the model is in an unavailable state such as pending evaluation, the model is ignored. An alternative model is obtained to vectorize the target text data. Specifically, before executing the query, check whether the recommended target text embedding model has completed the evaluation. If the target text embedding model is still being evaluated, an alternative model is selected for query. This can avoid query delays and ensure the real-time and response speed of the system. After the target text embedding model passes the evaluation, it is applied to the query operation to improve the query accuracy and efficiency. The alternative model can be a text embedding model that is second only to the target text embedding model in step S20. Figure 2The text feature extraction module in is used to extract the features of the target text data, including preprocessing operations on the target text data, such as word segmentation, stop word removal and normalization, and then analyzing the features of the target text data, including basic feature extraction and fine-grained feature extraction. The embedding model recommendation module includes determining the preliminary candidate model based on the basic features of the target text data and determining the target text embedding model based on the fine-grained features. The vector database query optimization module includes model state judgment before querying and determining the model of the available state, and then performing the operation of vectorizing the target text data based on the available model. In this embodiment, an original data storage module may also be included to store the target text data input by the user. Each module is provided with an interface so that it can interact with other modules for data, and the interface design adopts the gRPC framework for communication. In this embodiment, the model recommendation result of the embedding model recommendation module includes the recommended target text embedding model and its parameter settings, and the optimal embedding method is selected according to the recommended target text embedding model for querying. In addition, by introducing Bayesian optimization technology, the selection strategy of the target text embedding model is adjusted in real time according to the query conditions and the data characteristics of the target text data to improve the accuracy and efficiency of the query. The specific processing includes: optimizing the parameters of the target text embedding model, adjusting the query process to adapt to the recommended target text embedding model, and ensuring the accuracy and efficiency of the query results.
[0062] Furthermore, in order to support efficient text retrieval and other data operations, after step S30, the method further includes: determining an inverted index corresponding to the target text data;
[0063] The vectorized data is stored based on the inverted index.
[0064] It should be noted that, in order to enable the vector database in the data processing device to directly process the target text data input by the user, the data processing device of this embodiment also includes an original data storage module for storing the target text data directly input by the user. Then, the index mechanism of the target text data is implemented in the original data storage module to support efficient text retrieval. Specifically, an inverted index data structure is adopted to establish a position list containing the word appearing in the document for each word in the target text data. The inverted index consists of two main parts: realizing fast full-text search and partial matching query functions. It includes establishing a dictionary and an inverted list. The dictionary is used to store all index words, and a balanced binary tree (AVL tree) data structure is adopted to support fast search and insertion operations. Each word points to its corresponding inverted list. Inverted list: stores the document ID and position information containing the word. The skip list is introduced to optimize the access speed of the inverted list, and a jump pointer is added to the inverted list so that the search operation is completed in logarithmic time. In the process of index construction, multi-threaded parallel processing technology is adopted to speed up the index construction speed. Determining the inverted index corresponding to the target text data and storing the vectorized data based on the inverted index may be determining the inverted index corresponding to the target text data according to an inverted list of the target text data, and then storing the vectorized data corresponding to the target text data based on the inverted index.
[0065] This embodiment performs feature extraction on the target text data to obtain feature extraction results; determines a target text embedding model corresponding to the target text data based on the feature extraction results; and vectorizes the target text data according to the target text embedding model to obtain vectorized data corresponding to the target text data. Since this embodiment determines the target text embedding model based on the features of the target text data, and vectorizes the target text data according to the target text embedding model, compared to the existing method of vectorizing text data through a fixed text embedding model, the above method of this embodiment can dynamically select a text embedding model according to the features of the text data, thereby improving the efficiency of text data vectorization.
[0066] Based on the first embodiment of the present application, in the second embodiment of the present application, the same or similar contents as those in the above-mentioned embodiment 1 can be referred to the above introduction, and will not be repeated in the following. Figure 3 , Figure 3 The flowchart provided in the second embodiment of the data processing method of the present application, the step S20 further includes the following steps:
[0067] Step S201: selecting a candidate model corresponding to the target text data according to the basic features;
[0068] It should be noted that the selection of the candidate model corresponding to the target text data according to the basic features can be based on the basic features of the target text data, preliminarily screening out a list of candidate models, and matching a suitable embedding model. For example, if the target text data is a short text, FastText is recommended, and if the target text data is a long text, BERT is recommended. Then, the model performance in the candidate model list is preliminarily evaluated based on the pre-constructed model performance database, and the candidate models with better performance are screened out. The model performance database stores the performance test results of each candidate text embedding model obtained after the candidate text embedding model is pre-tested based on each test sample data.
[0069] Step S202: Determine a target text embedding model corresponding to the target text data according to the fine-grained features and the candidate models.
[0070] It should be noted that determining the target text embedding model corresponding to the target text data based on the fine-grained features and the candidate models can be based on the fine-grained features of the target text data, evaluating the adaptability of the model, and then selecting a text embedding model that is more closely matched with the fine-grained features of the target text data, such as the domain, entity, keyword, etc.
[0071] In order to make the vectorized data more consistent with the semantic information of the target text data, the step S202 may include: performing an accuracy test on the candidate model based on a preset test text set to obtain an accuracy test result;
[0072] Determining the suitability of the candidate model according to the fine-grained features;
[0073] A target text embedding model corresponding to the target text data is determined based on the adaptability and the accuracy test results.
[0074] It should be noted that, in this embodiment, a predefined test text set (i.e., the preset test text set) is used to perform a precision test on each candidate model to evaluate its embedding effect. Specifically, a test text that is most similar to the subject, semantics, text length, and other information of the target text data can be selected from the predefined test text set, and then a precision test is performed on each candidate model according to the test text to obtain a precision test result. Then, the target text embedding model is selected based on the precision test result and the adaptability of the candidate model. The score corresponding to the precision test result and the score corresponding to the adaptability of the candidate model can be added together and the candidate model with the highest total score can be selected as the target text embedding model. The score can also be calculated after assigning corresponding weights to the precision test result and adaptability.
[0075] This embodiment selects a candidate model corresponding to the target text data according to the basic features; and determines a target text embedding model corresponding to the target text data according to the fine-grained features and the candidate models. This embodiment recommends a target text embedding model corresponding to the target text data based on the basic features and fine-grained features of the target text data. It can improve the vectorization efficiency of the target text data.
[0076] It should be noted that the above examples are only used to understand the present application and do not constitute a limitation on the data processing method of the present application. More simple transformations based on this technical concept are all within the scope of protection of the present application.
[0077] This application also provides a data processing device, please refer to Figure 4 , the data processing device comprises:
[0078] The feature extraction module 10 is used to extract features from the target text data and obtain feature extraction results;
[0079] A determination module 20, configured to determine a target text embedding model corresponding to the target text data based on the feature extraction result;
[0080] The vectorization module 30 is used to vectorize the target text data according to the target text embedding model to obtain vectorized data corresponding to the target text data.
[0081] This embodiment performs feature extraction on the target text data to obtain feature extraction results; determines a target text embedding model corresponding to the target text data based on the feature extraction results; and vectorizes the target text data according to the target text embedding model to obtain vectorized data corresponding to the target text data. Since this embodiment determines the target text embedding model based on the features of the target text data, and vectorizes the target text data according to the target text embedding model, compared to the existing method of vectorizing text data through a fixed text embedding model, the above method of this embodiment can dynamically select a text embedding model according to the features of the text data, thereby improving the efficiency of text data vectorization.
[0082] The data processing device provided by the present application adopts the data processing method in the above embodiment, which can solve the technical problem of low efficiency of data vectorization through a fixed text embedding model. Compared with the prior art, the beneficial effects of the data processing device provided by the present application are the same as the beneficial effects of the data processing method provided by the above embodiment, and other technical features in the data processing device are the same as the features disclosed in the above embodiment method, which will not be repeated here.
[0083] The present application provides a data processing device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the data processing method in the above-mentioned embodiment 1.
[0084] Reference below Figure 5 , which shows a schematic diagram of the structure of a data processing device suitable for implementing an embodiment of the present application. The data processing device in the embodiment of the present application may include, but is not limited to, mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions), PMPs (Portable Media Players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 5 The data processing device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.
[0085] like Figure 5 As shown, the data processing device may include a processing device 1001 (e.g., a central processing unit, a graphics processor, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM: Read Only Memory) 1002 or a program loaded from a storage device 1003 to a random access memory (RAM: Random Access Memory) 1004. Various programs and data required for the operation of the data processing device are also stored in the RAM 1004. The processing device 1001, the ROM 1002, and the RAM 1004 are connected to each other via a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Typically, the following systems may be connected to the I / O interface 1006: input devices 1007 including, for example, a touch screen, a touchpad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; output devices 1008 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; storage devices 1003 including, for example, a magnetic tape, a hard disk, etc.; and communication devices 1009. The communication device 1009 may allow the data processing device to communicate with other devices wirelessly or by wire to exchange data. Although the figure shows a data processing device with various systems, it should be understood that it is not required to implement or have all the systems shown. More or fewer systems may be implemented or have alternatively.
[0086] In particular, according to the embodiments disclosed in the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device, or installed from a storage device 1003, or installed from a ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiment disclosed in the present application are executed.
[0087] The data processing device provided by the present application adopts the data processing method in the above embodiment, which can solve the technical problem of low efficiency of data vectorization through a fixed text embedding model. Compared with the prior art, the beneficial effects of the data processing device provided by the present application are the same as the beneficial effects of the data processing method provided by the above embodiment, and other technical features in the data processing device are the same as the features disclosed in the method of the previous embodiment, which will not be repeated here.
[0088] It should be understood that the various parts disclosed in this application can be implemented by hardware, software, firmware or a combination thereof. In the description of the above embodiments, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.
[0089] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.
[0090] The present application provides a computer-readable storage medium having computer-readable program instructions (ie, computer programs) stored thereon, wherein the computer-readable program instructions are used to execute the data processing method in the above-mentioned embodiment.
[0091] The computer-readable storage medium provided in the present application may be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, systems or devices, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, system or device. The program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination of the above.
[0092] The computer-readable storage medium may be included in a data processing device, or may exist independently without being incorporated into a data processing device.
[0093] Computer program code for performing the operations of the present application may be written in one or more programming languages or a combination thereof, including object-oriented programming languages such as Java, Smalltalk, C++, and conventional procedural programming languages such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).
[0094] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.
[0095] The modules involved in the embodiments described in this application may be implemented by software or hardware, wherein the name of the module does not constitute a limitation on the unit itself in some cases.
[0096] The readable storage medium provided in the present application is a computer-readable storage medium, which stores computer-readable program instructions (i.e., computer programs) for executing the above-mentioned data processing method, and can solve the technical problem that the existing data vectorization is inefficient through a fixed text embedding model. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in the present application are the same as the beneficial effects of the data processing method provided in the above-mentioned embodiment, and will not be repeated here.
[0097] The present application also provides a computer program product, including a computer program, which implements the steps of the above-mentioned data processing method when executed by a processor.
[0098] The computer program product provided by the present application can solve the technical problem of low efficiency of data vectorization through a fixed text embedding model. Compared with the prior art, the beneficial effects of the computer program product provided by the present application are the same as the beneficial effects of the data processing method provided by the above embodiment, which will not be repeated here.
[0099] The above descriptions are only some embodiments of the present application, and are not intended to limit the patent scope of the present application. All equivalent structural changes made using the contents of the present application specification and drawings under the technical concept of the present application, or direct / indirect applications in other related technical fields are included in the patent protection scope of the present application.
Claims
1. A data processing method, characterized in that: The data processing method comprises the following steps: Perform feature extraction on the target text data to obtain feature extraction results; Determine a target text embedding model corresponding to the target text data based on the feature extraction result; The target text data is vectorized according to the target text embedding model to obtain vectorized data corresponding to the target text data.
2. The data processing method according to claim 1, characterized in that: The feature extraction results include basic features and fine-grained features; The step of extracting features from the target text data to obtain feature extraction results includes: Extracting basic features of the target text data to obtain basic features corresponding to the target text data; Fine-grained features are extracted from the target text data to obtain fine-grained features corresponding to the target text data.
3. The data processing method according to claim 2, characterized in that: The step of determining the target text embedding model corresponding to the target text data based on the feature extraction result comprises: Selecting a candidate model corresponding to the target text data according to the basic features; A target text embedding model corresponding to the target text data is determined according to the fine-grained features and the candidate models.
4. The data processing method according to claim 3, characterized in that: The step of determining the target text embedding model corresponding to the target text data according to the fine-grained features and the candidate models comprises: Performing an accuracy test on the candidate model based on a preset test text set to obtain an accuracy test result; Determining the suitability of the candidate model according to the fine-grained features; A target text embedding model corresponding to the target text data is determined based on the adaptability and the accuracy test results.
5. The data processing method according to any one of claims 1 to 4, characterized in that: The step of vectorizing the target text data according to the target text embedding model to obtain vectorized data corresponding to the target text data includes: Determining a current state of the target text embedding model; When the current state is the target state, the target text data is vectorized by using the target text embedding model to obtain vectorized data corresponding to the target text data.
6. The data processing method according to any one of claims 1 to 4, characterized in that: After the step of vectorizing the target text data according to the target text embedding model to obtain vectorized data corresponding to the target text data, the method further includes: Determine an inverted index corresponding to the target text data; The vectorized data is stored based on the inverted index.
7. A data processing device, characterized in that: The data processing device comprises: A feature extraction module is used to extract features from target text data and obtain feature extraction results; A determination module, used for determining a target text embedding model corresponding to the target text data based on the feature extraction result; A vectorization module is used to vectorize the target text data according to the target text embedding model to obtain vectorized data corresponding to the target text data.
8. A data processing device, characterized in that: The device comprises: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the data processing method according to any one of claims 1 to 6.
9. A storage medium, characterized in that: The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the data processing method according to any one of claims 1 to 6 are implemented.
10. A computer program product, characterized in that The computer program product comprises a computer program, and when the computer program is executed by a processor, the steps of the data processing method according to any one of claims 1 to 6 are implemented.