Vector conversion method, apparatus and system for semi-structured data, and device and medium

Abstract

A vector conversion method, apparatus and system for semi-structured data, and a device and a medium, which relate to the technical field of big data processing. The method comprises: reading a first semi-structured data file, and acquiring data to be converted (S302); parsing a pre-configured rule file, and determining a correlation between said data and a field to be extracted in the rule file, wherein the rule file is a file comprising a first mapping relationship between said field and a data structure of vector storage middleware, and a second mapping relationship between said field and a vector conversion algorithm (S304); and on the basis of the correlation and the second mapping relationship, calling the vector conversion algorithm to perform vector conversion on said data, and performing storage in the vector storage middleware on the basis of the first mapping relationship (S306). The efficiency of vector conversion of semi-structured data can be improved by means of a rule file, and storage is performed in a corresponding vector database, thereby realizing the persistence of the vector database; and direct calling is performed during use, thereby achieving convenience and fast performance.

Description

Semi-structured data vector conversion method, device, system, equipment and medium

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This disclosure claims priority to Chinese patent application number 202311378508.0, filed on October 23, 2023, entitled “Method, device, system, equipment and medium for semi-structured data vector conversion”, the entire contents of which are incorporated herein by reference. Technical Field

[0003] The present disclosure relates to the field of big data processing technology, and in particular to a method, device, system, equipment and medium for converting semi-structured data vectors. Background Art

[0004] With the development of science and technology and the changes in IT architecture, semi-structured data is becoming increasingly important and is widely used in many application systems, interface calls, data storage and other fields. The concise and clear hierarchical structure has become an ideal data exchange language. When data is to be analyzed or finally stored on disk, it usually needs to be converted into other formats. Due to the emergence of artificial intelligence technology and various applications, the common practice is to vectorize the relevant data.

[0005] However, in related technologies, the processing efficiency of data vectorization is low and vectors are inconvenient to use.

[0006] It should be noted that the information disclosed in the above background technology section is only used to enhance the understanding of the background of the present disclosure, and therefore may include information that does not constitute prior art known to ordinary technicians in the field.

[0007] Summary of the Invention

[0008] The present disclosure provides a user plane routing selection method and related equipment in an air-space-ground computing power fusion network, which at least to a certain extent overcomes the technical problems in related technologies of low data vectorization processing efficiency and inconvenient vector use.

[0009] Other features and advantages of the present disclosure will become apparent from the following detailed description, or may be learned in part by practice of the present disclosure.

[0010] In a first aspect, embodiments of the present disclosure provide a method for converting a semi-structured data vector, the method comprising:

[0011] Reading the first semi-structured data file to obtain data to be converted;

[0012] Parsing a pre-configured rule file to determine the correspondence between the data to be converted and the fields to be extracted in the rule file; the rule file includes a first mapping relationship between the fields to be extracted and the data structure of the vector storage middleware, and a second mapping relationship between the fields to be extracted and the vectorized conversion algorithm;

[0013] A vector conversion algorithm is called to perform vector conversion on the data to be converted.

[0014] In a possible embodiment, the configuration method of the rule file includes:

[0015] Determining the hierarchical position and field name of the field to be extracted according to the second semi-structured data file;

[0016] A rule file is established, including a first mapping relationship between the field to be extracted and the data structure of the vector storage middleware, and a second mapping relationship between the field to be extracted and the vectorized conversion algorithm.

[0017] In a possible embodiment, determining the hierarchical position and field name of the field to be extracted based on the second semi-structured data file includes:

[0018] Simplifying the data information in the second semi-structured data file to obtain a basic framework structure corresponding to the second semi-structured data file;

[0019] Saving the basic framework structure as a semi-structured data template;

[0020] According to the semi-structured data template, the hierarchical position and field name of the field to be extracted are determined.

[0021] In a possible embodiment, determining the hierarchical position and field name of the field to be extracted includes:

[0022] Display all fields in the data selection interface;

[0023] In response to the user checking the fields to be extracted on the data selection interface, the hierarchical positions and field names of the multiple fields to be extracted are recorded.

[0024] In a possible embodiment, establishing a rule file including a first mapping relationship between the field to be extracted and the data structure of the vector storage middleware, and a second mapping relationship between the field to be extracted and the vectorized conversion algorithm includes:

[0025] Establishing a first mapping relationship according to each of the field names and the vector storage middleware;

[0026] Determining the hierarchical relationship according to the hierarchical position of each of the fields to be extracted;

[0027] If the hierarchical relationship is determined to be passed, a vector conversion algorithm is added to the fields to be extracted that have passed the determination;

[0028] The first mapping relationship and the second mapping relationship between the field to be extracted that passes the determination and the vectorized conversion algorithm are stored as a rule file.

[0029] In a possible embodiment, establishing a first mapping relationship according to each field name and the vector storage middleware includes:

[0030] Reading in schema information of a vector storage middleware; the schema information is used to describe a data structure in the vector storage middleware; the vector storage middleware is used to store vectors after semi-structured data conversion;

[0031] A first mapping relationship is established according to each of the field names and the mode information.

[0032] In a possible embodiment, determining the hierarchical relationship according to the hierarchical position of each of the fields to be extracted includes:

[0033] Determining the amount of data for each field to be extracted based on the hierarchical position of each field to be extracted;

[0034] If the amount of data in each of the fields to be extracted is consistent, the hierarchical relationship is determined to be successful.

[0035] In a possible embodiment, after calling a vector conversion algorithm to perform vector conversion on the data to be converted according to the corresponding relationship and the second mapping relationship and storing the data in the vector storage middleware according to the first mapping relationship, the method further includes:

[0036] In response to a user's service provision request, vectors corresponding to the semi-structured data are extracted from the vector storage middleware to provide services to external applications.

[0037] In a second aspect, an embodiment of the present disclosure provides a semi-structured data vector conversion device, comprising:

[0038] an acquisition unit, configured to read the first semi-structured data file and acquire data to be converted;

[0039] a parsing unit configured to parse a pre-configured rule file to determine a correspondence between the data to be converted and the fields to be extracted in the rule file; the rule file is a file including a first mapping relationship between the fields to be extracted and the data structure of the vector storage middleware, and a second mapping relationship between the fields to be extracted and the vectorized conversion algorithm;

[0040] The conversion storage unit is configured to call a vector conversion algorithm to perform vector conversion on the data to be converted according to the corresponding relationship and the second mapping relationship, and store the data in the vector storage middleware according to the first mapping relationship.

[0041] In a third aspect, an embodiment of the present disclosure provides a semi-structured data vector conversion system, comprising: a data extraction module, a rule configuration module, and a vector conversion algorithm tool library;

[0042] The data extraction module is configured to read the first semi-structured data file to obtain data to be converted;

[0043] The data extraction module is configured to parse a rule file pre-configured in the rule configuration module to determine a correspondence between the data to be converted and the fields to be extracted in the rule file; the rule file is a file including a first mapping relationship between the fields to be extracted and the data structure of the vector storage middleware, and a second mapping relationship between the fields to be extracted and the vectorized conversion algorithm;

[0044] The data extraction module is configured to call a vector conversion algorithm from the vector conversion algorithm tool library according to the corresponding relationship and the second mapping relationship to perform vector conversion on the data to be converted and store it in the vector storage middleware according to the first mapping relationship.

[0045] In a fourth aspect, an embodiment of the present disclosure provides an electronic device, comprising: a processor; and a memory configured to store executable instructions of the processor; wherein the processor is configured to execute the method described in the first aspect above by executing the executable instructions.

[0046] In a fifth aspect, an embodiment of the present disclosure provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the method described in the first aspect is implemented.

[0047] In a sixth aspect, according to another aspect of the present disclosure, a computer program product or computer program is further provided, the computer program product or computer program including computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform any of the methods described above.

[0048] The embodiments of the present disclosure provide a method, apparatus, system, device, and medium for converting semi-structured data into vectors. The method includes: reading a first semi-structured data file, obtaining the data to be converted, parsing a pre-configured rule file, and determining the correspondence between the data to be converted and the fields to be extracted in the rule file; the rule file is a file including a first mapping relationship between the fields to be extracted and the data structure of the vector storage middleware, and a second mapping relationship between the fields to be extracted and the vectorized conversion algorithm; the vector conversion algorithm is called, and the data to be converted is vectorized according to the corresponding relationship and the second mapping relationship, and then stored in the vector storage middleware according to the first mapping relationship. The rule file can improve the efficiency of vector conversion of semi-structured data, and can be stored in a corresponding vector database to achieve persistence of the vector database, which can be directly called when used, which is convenient and fast.

[0049] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] The accompanying drawings are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the specification, are used to explain the principles of the present disclosure. Obviously, the drawings described below are only some embodiments of the present disclosure, and those skilled in the art can derive other drawings based on these drawings without inventive effort.

[0051] FIG1 shows a schematic structural diagram of a semi-structured data vector conversion system according to an embodiment of the present disclosure;

[0052] FIG2 is a schematic diagram showing a method for converting semi-structured data vectors implemented by a semi-structured data vector conversion system according to an embodiment of the present disclosure;

[0053] FIG3 is a schematic diagram showing a process of a semi-structured data vector conversion method according to an embodiment of the present disclosure;

[0054] FIG4 is a schematic diagram showing a flow chart of a rule file configuration process in an embodiment of the present disclosure;

[0055] FIG5 is a schematic diagram showing a flow chart of another rule file configuration process in an embodiment of the present disclosure;

[0056] FIG6 shows a schematic structural diagram of a semi-structured data vector conversion device according to an embodiment of the present disclosure;

[0057] FIG7 shows a schematic structural diagram of an electronic device according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0058] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be embodied in many forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that this disclosure will be thorough and complete and will fully convey the concepts of the example embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

[0059] In addition, the accompanying drawings are merely schematic illustrations of the present disclosure and are not necessarily drawn to scale. Identical reference numerals in the figures denote identical or similar parts, and thus repetitive descriptions thereof will be omitted. Some of the block diagrams shown in the accompanying drawings are functional entities that do not necessarily correspond to physically or logically separate entities. These functional entities may be implemented in software, in one or more hardware modules or integrated circuits, or in different networks and / or processor devices and / or microcontroller devices.

[0060] Semi-structured data, due to its conciseness and clear hierarchical structure, makes it an ideal data exchange language. However, during the data exchange process, the data needs to be converted into other formats. The usual practice is to vectorize the relevant data. However, in related technologies, the processing efficiency of data vectorization is low, and vectors are inconvenient to use.

[0061] Based on this, an embodiment of the present disclosure provides a method for converting semi-structured data into vectors. The method includes: reading a first semi-structured data file, obtaining the data to be converted, parsing a pre-configured rule file, and determining the correspondence between the data to be converted and the fields to be extracted in the rule file; the rule file is a file including a first mapping relationship between the fields to be extracted and the data structure of the vector storage middleware, and a second mapping relationship between the fields to be extracted and the vectorized conversion algorithm; calling the vector conversion algorithm, performing vector conversion on the data to be converted based on the correspondence and the second mapping relationship, and storing the data in the vector storage middleware based on the first mapping relationship. The rule file can improve the efficiency of vector conversion of semi-structured data, and storage in a corresponding vector database can achieve persistence of the vector database, making it directly callable for quick and convenient use.

[0062] FIG1 shows a schematic diagram of a semi-structured data vector conversion system in an embodiment of the present disclosure.

[0063] As shown in FIG. 1 , the semi-structured data vector conversion system 100 may include a data extraction module 110 , a rule configuration module 120 and a vector conversion algorithm tool library 130 .

[0064] Among them, the data extraction module 110 is configured to read the first semi-structured data file to obtain the data to be converted; the data extraction module 110 is configured to parse the rule file pre-configured in the rule configuration module 120 to determine the correspondence between the data to be converted and the fields to be extracted in the rule file; the rule file is a file including a first mapping relationship between the fields to be extracted and the data structure of the vector storage middleware, and a second mapping relationship between the fields to be extracted and the vectorized conversion algorithm; the data extraction module 110 is configured to call the vector conversion algorithm in the vector conversion algorithm tool library 130, perform vector conversion on the data to be converted according to the corresponding relationship and the second mapping relationship, and store it in the vector storage middleware according to the first mapping relationship.

[0065] Furthermore, rule configuration module 120 can also be configured to provide a visual user interface for users to select fields to be extracted, and to configure a first mapping relationship between the fields to be extracted and the data structure of the vector storage middleware, as well as a second mapping relationship between the fields to be extracted and the vectorized conversion algorithm. Ultimately, the first mapping relationship between the fields to be extracted and the data structure of the vector storage middleware, and the second mapping relationship between the fields to be extracted and the vectorized conversion algorithm, are stored as a rule file in a specific format.

[0066] The vector conversion algorithm tool library 130 provides a commonly used vectorization tool set and encapsulates it into capabilities, including a variety of tools with vector conversion capabilities, which can be understood as various vector conversion algorithms. For example, Word to Vector (Word2Vec), Global Vectors for Word Representation (GloVe), Fast Text Representation (FastText), and Bidirectional Encoder Representations from Transformers (BERT). It is also possible to customize the implementation and add related methods, and users can make personalized choices based on their needs.

[0067] In one possible implementation, the present disclosure provides a process for implementing a semi-structured data vector conversion method in a semi-structured data vector conversion system. This example uses a JSON file as an example. FIG2 does not show the specific configuration process; only the connection relationship and simplified process are shown.

[0068] As shown in FIG2 , the semi-structured data vector conversion system 100 simplifies the second JSON data file to obtain a JSON template. The rule configuration module 120 can provide a visual interface, i.e., a data selection interface, based on the JSON template, so that the user can select the fields to be extracted from all fields in the data selection interface. The rule configuration module 120 records the field names and hierarchical positions of the fields to be extracted.

[0069] The rule configuration module 120 is also configured to read in the schema information of the vector storage middleware. The schema information is used to describe the data structure in the vector storage middleware. The vector storage middleware is configured to store vectors after semi-structured data conversion, and establish a first mapping relationship based on each field name and schema information.

[0070] The rule configuration module 120 is also configured to determine the hierarchical relationship based on the hierarchical position of each field to be extracted. If the hierarchical relationship is determined to be passed, a vector conversion algorithm is added to the field to be extracted that has passed the determination, and the first mapping relationship and the second mapping relationship between the field to be extracted that has passed the determination and the vectorized conversion algorithm are stored as a rule file.

[0071] Furthermore, the rule configuration module 120 specifically determines the hierarchical relationship in the following manner: determining the amount of data in each field to be extracted according to the hierarchical position of each field to be extracted; if the amount of data in each field to be extracted is consistent, the hierarchical relationship is determined to be successful.

[0072] In the above manner, the rule configuration module 120 outputs the rule file.

[0073] The data extraction module 110 reads the first JSON data file, obtains the data to be converted, parses the rule file pre-configured in the rule configuration module 120, determines the correspondence between the data to be converted and the fields to be extracted in the rule file, and based on the correspondence and the second mapping relationship, calls the vector conversion algorithm in the vector conversion algorithm tool library 130 to perform vector conversion on the data to be converted and stores it in the vector storage middleware according to the first mapping relationship.

[0074] This exemplary implementation is described in detail below with reference to the accompanying drawings and examples.

[0075] First, a semi-structured data vector conversion method is provided in an embodiment of the present disclosure. The method can be executed by any electronic device with computing and processing capabilities. In the following process, the electronic device is taken as a semi-structured data vector conversion system 100 as an example.

[0076] FIG3 is a flow chart showing a method for converting a semi-structured data vector in an embodiment of the present disclosure. As shown in FIG3 , the method provided in an embodiment of the present disclosure includes the following steps:

[0077] S302: Read the first semi-structured data file to obtain data to be converted.

[0078] In a possible embodiment, the semi-structured data may be data in JSON format, ie, JavaScript object representation, and the semi-structured data file may be a JSON file. The semi-structured data may also include data in Extensible Markup Language (XML) format.

[0079] Parse the first semi-structured data file to obtain the data to be converted and prepare for vector conversion.

[0080] S304: Parse the pre-configured rule file to determine the correspondence between the data to be converted and the fields to be extracted in the rule file. The rule file is a file including a first mapping relationship between the fields to be extracted and the data structure of the vector storage middleware, and a second mapping relationship between the fields to be extracted and the vectorized conversion algorithm.

[0081] In one possible embodiment, by parsing the configured rule file, the data to be converted can be quickly extracted according to the generation extraction field, and the data to be converted can be vectorized according to the mapping relationship in the rule file, and the vector conversion can be quickly completed without the need for hard coding. It is universal for all semi-structured format files, which can improve the efficiency of data exchange, thereby effectively shortening the R&D cycle and improving the flexibility of data extraction and conversion.

[0082] The first mapping relationship maps the fields to be extracted to the data structure of the vector storage middleware. The vector storage middleware includes scalars and vectors. Scalars can be understood as fields corresponding to semi-structured data, and vectors can be understood as vector data after field-vector conversion. The first mapping relationship can be used to represent the storage location of data corresponding to a series of fields to be extracted after vector conversion.

[0083] Among them, the vector storage middleware is a database for storing vectors, which can exist in the form of a two-dimensional table.

[0084] S306: According to the corresponding relationship and the second mapping relationship, a vector conversion algorithm is called to perform vector conversion on the data to be converted and the data is stored in the vector storage middleware according to the first mapping relationship.

[0085] In a possible embodiment, according to the second mapping relationship and the corresponding relationship, the vector conversion algorithm of the data to be converted is called from the vector conversion algorithm tool library, and the data to be converted is vector converted by the vector conversion algorithm to obtain the vector to be used, and the vector to be used is stored in the vector storage middleware according to the first mapping relationship.

[0086] In one possible embodiment, after storing the vectors to be used in the vector storage middleware, the vectors corresponding to the semi-structured data can be retrieved from the vector storage middleware in response to a user's service provision request to provide services to external applications. The middleware knows where the vectors are stored and can be directly queried within the vector storage middleware.

[0087] Through the above-mentioned method in the present disclosure, the three modules of data extraction module, rule configuration module and vector conversion algorithm tool library cooperate to improve the efficiency of data extraction and circulation, improve the efficiency of vector conversion, realize the persistence of vector database, and provide services to external applications by vector storage middleware, which can be directly called when used, which is convenient and fast.

[0088] In a possible implementation, FIG4 shows a schematic flow chart of a rule file configuration method, as shown in FIG4 , specifically including the following steps:

[0089] S402: Determine the hierarchical position and field name of the field to be extracted according to the second semi-structured data file.

[0090] In a possible embodiment, the second semi-structured data file may be directly obtained to determine the hierarchical position and field name of the field to be extracted.

[0091] However, some semi-structured data files contain a large amount of data, including a lot of specific data content, resulting in a large file size. When configuring the rule file, reading files with such a large amount of data may encounter performance issues. Therefore, the hierarchical position and field name of the field to be extracted can be determined in the following two ways.

[0092] In another possible embodiment, the data information in the second semi-structured data file is simplified to obtain a basic framework structure corresponding to the second semi-structured data file, and the basic framework structure is saved as a semi-structured data template. According to the semi-structured data template, the hierarchical position and field name of the field to be extracted are determined.

[0093] In another possible embodiment, the semi-structured data template of the second semi-structured data file may be directly obtained, and the hierarchical position and field name of the field to be extracted may be determined from the semi-structured data template.

[0094] The basic framework structure of the second semi-structured data file is basically the same as the basic framework structure of the first semi-structured data file. The data included in the first semi-structured data file and the second semi-structured data file need to be semi-structured data of the same format, for example, both can be data in JSON format.

[0095] Furthermore, whether the hierarchical position and field name of the field to be extracted are determined through the above-mentioned second semi-structured data file, or the hierarchical position and field name of the field to be extracted are determined according to the semi-structured data template, it can be determined through the following visual interface. The specific method may include: displaying all fields in the data selection interface, and in response to the user checking the field to be extracted in the data selection interface, recording the hierarchical position and field name of multiple fields to be extracted.

[0096] Through the visual interface provided by the rule configuration module in the present disclosure, ie, the data selection interface, the user can independently select which fields to be included in the rule file as fields to be extracted during the process of configuring the rule file.

[0097] S404: Establish a rule file including a first mapping relationship between the field to be extracted and the data structure of the vector storage middleware, and a second mapping relationship between the field to be extracted and the vectorized conversion algorithm.

[0098] In a possible embodiment, a specific method of establishing a rule file may include the following steps: establishing a first mapping relationship based on each field name and vector storage middleware, judging the hierarchical relationship based on the hierarchical position of each field to be extracted, and if the hierarchical relationship is judged to be passed, adding a vector conversion algorithm to the field to be extracted that has passed the judgment, and storing the first mapping relationship and the second mapping relationship between the field to be extracted that has passed the judgment and the vectorization conversion algorithm as a rule file.

[0099] In another possible embodiment, the step of establishing a first mapping relationship based on each field name and the vector storage middleware may include: reading in schema information of the vector storage middleware; the schema information is used to describe the data structure in the vector storage middleware; the vector storage middleware is configured to store vectors after semi-structured data conversion, and establishing a first mapping relationship based on each field name and the schema information.

[0100] In another possible embodiment, since semi-structured data files have a multi-layered nested structure, unlike a two-dimensional table schema, a feasibility check is required before conversion. The selected fields to be extracted are constrained. Specifically, the constraints can be based on hierarchical position. The hierarchical relationship of the selected fields to be extracted is determined. If the determination passes, a vector conversion algorithm is added, and the first mapping relationship and the passed fields to be extracted are stored as a rule file with the second mapping relationship of the vectorized conversion algorithm. When stored as a rule file, the format of the rule file must comply with the XML specification and record the relevant configuration content.

[0101] The specific method of judging the hierarchical relationship according to the hierarchical position can be taken as an example: according to the hierarchical position of each field to be extracted, the amount of data of each field to be extracted is determined. If the amount of data of each field to be extracted is consistent, the hierarchical relationship is judged to be successful.

[0102] For example, a short code snippet is used as an example to illustrate the process. The specific code is as follows:

[0103] If the selected fields to be extracted are "name", "age", and "address", after recording their field names and hierarchical positions, a first mapping relationship is established, and their hierarchical relationship is determined. The number of data in the above three fields to be extracted is 1, so their hierarchical relationship is determined to be successful.

[0104] If the selected fields to be extracted are "name," "age," "type," and "value," the number of data points for "type" and "value" is 2, which is different from the number of data points for "name" and "age." After being converted into vectors, these fields cannot be stored in the vector storage middleware. Even if a rule file is created, an error will occur when storing them in the vector storage middleware after conversion. Therefore, the hierarchical relationship determination fails. If the hierarchical relationship determination fails, the user is returned to the data selection interface, prompted that the hierarchical relationship of the selected fields to be extracted fails, and is asked to reselect the fields to be extracted.

[0105] In a possible implementation, the present disclosure provides another flowchart of a rule file configuration process, as shown in FIG5 , including the following steps:

[0106] S502: Simplify the data information in the second semi-structured data file to obtain a basic framework structure corresponding to the second semi-structured data file.

[0107] S504: Save the basic framework structure as a semi-structured data template.

[0108] S506: Display all fields in the data selection interface according to the semi-structured data template.

[0109] S508: In response to the user selecting a field to be extracted in the data selection interface, the hierarchical positions and field names of the multiple fields to be extracted are recorded.

[0110] S510: Reading in schema information of the vector storage middleware, where the schema information is used to describe the data structure in the vector storage middleware. The vector storage middleware is configured to store vectors after semi-structured data conversion.

[0111] S512: Establish a first mapping relationship according to each field name and mode information.

[0112] S514: Based on the hierarchical position of each field to be extracted, determine whether the amount of data in each field to be extracted is consistent; if consistent, execute S516; if inconsistent, prompt an error and return to step S506.

[0113] S516: Add a vector conversion algorithm for the fields to be extracted that have passed the determination.

[0114] S518: The first mapping relationship and the passed field to be extracted and the second mapping relationship of the vectorized conversion algorithm are stored as a rule file.

[0115] Through the above-mentioned method provided by the embodiment of the present disclosure, a rule file is pre-configured, which includes a first mapping relationship between the field to be extracted and the data structure of the vector storage middleware, and a second mapping relationship between the field to be extracted and the vectorization conversion algorithm. This can not only directly determine the algorithm for vectorizing the data to be converted, but also determine the location where the data is stored after the conversion. This not only makes data extraction faster and eliminates the need for hard coding, thereby improving the efficiency of vector conversion, but is also universal for all semi-structured data files. Vectors can also be stored in a database to achieve vector database persistence and provide services to external applications, effectively shortening the R&D cycle, improving the flexibility of data extraction, and allowing direct calling when used, which is convenient and fast.

[0116] Based on the same inventive concept, the present disclosure also provides a device for converting semi-structured data vectors, as shown in the following embodiment. Since the principles of the device embodiment are similar to those of the above-mentioned method embodiment, the implementation of the device embodiment can refer to the implementation of the above-mentioned method embodiment, and the repeated parts will not be repeated here.

[0117] Figure 6 shows a structural schematic diagram of a semi-structured data vector conversion device in an embodiment of the present disclosure. As shown in Figure 6, the semi-structured data vector conversion device 60 includes: an acquisition unit 601, configured to read a first semi-structured data file to obtain data to be converted; a parsing unit 602, configured to parse a pre-configured rule file to determine the correspondence between the data to be converted and the fields to be extracted in the rule file, where the rule file is a file including a first mapping relationship between the fields to be extracted and the data structure of the vector storage middleware, and a second mapping relationship between the fields to be extracted and the vectorized conversion algorithm; a conversion storage unit 603, configured to call the vector conversion algorithm to perform vector conversion on the data to be converted based on the corresponding relationship and the second mapping relationship and store the data in the vector storage middleware according to the first mapping relationship.

[0118] Those skilled in the art will appreciate that various aspects of the present disclosure may be implemented as systems, methods, or program products. Therefore, various aspects of the present disclosure may be implemented in the following forms: a complete hardware implementation, a complete software implementation (including firmware, microcode, etc.), or a combination of hardware and software implementations, which may be collectively referred to herein as "circuits," "modules," or "systems."

[0119] The electronic device 700 according to this embodiment of the present disclosure is described below with reference to Figure 7. The electronic device 700 shown in Figure 7 is merely an example and should not limit the functions and scope of use of the embodiments of the present disclosure.

[0120] As shown in Figure 7, electronic device 700 is implemented as a general-purpose computing device. Components of electronic device 700 may include, but are not limited to, the aforementioned at least one processing unit 710, the aforementioned at least one storage unit 720, and a bus 730 connecting various system components (including storage unit 720 and processing unit 710).

[0121] The storage unit stores program code, which can be executed by the processing unit 710, so that the processing unit 710 performs the steps of various exemplary embodiments of the present disclosure described in the "Exemplary Methods" section above. For example, the processing unit 710 can perform the steps of any of the above method embodiments.

[0122] The storage unit 720 may include a readable medium in the form of a volatile storage unit, such as a random access memory unit (RAM) 7201 and / or a cache memory unit 7202 , and may further include a read-only memory unit (ROM) 7203 .

[0123] The storage unit 720 may also include a program / utility 7204 having a set (at least one) of program modules 7205, such program modules 7205 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.

[0124] Bus 730 may represent one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.

[0125] The electronic device 700 can also communicate with one or more external devices 740 (e.g., a keyboard, a pointing device, a Bluetooth device, etc.), one or more devices that enable a user to interact with the electronic device 700, and / or any device that enables the electronic device 700 to communicate with one or more other computing devices (e.g., a router, a modem, etc.). Such communication can occur via an input / output (I / O) interface 750. Furthermore, the electronic device 700 can also communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network such as the Internet) via a network adapter 760. As shown, the network adapter 760 communicates with other modules of the electronic device 700 via a bus 730. It should be understood that, although not shown in the figure, other hardware and / or software modules can be used in conjunction with the electronic device 700, including but not limited to microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0126] Through the description of the above embodiments, it is easy for those skilled in the art to understand that the example embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solution according to the embodiments of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, and includes several instructions to enable a computing device (which can be a personal computer, a server, a terminal device, or a network device, etc.) to execute the method according to the embodiments of the present disclosure.

[0127] In particular, according to embodiments of the present disclosure, the processes described above with reference to the flowcharts may be implemented as a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the methods of the above embodiments.

[0128] In an exemplary embodiment of the present disclosure, a computer-readable storage medium is also provided. The computer-readable storage medium may be a readable signal medium or a readable storage medium. A program product capable of implementing the above-mentioned method of the present disclosure is stored thereon. In some possible implementations, various aspects of the present disclosure may also be implemented in the form of a program product, which includes program code. When the program product is run on a terminal device, the program code is used to cause the terminal device to execute the steps according to various exemplary embodiments of the present disclosure described in the "Exemplary Methods" section above of this specification.

[0129] More specific examples of computer-readable storage media in the present disclosure may include, but are not limited to, an electrical connection having one or more conductors, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), optical fibers, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0130] In the present disclosure, a computer-readable storage medium may include a data signal propagated in baseband or as part of a carrier wave, which carries readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A readable signal medium may also be any readable medium other than a readable storage medium that can transmit, propagate, or transfer a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0131] Alternatively, the program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination thereof.

[0132] In a specific implementation, the program code for performing the operations of the present disclosure may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, and the like, as well as conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a standalone software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In the case of a remote computing device, the remote computing device may be connected to the user computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0133] It should be noted that although several modules or units of the device for action execution are mentioned in the detailed description above, this division is not mandatory. In fact, according to the embodiments of the present disclosure, the features and functions of two or more modules or units described above can be concretized in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided into multiple modules or units to be concretized.

[0134] Furthermore, although the steps of the method of the present disclosure are described in a particular order in the accompanying drawings, this does not require or imply that the steps must be performed in this particular order, or that all steps shown must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps.

[0135] Through the description of the above embodiments, it is easy for those skilled in the art to understand that the example embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solution according to the embodiments of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, and includes several instructions to enable a computing device (which can be a personal computer, a server, a mobile terminal, or a network device, etc.) to execute the method according to the embodiments of the present disclosure.

[0136] Other embodiments of the present disclosure will readily occur to those skilled in the art after considering the specification and practicing the invention disclosed herein. This disclosure is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art not disclosed herein. The description and examples are to be considered as exemplary only, with the true scope and spirit of the present disclosure being indicated by the appended claims.

Claims

1. A method for converting semi-structured data vectors, wherein: The method comprises: Reading a first semi-structured data file to obtain data to be converted; Parsing a pre-configured rule file to determine the correspondence between the data to be converted and the fields to be extracted in the rule file; the rule file is a file including a first mapping relationship between the fields to be extracted and the data structure of the vector storage middleware, and a second mapping relationship between the fields to be extracted and the vectorized conversion algorithm; According to the corresponding relationship and the second mapping relationship, a vector conversion algorithm is called to perform vector conversion on the data to be converted and the data is stored in the vector storage middleware according to the first mapping relationship.

2. The method according to claim 1, wherein: The configuration method of the rule file includes: Determine the hierarchical position and field name of the field to be extracted according to the second semi-structured data file; A rule file is established including a first mapping relationship between the field to be extracted and the data structure of the vector storage middleware, and a second mapping relationship between the field to be extracted and the vectorization conversion algorithm.

3. The method according to claim 2, wherein: The step of determining the hierarchical position and field name of the field to be extracted according to the second semi-structured data file includes: Simplifying the data information in the second semi-structured data file to obtain a basic framework structure corresponding to the second semi-structured data file; Saving the basic framework structure as a semi-structured data template; According to the semi-structured data template, the hierarchical position and field name of the field to be extracted are determined.

4. The method according to any one of claims 2 or 3, wherein: Determining the hierarchical position and field name of the field to be extracted includes: Display all fields in the data selection interface; In response to the user checking the fields to be extracted in the data selection interface, the hierarchical positions and field names of the multiple fields to be extracted are recorded.

5. The method according to claim 2, wherein: The step of establishing a rule file including a first mapping relationship between the field to be extracted and the data structure of the vector storage middleware, and a second mapping relationship between the field to be extracted and the vectorized conversion algorithm includes: Establishing a first mapping relationship according to each of the field names and the vector storage middleware; Determine the hierarchical relationship according to the hierarchical position of each of the fields to be extracted; If the hierarchical relationship is judged to be passed, a vector conversion algorithm is added to the fields to be extracted that have passed the judgment; The first mapping relationship and the second mapping relationship between the field to be extracted that has passed the determination and the vectorized conversion algorithm are stored as a rule file.

6. The method according to claim 5, wherein: The step of establishing a first mapping relationship according to each of the field names and the vector storage middleware includes: Reading in mode information of the vector storage middleware; the mode information is used to describe the data structure in the vector storage middleware; the vector storage middleware is used to store vectors after semi-structured data conversion; A first mapping relationship is established according to each of the field names and the mode information.

7. The method according to claim 5, wherein: The step of determining the hierarchical relationship according to the hierarchical position of each of the fields to be extracted includes: Determining the amount of data in each of the fields to be extracted according to the hierarchical position of each of the fields to be extracted; If the amount of data in each of the fields to be extracted is consistent, the hierarchical relationship is determined to be successful.

8. The method according to claim 1, wherein: After calling a vector conversion algorithm to perform vector conversion on the data to be converted according to the corresponding relationship and the second mapping relationship and storing the data in the vector storage middleware according to the first mapping relationship, the method further includes: In response to a service provision request from a user, vectors corresponding to the semi-structured data are extracted from the vector storage middleware to provide services to external applications.

9. A semi-structured data vector conversion device, wherein: include: An acquisition unit, configured to read a first semi-structured data file to acquire data to be converted; A parsing unit, configured to parse a pre-configured rule file to determine a correspondence between the data to be converted and the fields to be extracted in the rule file; the rule file is a file including a first mapping relationship between the fields to be extracted and the data structure of the vector storage middleware, and a second mapping relationship between the fields to be extracted and the vectorized conversion algorithm; The conversion storage unit is configured to call a vector conversion algorithm to perform vector conversion on the data to be converted according to the corresponding relationship and the second mapping relationship, and store the data in the vector storage middleware according to the first mapping relationship.

10. A semi-structured data vector conversion system, wherein: include: Data extraction module, rule configuration module and vector conversion algorithm tool library; The data extraction module is configured to read the first semi-structured data file to obtain data to be converted; The data extraction module is configured to parse the rule file pre-configured in the rule configuration module to determine the correspondence between the data to be converted and the fields to be extracted in the rule file; the rule file is a file including a first mapping relationship between the fields to be extracted and the data structure of the vector storage middleware, and a second mapping relationship between the fields to be extracted and the vectorized conversion algorithm; The data extraction module is configured to call a vector conversion algorithm in the vector conversion algorithm tool library according to the corresponding relationship and the second mapping relationship to perform vector conversion on the data to be converted and store it in the vector storage middleware according to the first mapping relationship.

11. An electronic device, wherein: include: processor; as well as a memory configured to store executable instructions of the processor; The processor is configured to perform the method of any one of claims 1 to 8 by executing the executable instructions.

12. A computer-readable storage medium having a computer program stored thereon, wherein: When the computer program is executed by a processor, the method according to any one of claims 1 to 8 is implemented.

13. A computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the method according to any one of claims 1 to 8 is implemented.

Images