Semi-structured data vector conversion method, apparatus, system, device, and medium

Abstract

The present disclosure provides a semi-structured data vector conversion method, device, system, equipment and medium, relating to the technical field of big data processing, which comprises the following steps: reading a first semi-structured data file, obtaining data to be converted, parsing a pre-configured rule file, and determining the correspondence between the data to be converted and the fields to be extracted in the rule file; the rule file is a file comprising a first mapping relationship between the data structure of the fields to be extracted and the vector storage middleware, and a second mapping relationship between the fields to be extracted and the vectorization conversion algorithm; according to the correspondence and the second mapping relationship, the vector conversion algorithm is called to perform vector conversion on the data to be converted and stored in the vector storage middleware according to the first mapping relationship. Through the rule file, the efficiency of the vector conversion of the semi-structured data can be improved, and the semi-structured data can be stored in the corresponding vector database, realizing the persistence of the vector database and being directly called during use, which is convenient and fast.

Description

Technical Field

[0001] This disclosure relates to the field of big data processing technology, and in particular to a method, apparatus, system, device and medium for converting semi-structured data vectors. Background Technology

[0002] With the development of science and technology and the evolution of IT architecture, semi-structured data has become increasingly important and is widely used in many application systems, interface calls, data storage and other fields. The concise and clear hierarchical structure has become an ideal data exchange language. When data needs to be processed and analyzed or finally stored on disk, it usually needs to be converted into other formats. Due to the emergence of artificial intelligence technology and various applications, the common practice is to vectorize the relevant data.

[0003] However, in related technologies, the processing efficiency of data vectorization is low, and vectors are inconvenient to use.

[0004] It should be noted that the information disclosed in the background section above is only used to enhance the understanding of the background of this disclosure, and therefore may include information that does not constitute prior art known to those skilled in the art. Summary of the Invention

[0005] This disclosure provides a method, apparatus, system, device, and medium for converting semi-structured data into vectors, which at least to some extent overcomes the technical problems of low processing efficiency and inconvenient use of vectors in related technologies.

[0006] Other features and advantages of this disclosure will become apparent from the following detailed description, or may be learned in part from practice of this disclosure.

[0007] In a first aspect, embodiments of this disclosure provide a semi-structured data vector conversion method, the method comprising:

[0008] Read the first half of the structured data file to obtain the data to be converted;

[0009] The pre-configured rule file is parsed to determine the correspondence between the data to be converted and the fields to be extracted in the rule file; the rule file is a file that includes a first mapping relationship between the fields to be extracted and the data structure of the vector storage middleware, and a second mapping relationship between the fields to be extracted and the vectorization conversion algorithm.

[0010] The vector conversion algorithm is invoked to perform vector conversion on the data to be converted.

[0011] In one possible embodiment, the rule file is configured in the following ways:

[0012] Based on the second half of the structured data file, determine the hierarchical position and field name of the fields to be extracted;

[0013] A rule file is established that includes a first mapping relationship between the field to be extracted and the data structure of the vector storage middleware, and a second mapping relationship between the field to be extracted and the vectorization conversion algorithm.

[0014] In one possible embodiment, determining the hierarchical position and field name of the field to be extracted based on the second semi-structured data file includes:

[0015] Simplify the data information in the second half-structured data file to obtain the basic framework structure corresponding to the second half-structured data file;

[0016] Save the aforementioned basic framework structure as a semi-structured data template;

[0017] Based on the semi-structured data template, determine the hierarchical position and field name of the field to be extracted.

[0018] In one possible embodiment, determining the hierarchical position and field name of the field to be extracted includes:

[0019] Display all fields in the data selection interface;

[0020] In response to the user selecting the fields to be extracted on the data selection interface, the hierarchical position and field name of the multiple fields to be extracted are recorded.

[0021] In one possible embodiment, the rule file for establishing a first mapping relationship between the field to be extracted and the data structure of the vector storage middleware, and a second mapping relationship between the field to be extracted and the vectorization transformation algorithm, includes:

[0022] A first mapping relationship is established based on each field name and the vector storage middleware;

[0023] Determine the hierarchical relationship based on the hierarchical position of each field to be extracted;

[0024] If the hierarchical relationship is determined to be valid, a vector conversion algorithm is added to the fields to be extracted that have passed the determination.

[0025] The first mapping relationship and the second mapping relationship between the fields to be extracted that have passed the judgment and the vectorization conversion algorithm are stored as a rule file.

[0026] In one possible embodiment, establishing the first mapping relationship based on each of the field names and the vector storage middleware includes:

[0027] The schema information of the vector storage middleware is read in; the schema information is used to describe the data structure in the vector storage middleware; the vector storage middleware is used to store vectors after the semi-structured data has been converted.

[0028] A first mapping relationship is established based on each field name and the pattern information.

[0029] In one possible embodiment, determining the hierarchical relationship based on the hierarchical position of each of the fields to be extracted includes:

[0030] The amount of data for each field to be extracted is determined based on its hierarchical position.

[0031] If the amount of data in each of the fields to be extracted is the same, then the hierarchical relationship determination is successful.

[0032] In one possible embodiment, after invoking a vector conversion algorithm to perform vector conversion on the data to be converted according to the correspondence and the second mapping relationship, and storing it in a vector storage middleware according to the first mapping relationship, the method further includes:

[0033] In response to a user's service provision request, the vector corresponding to the semi-structured data is extracted from the vector storage middleware to provide services to external applications.

[0034] Secondly, embodiments of this disclosure provide a semi-structured data vector conversion apparatus, comprising:

[0035] The acquisition unit is used to read the first semi-structured data file and obtain the data to be converted.

[0036] The parsing unit is used to parse the pre-configured rule file and determine the correspondence between the data to be converted and the fields to be extracted in the rule file; the rule file is a file that includes a first mapping relationship between the fields to be extracted and the data structure of the vector storage middleware, and a second mapping relationship between the fields to be extracted and the vectorization conversion algorithm.

[0037] The conversion storage unit is used to call a vector conversion algorithm to perform vector conversion on the data to be converted according to the correspondence and the second mapping relationship, and store it in the vector storage middleware according to the first mapping relationship.

[0038] Thirdly, embodiments of this disclosure provide a semi-structured data vector conversion system, including: a data extraction module, a rule configuration module, and a vector conversion algorithm tool library;

[0039] The data extraction module is used to read the first semi-structured data file and obtain the data to be converted.

[0040] The data extraction module is used to parse the pre-configured rule file in the rule configuration module and determine the correspondence between the data to be converted and the fields to be extracted in the rule file; the rule file is a file that includes a first mapping relationship between the fields to be extracted and the data structure of the vector storage middleware, and a second mapping relationship between the fields to be extracted and the vectorization conversion algorithm.

[0041] The data extraction module is used to call a vector conversion algorithm from the vector conversion algorithm tool library to perform vector conversion on the data to be converted according to the correspondence and the second mapping relationship, and store it in the vector storage middleware according to the first mapping relationship.

[0042] Fourthly, embodiments of this disclosure provide an electronic device, including: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to perform the method described in the first aspect above by executing the executable instructions.

[0043] Fifthly, embodiments of this disclosure provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method described in the first aspect above.

[0044] Sixthly, according to another aspect of this disclosure, a computer program product or computer program is also provided, the computer program product or computer program including computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, causing the computer device to perform the methods described in any of the preceding claims.

[0045] This disclosure provides a method, apparatus, system, device, and medium for vector conversion of semi-structured data. The method includes: reading a first semi-structured data file, obtaining data to be converted, parsing a pre-configured rule file, and determining the correspondence between the data to be converted and the fields to be extracted in the rule file. The rule file is a file including a first mapping relationship between the fields to be extracted and the data structure of a vector storage middleware, and a second mapping relationship between the fields to be extracted and a vectorization conversion algorithm. The method involves calling a vector conversion algorithm, performing vector conversion on the data to be converted according to the correspondence and the second mapping relationship, and storing the data in the vector storage middleware according to the first mapping relationship. The rule file improves the efficiency of vector conversion of semi-structured data and enables persistent storage in a corresponding vector database, allowing for direct and convenient access during use.

[0046] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0047] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure. It is obvious that the drawings described below are merely some embodiments of this disclosure, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort.

[0048] Figure 1 This diagram illustrates the structure of a semi-structured data vector conversion system according to an embodiment of the present disclosure.

[0049] Figure 2 This diagram illustrates a semi-structured data vector conversion system implementing a semi-structured data vector conversion method according to an embodiment of the present disclosure.

[0050] Figure 3 A flowchart illustrating a semi-structured data vector conversion method according to an embodiment of this disclosure is shown.

[0051] Figure 4 A flowchart illustrating a rule file configuration process in an embodiment of this disclosure is shown.

[0052] Figure 5 This illustration shows a flowchart of the rule file configuration process in an embodiment of this disclosure;

[0053] Figure 6 This diagram illustrates the structure of a semi-structured data vector conversion device according to an embodiment of the present disclosure.

[0054] Figure 7 A schematic diagram of the structure of an electronic device according to an embodiment of the present disclosure is shown. Detailed Implementation

[0055] Exemplary embodiments will now be described more fully with reference to the accompanying drawings. However, these exemplary embodiments can be implemented in many forms and should not be construed as limited to the examples set forth herein; rather, they are provided so that this disclosure will be more comprehensive and complete, and will fully convey the concept of the exemplary embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

[0056] Furthermore, the accompanying drawings are merely illustrative of this disclosure and are not necessarily drawn to scale. The same reference numerals in the drawings denote the same or similar parts, and therefore repeated descriptions of them will be omitted. Some block diagrams shown in the drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities may be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0057] Semi-structured data, with its concise and clear hierarchical structure, is an ideal language for data exchange. However, during the data exchange process, it is necessary to convert the data into other formats. The usual approach is to vectorize the relevant data. However, in related technologies, the processing efficiency of data vectorization is low, and vectors are inconvenient to use.

[0058] Based on this, embodiments of this disclosure provide a method for vector conversion of semi-structured data. The method includes: reading a first semi-structured data file, obtaining data to be converted, parsing a pre-configured rule file, and determining the correspondence between the data to be converted and the fields to be extracted in the rule file. The rule file is a file containing a first mapping relationship between the fields to be extracted and the data structure of a vector storage middleware, and a second mapping relationship between the fields to be extracted and a vectorization conversion algorithm. The method involves calling a vector conversion algorithm, performing vector conversion on the data to be converted according to the correspondence and the second mapping relationship, and storing the data in the vector storage middleware according to the first mapping relationship. The rule file improves the efficiency of vector conversion of semi-structured data and enables persistence of the vector database, allowing for direct and convenient access during use.

[0059] Figure 1 A schematic diagram of a semi-structured data vector conversion system according to an embodiment of this disclosure is shown.

[0060] like Figure 1 As shown, the semi-structured data vector conversion system 100 may include a data extraction module 110, a rule configuration module 120, and a vector conversion algorithm tool library 130.

[0061] Specifically, the data extraction module 110 reads the first semi-structured data file to obtain the data to be converted; the data extraction module 110 parses the pre-configured rule file in the rule configuration module 120 to determine the correspondence between the data to be converted and the fields to be extracted in the rule file; the rule file is a file that includes a first mapping relationship between the data structure of the fields to be extracted and the vector storage middleware, and a second mapping relationship between the fields to be extracted and the vectorization conversion algorithm; the data extraction module 110 calls the vector conversion algorithm in the vector conversion algorithm tool library 130, performs vector conversion on the data to be converted according to the correspondence and the second mapping relationship, and stores it in the vector storage middleware according to the first mapping relationship.

[0062] Furthermore, the rule configuration module 120 can also provide a visual user interface for users to select fields to be extracted, and provide the function of configuring the first mapping relationship between the fields to be extracted and the data structure of the vector storage middleware, as well as the second mapping relationship between the fields to be extracted and the vectorization conversion algorithm. Finally, the first mapping relationship between the fields to be extracted and the data structure of the vector storage middleware, and the second mapping relationship between the fields to be extracted and the vectorization conversion algorithm are stored as a rule file in a specific format.

[0063] The Vector Transformation Algorithm Toolkit 130 provides a collection of commonly used vectorization tools, encapsulated as capabilities. This includes various tools with vector transformation capabilities, which can be understood as different vector transformation algorithms. Examples include Word to Vector (Word2Vec), Global Vectors for Word Representation (GloVe), FastText, and Bidirectional Encoder Representations from Transformers (BERT). Users can also customize implementations and add related methods, allowing for personalized selection based on their needs.

[0064] In one possible implementation, this disclosure provides a process for implementing a semi-structured data vector conversion method in a semi-structured data vector conversion system. Taking a JSON file as an example, the semi-structured data file is used. Figure 2 The detailed configuration process is not displayed; only the connection relationships and a simplified process are shown.

[0065] like Figure 2As shown, the semi-structured data vector conversion system 100 simplifies the second JSON data file to obtain a JSON template. The rule configuration module 120 can provide a visual interface, namely a data selection interface, based on the JSON template, so that users can select the fields to be extracted from all fields in the data selection interface. The rule configuration module 120 records the field name and hierarchical position of the fields to be extracted.

[0066] The rule configuration module 120 is also used to read in the schema information of the vector storage middleware. The schema information is used to describe the data structure in the vector storage middleware. The vector storage middleware is used to store vectors after the semi-structured data has been converted. Based on each field name and schema information, the first mapping relationship is established.

[0067] The rule configuration module 120 is also used to determine the hierarchical relationship based on the hierarchical position of each field to be extracted. If the hierarchical relationship is determined to be valid, a vector conversion algorithm is added to the field to be extracted that has been determined to be valid. The first mapping relationship and the second mapping relationship between the field to be extracted that has been determined to be valid and the vectorization conversion algorithm are stored as a rule file.

[0068] Furthermore, the specific method by which the rule configuration module 120 determines the hierarchical relationship includes: determining the number of data for each field to be extracted based on its hierarchical position; if the number of data for each field to be extracted is consistent, the hierarchical relationship determination is successful.

[0069] Using the above method, the rule configuration module 120 outputs the rule file.

[0070] The data extraction module 110 reads the first JSON data file, obtains the data to be converted, parses the pre-configured rule file in the rule configuration module 120, determines the correspondence between the data to be converted and the fields to be extracted in the rule file, and calls the vector conversion algorithm in the vector conversion algorithm tool library 130 to perform vector conversion on the data to be converted according to the first mapping relationship and stores it in the vector storage middleware.

[0071] The following detailed description of this exemplary implementation method is provided in conjunction with the accompanying drawings and embodiments.

[0072] First, this disclosure provides a semi-structured data vector conversion method, which can be executed by any electronic device with computing capabilities. In the following process, the electronic device is taken as a semi-structured data vector conversion system 100.

[0073] Figure 3 This diagram illustrates a flowchart of a semi-structured data vector conversion method according to an embodiment of the present disclosure, as shown below. Figure 3 As shown, the method provided in this embodiment includes the following steps:

[0074] S302: Read the first semi-structured data file and obtain the data to be converted.

[0075] In one possible embodiment, the semi-structured data can be in JSON format, i.e., represented by JavaScript objects, and the semi-structured data file can be a JSON file. The semi-structured data can also include data in Extensible Markup Language (XML) format.

[0076] The first half of the structured data file is parsed to obtain the data to be converted, and then vector conversion is performed.

[0077] S304: Parse the pre-configured rule file to determine the correspondence between the data to be converted and the fields to be extracted in the rule file. The rule file is a file that includes the first mapping relationship between the fields to be extracted and the data structure of the vector storage middleware, and the second mapping relationship between the fields to be extracted and the vectorization conversion algorithm.

[0078] In one possible embodiment, parsing the configured rule file allows for the rapid extraction of data to be transformed based on the fields to be extracted. Furthermore, the data to be transformed can be processed into vectors according to the mapping relationships in the rule file, enabling rapid vector transformation without the need for hard coding. This method is applicable to all semi-structured file formats, thereby improving the efficiency of data exchange, effectively shortening the development cycle, and enhancing the flexibility of data extraction and transformation.

[0079] The first mapping relationship is the mapping between the fields to be extracted and the data structure of the vector storage middleware. The vector storage middleware includes scalars and vectors; scalars can be understood as fields corresponding to semi-structured data, and vectors can be understood as vector data after field vector conversion. The first mapping relationship can be used to characterize the storage location of a series of data corresponding to the fields to be extracted after vector conversion, etc.

[0080] Among them, the vector storage middleware is a database that stores vectors, which can exist in the form of a two-dimensional table.

[0081] S306: Based on the correspondence and the second mapping relationship, call the vector conversion algorithm to perform vector conversion on the data to be converted and store it in the vector storage middleware according to the first mapping relationship.

[0082] In one possible embodiment, based on the second mapping relationship and the correspondence relationship, the vector conversion algorithm of the data to be converted is called from the vector conversion algorithm tool library, and the vector conversion algorithm is used to perform vector conversion on the data to be converted to obtain the vector to be used. The vector to be used is then stored in the vector storage middleware according to the first mapping relationship.

[0083] In one possible implementation, after storing the vector to be used in a vector storage middleware, when needed, in response to a user's service request, the vector corresponding to the semi-structured data can be extracted from the vector storage middleware to provide services to external applications. The exact location of the vector within the middleware is known; it can be directly queried within the middleware.

[0084] The method described in this disclosure improves the efficiency of data extraction and transfer by having three modules—data extraction module, rule configuration module, and vector transformation algorithm tool library—work together. This also enhances the efficiency of vector transformation, enables persistent vector database, and allows the vector storage middleware to provide services to external applications. These applications can be directly called upon when in use, making the process convenient and fast.

[0085] In one possible implementation, Figure 4 A flowchart illustrating a rule file configuration method is shown, such as... Figure 4 As shown, the specific steps include:

[0086] S402: Based on the second half of the structured data file, determine the hierarchical position and field name of the field to be extracted.

[0087] In one possible embodiment, the second half-structured data file can be directly obtained to determine the hierarchical position and field name of the field to be extracted.

[0088] However, since some semi-structured data files contain a large amount of data with a lot of specific data content, resulting in very large files, reading such large files may encounter performance issues during the configuration of rule files. Therefore, the hierarchical position and field name of the field to be extracted can be determined in the following two ways.

[0089] In another possible embodiment, the data information in the second semi-structured data file is simplified to obtain the basic framework structure corresponding to the second semi-structured data file. The basic framework structure is saved as a semi-structured data template. Based on the semi-structured data template, the hierarchical position and field name of the field to be extracted are determined.

[0090] In another possible embodiment, the semi-structured data template of the second semi-structured data file can be directly obtained, and the hierarchical position and field name of the field to be extracted can be determined from the semi-structured data template.

[0091] The basic framework of the second half-structured data file is basically the same as that of the first half-structured data file. The data included in the first and second half-structured data files must be semi-structured data in the same format, such as JSON data.

[0092] Furthermore, whether determining the hierarchical position and field name of the field to be extracted through the second semi-structured data file mentioned above, or determining the hierarchical position and field name of the field to be extracted based on the semi-structured data template, it can be determined through the following visualization interface. The specific method may include: displaying all fields in the data selection interface, responding to the user selecting the field to be extracted in the data selection interface, and recording the hierarchical position and field name of multiple fields to be extracted.

[0093] Users can use the visual interface provided by the rule configuration module in this disclosure, namely the data selection interface, to select which fields to be extracted and put into the rule file during the rule configuration process.

[0094] S404: Establish a rule file that includes a first mapping relationship between the fields to be extracted and the data structure of the vector storage middleware, and a second mapping relationship between the fields to be extracted and the vectorization conversion algorithm.

[0095] In one possible embodiment, the specific method of establishing the rule file may include the following steps: establishing a first mapping relationship based on each field name and vector storage middleware; determining the hierarchical relationship based on the hierarchical position of each field to be extracted; if the hierarchical relationship is determined to be valid, adding a vector conversion algorithm to the fields to be extracted that have been determined to be valid; and storing the first mapping relationship and the second mapping relationship between the fields to be extracted that have been determined to be valid and the vectorization conversion algorithm as a rule file.

[0096] In another possible embodiment, the step of establishing a first mapping relationship based on each field name and the vector storage middleware may include: reading in the schema information of the vector storage middleware; the schema information is used to describe the data structure in the vector storage middleware; the vector storage middleware is used to store vectors after the semi-structured data conversion, and establishing a first mapping relationship based on each field name and the schema information.

[0097] In another possible implementation, since the semi-structured data file has a multi-layered nested structure, unlike the two-dimensional table schema, a feasibility test is required before conversion. Constraints are applied to the selected fields to be extracted. The specific constraint method can be based on the hierarchical position. The hierarchical relationship of the selected fields to be extracted is determined. If the determination passes, a vector conversion algorithm is added. The first mapping relationship and the second mapping relationship between the determined extracted fields and the vectorization conversion algorithm are stored as a rule file. When storing as a rule file, the rule file format must conform to the XML specification and record relevant configuration content.

[0098] One specific method for determining hierarchical relationships based on hierarchical position can be illustrated by the following: Determine the number of data points for each field to be extracted based on its hierarchical position. If the number of data points for each field to be extracted is consistent, the hierarchical relationship determination is successful.

[0099] For example, let's illustrate this with a small code snippet. The specific code is shown below:

[0100]

[0101] If the selected fields to be extracted are "name", "age" and "address", after recording their field names and hierarchical positions, establish the first mapping relationship and determine their hierarchical relationship. Since the number of data for the above three fields to be extracted is 1, their hierarchical relationship is determined to be successful.

[0102] If the selected fields to be extracted are "name", "age", "type", and "value", it can be seen that the number of data entries for "type" and "value" is 2, which is different from the number of data entries for "name" and "age". After being converted into vectors, they cannot be stored in the vector storage middleware. Even if a rule file is created, an error will occur when storing the converted vectors in the vector storage middleware. Therefore, the hierarchical relationship determination fails. If the hierarchical relationship determination fails, the user will be returned to the data selection interface, and will be prompted that the hierarchical relationship of the selected fields to be extracted fails, and the fields to be extracted should be selected again.

[0103] In one possible implementation, embodiments of this disclosure provide a flowchart illustrating another rule file configuration process, such as... Figure 5 As shown, it includes the following steps:

[0104] S502: Simplify the data information in the second half of the structured data file to obtain the basic framework structure corresponding to the second half of the structured data file.

[0105] S504: Save the basic framework structure as a semi-structured data template.

[0106] S506: Based on the semi-structured data template, display all fields in the data selection interface.

[0107] S508: In response to the user selecting fields to be extracted in the data selection interface, record the hierarchical position and field name of multiple fields to be extracted.

[0108] S510: Read in the schema information of the vector storage middleware. The schema information is used to describe the data structure in the vector storage middleware. The vector storage middleware is used to store vectors after the semi-structured data has been converted.

[0109] S512: Establish the first mapping relationship based on each field name and pattern information.

[0110] S514: Determine whether the amount of data for each field to be extracted is consistent based on its hierarchical position. If consistent, proceed to S516; otherwise, prompt an error and return to step S506.

[0111] S516: Add a vector conversion algorithm to the fields to be extracted that have passed the judgment.

[0112] S518: Store the first mapping relationship and the second mapping relationship between the fields to be extracted that have passed the judgment and the vectorization conversion algorithm as a rule file.

[0113] The method provided in this embodiment of the present disclosure pre-configures a rule file, which includes a first mapping relationship between the field to be extracted and the data structure of the vector storage middleware, and a second mapping relationship between the field to be extracted and the vectorization conversion algorithm. This not only directly determines the algorithm for vectorization processing of the data to be converted, but also determines the location where the converted data will be stored. This not only makes data extraction faster and eliminates the need for hard coding, thus improving the efficiency of vector conversion, but also makes it applicable to all semi-structured data files. Furthermore, it allows vectors to be stored in a database, achieving persistent vector database services, providing services to external applications, effectively shortening the development cycle, improving the flexibility of data extraction, and enabling convenient and quick direct access during use.

[0114] Based on the same inventive concept, this disclosure also provides a semi-structured data vector conversion device, as shown in the following embodiment. Since the principle by which this device embodiment solves the problem is similar to that of the above-described method embodiment, the implementation of this device embodiment can refer to the implementation of the above-described method embodiment, and repeated details will not be elaborated further.

[0115] Figure 6 This diagram illustrates the structure of a semi-structured data vector conversion device according to an embodiment of the present disclosure, such as... Figure 6 As shown, the semi-structured data vector conversion device 60 includes: an acquisition unit 601, used to read a first semi-structured data file and acquire data to be converted; a parsing unit 602, used to parse a pre-configured rule file and determine the correspondence between the data to be converted and the fields to be extracted in the rule file, wherein the rule file is a file including a first mapping relationship between the fields to be extracted and the data structure of the vector storage middleware, and a second mapping relationship between the fields to be extracted and the vectorization conversion algorithm; and a conversion storage unit 603, which, according to the correspondence and the second mapping relationship, calls the vector conversion algorithm to perform vector conversion on the data to be converted and stores it in the vector storage middleware according to the first mapping relationship.

[0116] Those skilled in the art will understand that various aspects of this disclosure can be implemented as a system, method, or program product. Therefore, various aspects of this disclosure can be specifically implemented in the following forms: a completely hardware implementation, a completely software implementation (including firmware, microcode, etc.), or a combination of hardware and software aspects, collectively referred to herein as a "circuit," "module," or "system."

[0117] The following reference Figure 7 To describe an electronic device 700 according to such an embodiment of the present disclosure. Figure 7 The electronic device 700 shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments disclosed herein.

[0118] like Figure 7 As shown, the electronic device 700 is manifested in the form of a general-purpose computing device. The components of the electronic device 700 may include, but are not limited to: at least one processing unit 710, at least one storage unit 720, and a bus 730 connecting different system components (including storage unit 720 and processing unit 710).

[0119] The storage unit stores program code that can be executed by the processing unit 710, causing the processing unit 710 to perform the steps described in the "Exemplary Methods" section of this specification according to various exemplary embodiments of this disclosure. For example, the processing unit 710 can perform the steps of any of the above method embodiments.

[0120] Storage unit 720 may include a readable medium in the form of a volatile storage unit, such as random access memory (RAM) 7201 and / or cache memory 7202, and may further include a read-only memory (ROM) 7203.

[0121] The storage unit 720 may also include a program / utility 7204 having a set (at least one) program module 7205, such program module 7205 including but not limited to: an operating system, one or more application programs, other program modules and program data, each or some combination of these examples may include an implementation of a network environment.

[0122] Bus 730 can represent one or more of several types of bus structures, including a memory cell bus or memory cell controller, a peripheral bus, a graphics acceleration port, a processing unit, or a local bus using any of the various bus structures.

[0123] Electronic device 700 can also communicate with one or more external devices 740 (e.g., keyboard, pointing device, Bluetooth device, etc.), and with one or more devices that enable a user to interact with electronic device 700, and / or with any device that enables electronic device 700 to communicate with one or more other computing devices (e.g., router, modem, etc.). This communication can be performed via input / output (I / O) interface 750. Furthermore, electronic device 700 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) via network adapter 760. As shown, network adapter 760 communicates with other modules of electronic device 700 via bus 730. It should be understood that, although not shown in the figures, other hardware and / or software modules can be used in conjunction with electronic device 700, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0124] From the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solutions according to the embodiments of this disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) or on a network, including several instructions to cause a computing device (such as a personal computer, server, terminal device, or network device, etc.) to execute the methods according to the embodiments of this disclosure.

[0125] Specifically, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as a computer program product or a computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the methods described above.

[0126] In exemplary embodiments of this disclosure, a computer-readable storage medium is also provided, which may be a readable signal medium or a readable storage medium. A program product capable of implementing the methods described above is stored thereon. In some possible implementations, various aspects of this disclosure may also be implemented as a program product including program code, which, when run on a terminal device, causes the terminal device to perform the steps described in the "Exemplary Methods" section of this specification according to various exemplary embodiments of this disclosure.

[0127] More specific examples of computer-readable storage media in this disclosure may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0128] In this disclosure, a computer-readable storage medium may include a data signal propagated in baseband or as part of a carrier wave, carrying readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A readable signal medium may also be any readable medium other than a readable storage medium, capable of transmitting, propagating, or transmitting a program for use by or in connection with an instruction execution system, apparatus, or device.

[0129] Optionally, the program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical fiber, RF, etc., or any suitable combination thereof.

[0130] In practical implementation, program code for performing the operations of this disclosure can be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java and C++, and conventional procedural programming languages ​​such as C or similar languages. The program code can execute entirely on the user's computing device, partially on the user's device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0131] It should be noted that although several modules or units for the device used to perform actions have been mentioned in the detailed description above, this division is not mandatory. In fact, according to embodiments of this disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided and embodied by multiple modules or units.

[0132] Furthermore, although the steps of the method in this disclosure are described in a specific order in the accompanying drawings, this does not require or imply that the steps must be performed in that specific order, or that all the steps shown must be performed to achieve the desired result. Additional or alternative steps may be omitted, multiple steps may be combined into one step, and / or a step may be broken down into multiple steps.

[0133] From the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solutions according to the embodiments of this disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) or on a network, including several instructions to cause a computing device (such as a personal computer, server, mobile terminal, or network device, etc.) to execute the methods according to the embodiments of this disclosure.

[0134] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This disclosure is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the appended claims.

Claims

1. A method for converting semi-structured data vectors, characterized in that, The method includes: Read the first half of the structured data file to obtain the data to be converted; The pre-configured rule file is parsed to determine the correspondence between the data to be converted and the fields to be extracted in the rule file; the rule file is a file that includes a first mapping relationship between the fields to be extracted and the data structure of the vector storage middleware, and a second mapping relationship between the fields to be extracted and the vectorization conversion algorithm. Based on the correspondence and the second mapping relationship, the vector conversion algorithm is invoked to perform vector conversion on the data to be converted and stored in the vector storage middleware according to the first mapping relationship; The configuration methods for the rule files include: Based on the second half of the structured data file, determine the hierarchical position and field name of the fields to be extracted; A rule file is established that includes a first mapping relationship between the field to be extracted and the data structure of the vector storage middleware, and a second mapping relationship between the field to be extracted and the vectorization conversion algorithm; The rule file for establishing the first mapping relationship between the field to be extracted and the data structure of the vector storage middleware, and the second mapping relationship between the field to be extracted and the vectorization transformation algorithm, includes: A first mapping relationship is established based on each field name and the vector storage middleware; Determine the hierarchical relationship based on the hierarchical position of each field to be extracted; If the hierarchical relationship is determined to be valid, a vector conversion algorithm is added to the fields to be extracted that have passed the determination. Store the first mapping relationship and the second mapping relationship between the extracted fields that have passed the judgment and the vectorization conversion algorithm as a rule file; The step of determining the hierarchical relationship based on the hierarchical position of each field to be extracted includes: The amount of data for each field to be extracted is determined based on its hierarchical position. If the amount of data in each of the fields to be extracted is the same, then the hierarchical relationship determination is successful.

2. The method according to claim 1, characterized in that, The step of determining the hierarchical position and field name of the field to be extracted based on the second semi-structured data file includes: Simplify the data information in the second half-structured data file to obtain the basic framework structure corresponding to the second half-structured data file; Save the aforementioned basic framework structure as a semi-structured data template; Based on the semi-structured data template, determine the hierarchical position and field name of the field to be extracted.

3. The method according to any one of claims 1 or 2, characterized in that, Determining the hierarchical position and field name of the field to be extracted includes: Display all fields in the data selection interface; In response to the user selecting the fields to be extracted on the data selection interface, the hierarchical position and field name of multiple fields to be extracted are recorded.

4. The method according to claim 1, characterized in that, The step of establishing a first mapping relationship based on each field name and the vector storage middleware includes: The schema information of the vector storage middleware is read in; the schema information is used to describe the data structure in the vector storage middleware; the vector storage middleware is used to store vectors after the semi-structured data has been converted. A first mapping relationship is established based on each field name and the pattern information.

5. The method according to claim 1, characterized in that, After the step of calling a vector conversion algorithm to perform vector conversion on the data to be converted according to the correspondence and the second mapping relationship, and storing it in the vector storage middleware according to the first mapping relationship, the method further includes: In response to a user's service provision request, the vector corresponding to the semi-structured data is extracted from the vector storage middleware to provide services to external applications.

6. A semi-structured data vector conversion device, characterized in that, include: The acquisition unit is used to read the first semi-structured data file and obtain the data to be converted. The parsing unit is used to parse the pre-configured rule file and determine the correspondence between the data to be converted and the fields to be extracted in the rule file; the rule file is a file that includes a first mapping relationship between the fields to be extracted and the data structure of the vector storage middleware, and a second mapping relationship between the fields to be extracted and the vectorization conversion algorithm. A conversion storage unit is used to call a vector conversion algorithm to perform vector conversion on the data to be converted according to the correspondence and the second mapping relationship, and store it in the vector storage middleware according to the first mapping relationship; The semi-structured data vector conversion device is also used to configure a rule file, including: determining the hierarchical position and field name of the field to be extracted based on the second semi-structured data file; and establishing a rule file that includes a first mapping relationship between the field to be extracted and the data structure of the vector storage middleware, and a second mapping relationship between the field to be extracted and the vectorization conversion algorithm. The semi-structured data vector conversion device is also used to establish a first mapping relationship based on each field name and the vector storage middleware; and to determine the hierarchical relationship based on the hierarchical position of each field to be extracted. If the hierarchical relationship is determined to be valid, a vector conversion algorithm is added to the fields to be extracted that have passed the determination; the first mapping relationship and the second mapping relationship between the fields to be extracted that have passed the determination and the vectorization conversion algorithm are stored as a rule file. The semi-structured data vector conversion device is further configured to determine the amount of data for each field to be extracted based on the hierarchical position of each field to be extracted; if the amount of data for each field to be extracted is consistent, the hierarchical relationship determination is successful.

7. A semi-structured data vector conversion system, characterized in that, include: Data extraction module, rule configuration module, and vector transformation algorithm tool library; The data extraction module is used to read the first semi-structured data file and obtain the data to be converted. The data extraction module is used to parse the pre-configured rule file in the rule configuration module and determine the correspondence between the data to be converted and the fields to be extracted in the rule file; the rule file is a file that includes a first mapping relationship between the fields to be extracted and the data structure of the vector storage middleware, and a second mapping relationship between the fields to be extracted and the vectorization conversion algorithm. The data extraction module is used to call a vector conversion algorithm from the vector conversion algorithm tool library to perform vector conversion on the data to be converted according to the correspondence and the second mapping relationship, and store it in the vector storage middleware according to the first mapping relationship; The rule configuration module is also used to determine the hierarchical position and field name of the field to be extracted based on the second semi-structured data file; and to establish a rule file that includes a first mapping relationship between the field to be extracted and the data structure of the vector storage middleware, and a second mapping relationship between the field to be extracted and the vectorization conversion algorithm. The rule configuration module is also used to establish a first mapping relationship based on each field name and the vector storage middleware; and to determine the hierarchical relationship based on the hierarchical position of each field to be extracted. If the hierarchical relationship is determined to be valid, a vector conversion algorithm is added to the fields to be extracted that have passed the determination; the first mapping relationship and the second mapping relationship between the fields to be extracted that have passed the determination and the vectorization conversion algorithm are stored as a rule file. The rule configuration module is also used to determine the amount of data for each field to be extracted based on its hierarchical position; if the amount of data for each field to be extracted is the same, then the hierarchical relationship determination is successful.

8. An electronic device, characterized in that, include: processor; as well as Memory for storing the executable instructions of the processor; The processor is configured to execute the method of any one of claims 1 to 5 by executing the executable instructions.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the method described in any one of claims 1 to 5.

Images