Metadata construction method and device based on container technology, medium and equipment

Abstract

The invention provides a metadata construction method and device based on a container technology, a medium and equipment, and the method comprises the steps: receiving a metadata construction request of original data, and determining the parameter information of the original data; and calling a container for constructing metadata, inputting the parameter information into the container to wake up a client tool of the container, selecting a corresponding metadata processor based on the client tool, and enabling the container to obtain original data and construct corresponding metadata according to the original data. And according to the metadata constructed by the container, updating the metadata index table according to a preset storage rule. The metadata construction tasks are isolated through the container, so that the metadata construction tasks have good resource isolation. According to the method, the metadata construction task is executed in a multi-thread and high-concurrency mode through management and scheduling of the container, the availability and the resource utilization rate of the system are improved while the construction efficiency of the metadata construction task is improved, and the system can be easily and horizontally expanded by increasing nodes so as to deal with burst traffic.

Description

Technical Field

[0001] The present invention relates to the field of data processing, and in particular to a method, device, medium and equipment for constructing metadata based on container technology. Background Art

[0002] With the development of computer technology, metadata, as data describing data, provides detailed information about data, helps data management and use, and greatly improves the efficiency of data use. Therefore, the efficiency of metadata construction is crucial.

[0003] In the prior art, metadata is usually built through bare metal deployment, such as running a metadata building program on a bare metal, directly using underlying hardware resources to build metadata, etc., and metadata building requests are usually processed by a single service on a single server.

[0004] However, with the development of big data, for some data sets that need to be highly dynamic, such as real-time transaction data, sensor data, etc., or data that needs to be monitored in real time, such as system performance monitoring data, users need metadata to be updated in real time to ensure the efficiency of data use and make timely decisions, resulting in the construction and update requirements of metadata often being high concurrency and low latency. However, the existing metadata construction method is difficult to efficiently handle a large number of concurrent metadata construction requests. Therefore, this specification provides a metadata construction method, device, medium and equipment based on container technology. Summary of the invention

[0005] This specification provides a metadata construction method, device, medium and equipment based on container technology to partially solve the above-mentioned problems existing in the prior art.

[0006] This manual adopts the following technical solutions: A metadata construction method based on container technology, comprising: Receiving a metadata construction request for original data, and determining parameter information of the original data, wherein the parameter information at least includes a storage address of the original data; Calling a container for constructing metadata, inputting the parameter information into the container to wake up a client tool of the container, and based on a metadata processor called by the client tool, enabling the container to obtain the original data and construct corresponding metadata according to the original data; According to the metadata constructed by the container, the metadata index table is updated according to the preset storage rules.

[0007] Optionally, the parameter information also includes a data source type, specifically including: Inputting the parameter information into the container to wake up the client tool of the container, and based on the metadata processor called by the client tool, enabling the container to obtain the original data and construct corresponding metadata according to the original data, specifically including: The parameter information is input into the container, so that the container determines the data source type of the original data according to the parameter information, and selects a metadata processor corresponding to the data source type from the client tool of the container; the metadata construction task corresponding to the container is started through the selected metadata processor, so that the container executes the metadata construction task to obtain the original data and construct corresponding metadata according to the original data.

[0008] Optionally, the metadata construction request carries an update frequency of the metadata of the original data, and the method further includes: Inputting the parameter information and the update frequency into the container, so that the container periodically obtains the original data from the storage address and constructs metadata of the obtained original data according to the input parameter information and the update frequency; According to the metadata constructed by the container, the metadata index table is updated according to the preset storage rules, specifically including: According to the update frequency, the metadata index table is updated regularly based on the constructed metadata and according to the preset storage rules.

[0009] Optionally, updating the metadata index table according to the metadata constructed by the container and according to a preset storage rule specifically includes: Determine, based on the original data, a unique identifier of metadata constructed by the container; Determine a first identification set according to the unique identification of each metadata stored in the metadata index table; According to the unique identifier and the stored first identifier set, the metadata index table is updated according to a preset storage rule.

[0010] Optionally, updating the metadata index table according to the metadata constructed by the container and according to a preset storage rule specifically includes: Determining containers for constructing metadata; Determine a second identifier set according to the unique identifier corresponding to the metadata constructed by each container; According to the comparison between each unique identifier included in the second identifier set and each unique identifier in the first identifier set, each unique identifier included in the first identifier set but not included in the second identifier set is determined as an identifier to be deleted; The metadata corresponding to the to-be-deleted identifier is deleted from the metadata index table, the first identifier set is updated according to the second identifier set, and the metadata corresponding to each unique identifier in the second identifier set is updated to the metadata index table according to a preset storage rule.

[0011] Optionally, before updating the metadata index table according to the metadata constructed by the container and according to a preset storage rule, the method further includes: Determining metadata constructed by the container, and judging whether the metadata conforms to a preset format; If not, the metadata is converted into the preset format.

[0012] Optionally, the metadata is stored in the metadata index table in the form of key values, and the method further includes: In response to a query instruction, determining retrieval information of the original data to be retrieved according to the query instruction; A search is performed in the metadata index table according to the search information, and at least one key value combination is output according to the search result.

[0013] This specification provides a metadata construction device based on container technology, including: A determination module, configured to receive a metadata construction request for original data, and determine parameter information of the original data, wherein the parameter information at least includes a storage address of the original data; A construction module, configured to call a container for constructing metadata, input the parameter information into the container to wake up a client tool of the container, and based on a metadata processor called by the client tool, enable the container to obtain the original data and construct corresponding metadata according to the original data; The storage module is used to update the metadata index table according to the metadata constructed by the container and according to preset storage rules.

[0014] This specification provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the above-mentioned metadata construction method based on container technology when executing the program.

[0015] At least one of the above technical solutions adopted in this specification can achieve the following beneficial effects: In the metadata construction method based on container technology described in this specification, a metadata construction request of raw data is received, and parameter information of the raw data is determined. A container for constructing metadata is called, and parameter information is input into the container to wake up the client tool of the container, and a corresponding metadata processor is selected based on the client tool, so that the container obtains the raw data and constructs corresponding metadata according to the raw data. According to the metadata constructed by the container, a metadata index table is updated according to a preset storage rule.

[0016] From the above method, it can be seen that the metadata construction tasks are isolated through containers, so that each metadata construction task has good resource isolation. Through the management and scheduling of containers, multi-threaded high-concurrency execution of metadata construction tasks is achieved, which improves the construction efficiency of metadata construction tasks while improving the availability and resource utilization of the system. It can be easily expanded horizontally by adding nodes to cope with sudden traffic. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The drawings described herein are used to provide a further understanding of this specification and constitute a part of this specification. The illustrative embodiments and descriptions of this specification are used to explain this specification and do not constitute an improper limitation on this specification. In the drawings: Figure 1 A schematic diagram of the process of a metadata construction method based on container technology provided in this specification; Figure 2 A schematic diagram of a server processing parallel metadata construction requests provided in this specification; Figure 3 A schematic diagram of a container provided for this specification; Figure 4 A schematic diagram of metadata construction and storage provided in this specification; Figure 5 A schematic diagram of a metadata construction device based on container technology provided in this specification; Figure 6 The corresponding Figure 1 Schematic diagram of electronic equipment. DETAILED DESCRIPTION

[0018] In order to make the purpose, technical solutions and advantages of this specification more clear, the technical solutions of this specification will be clearly and completely described in combination with the specific embodiments of this specification and the corresponding drawings. Obviously, the described embodiments are only part of the embodiments of this specification, not all of the embodiments. Based on the embodiments in this specification, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0019] With the development of computer technology, metadata, as data describing data, provides detailed information about data, helps data management and use, and greatly improves the efficiency of data use. However, with the development of big data, for some data sets that require high dynamics, such as real-time transaction data, sensor data, etc., or data that requires real-time monitoring, such as system performance monitoring data, users need metadata to be updated in real time to ensure the efficiency of data use and make timely decisions, resulting in the construction and update requirements of metadata often being high concurrency and low latency. The existing bare metal single service generation metadata construction method, or the use of traditional virtual machines to deploy single services to build metadata, will cause the server to be difficult to efficiently handle a large number of concurrent metadata construction requests. In particular, when multiple metadata construction tasks are running at the same time, resource competition and resource conflicts may occur between the metadata construction tasks, and data competition and even system crashes may occur. Therefore, this specification provides a metadata construction method based on container technology, so as to dynamically adjust the execution order and resource allocation of tasks through container technology, optimize system performance, reduce resource competition and conflicts, and improve the reliability and stability of metadata construction.

[0020] The technical solutions provided by the embodiments of this specification are described in detail below in conjunction with the accompanying drawings.

[0021] Figure 1 A schematic diagram of the process of a metadata construction method based on container technology provided in this specification, specifically: S100: Receive a metadata construction request for original data, and determine parameter information of the original data, where the parameter information at least includes a storage address of the original data.

[0022] In one or more embodiments of the present specification, there is no restriction on which specific device executes the metadata construction method based on container technology, such as a mobile terminal and a server. However, in container technology, a server is usually responsible for managing and scheduling containers. Therefore, the present specification also uses the server executing the metadata construction method based on container technology as an example for description. The server may be a single device or composed of multiple devices, such as a distributed server. The present specification does not impose any restrictions on this and may also refer to a server of a cloud service.

[0023] Since metadata is data that describes data, in order to perform the metadata construction task, the server should first obtain parameter information of the original data whose metadata is to be constructed, so that the container constructs the metadata of the original data based on the parameter information of the original data.

[0024] Specifically, the server can determine the parameter information of the original data according to the metadata construction request. The parameter information is the data and information required to perform the metadata construction task, including at least the storage address of the original data. In addition, the parameter information can also include the data source type of the original data, the Uniform Resource Locator (URL) for accessing the storage address of the original data, and the interface information for accessing the storage address of the original data. The data source type is the type of the data storage platform. The server can connect to the data storage platform through the storage address of the data source determined by the metadata construction task.

[0025] In addition, in one or more embodiments of the present specification, there is no limitation on the specific manner in which the server determines the parameter information of the original data based on the metadata construction request. The parameter information of the original data may be carried in the metadata construction request, or the metadata construction request may carry the storage address of the parameter information of the original data. Of course, other methods may also be used. Since there are many possible methods, they will not be elaborated in this specification.

[0026] S102: calling a container for constructing metadata, inputting the parameter information into the container to wake up a client tool of the container, and based on a metadata processor called by the client tool, enabling the container to obtain the original data and construct corresponding metadata according to the original data.

[0027] With the development of big data, metadata construction often needs to be highly concurrent, that is, the server often receives a large number of metadata construction requests. In order to prevent server resource competition and conflicts caused by a large number of metadata construction tasks when responding to each metadata construction request with high concurrency, container technology can be used to package the program used to build metadata and the program's operating environment and other resources into an image. Then, the server responds to each metadata construction request and calls the container equipped with the image to execute each metadata construction task, thereby isolating the tasks executed in each container from each other. At the same time, the system performance is optimized through container management and scheduling, and resource competition and conflicts are reduced.

[0028] Specifically, Figure 2 As shown, Figure 2A schematic diagram of a server processing parallel metadata construction requests provided in this specification, the server calls at least one container for processing the metadata construction request for each metadata construction request, inputs parameter information into the container to wake up the client tool (client) preset by the container, and enables the container to obtain original data and construct corresponding metadata based on the original data. That is, when the server receives n metadata construction requests, the server calls at least n containers to process each metadata construction request at the same time. Through the server and the client in each container called, the server-client architecture is realized, so that when metadata is constructed, the traditional single server processes each parallel metadata construction request, which is transformed into multiple clients processing each metadata construction request at the same time, that is, through the container technology, the single server processes each parallel metadata construction request, which is transformed into multiple servers processing each parallel metadata construction request at the same time. Reduce resource competition and resource conflict, as well as the situation of system crash caused by data competition. At the same time, through the integration of different types of metadata processors by the client, the impact on the server executing unified metadata storage services is reduced.

[0029] It should be noted that the container image used to perform the metadata construction task is pre-built. The container image contains all the programs required to perform the metadata construction task and the program's operating environment, and each container image contains a corresponding client. The client can perform the metadata construction task by calling the metadata processor, and the server can interact with the client in each called container to enable the corresponding container to perform the corresponding metadata construction task.

[0030] Of course, in one or more embodiments of this specification, there is no limit on where the preset container image is stored, and it can be in the same device as the server or in a different device from the server. There is no limit on how many containers need to be called to execute a metadata building task, but at least one.

[0031] In addition, in one or more embodiments of the present specification, the specific form of the metadata construction request is not limited. It can be a string of codes, a string of data, a string of signals, or even a piece of text. Since there are many forms of instructions, this specification does not list them one by one.

[0032] In addition, when calling the container to perform the metadata construction task, the server can select the target running area according to the computing resources, running resources or storage resources of the device where the server is located, and run the container there. If the server is a distributed server, the server can also determine the node with the most idle resources to run the container based on the resources of each node, or determine that the called container is in the device running area based on other methods. The server can be set according to actual needs, and this description says that there is no restriction on this.

[0033] Then, in order to construct metadata through the container, the server can input parameter information of the original data into the container, so that the container performs the metadata construction task based on the input parameter information.

[0034] Specifically, the server inputs the obtained parameter information into the container, and then the container determines the storage address of the original data based on the parameter information, obtains the original data by accessing the storage address, and then constructs metadata corresponding to the original data based on the original data.

[0035] It should be noted that in one or more embodiments of this specification, the metadata is not limited to the specific protocol used to transmit between the client and the server, such as HTTPS, etc. Of course, this specification does not limit the subsequent transmission between the server and the storage through the protocol.

[0036] S104: According to the metadata constructed by the container, the metadata index table is updated according to a preset storage rule.

[0037] In order to achieve accuracy and efficiency in the subsequent retrieval of original data based on the constructed metadata, the server may update the metadata index table with the metadata constructed by each container according to preset storage rules.

[0038] Specifically, the server determines the metadata of the container that has been built, and then determines whether the metadata is stored in the metadata index table based on the metadata that has been built. If so, the metadata stored in the metadata index table is replaced by the metadata that has been built. If not, the metadata is directly stored.

[0039] It should be noted that after each container completes the construction of metadata, it directly updates the constructed metadata to the metadata index table accordingly. According to the task number of the metadata construction task, the metadata corresponding to the task number can be updated in the metadata index table. Of course, the metadata constructed by each container can also be updated to the metadata index table based on other numbers, such as updating the metadata constructed by the container according to the container number. The constructed metadata can be stored in the metadata index table in the form of a key-value combination to be compatible with metadata of different data source types, and to facilitate subsequent retrieval of original data based on metadata. Of course, other forms of preset storage rules can also be used for storage, which is not limited to this in one or more embodiments of this specification and can be set according to actual needs.

[0040] In addition, in one or more embodiments of the present specification, it is not limited what specific preset storage rules the server adopts to update the metadata index table. The preset storage rules refer to the content of the metadata stored in the metadata index table, such as the metadata construction task number, metadata, identification set, etc. The constructed metadata can also be stored using a meta-model, wherein the meta-model is a metadata storage rule that defines the structure and semantics of metadata and provides a basis for the management and use of metadata. The metadata index table constructed according to the meta-model can store metadata numbers, metadata names, data source types, and metadata. Metadata includes: corresponding key-value combinations stored according to the data source types to which different metadata belong.

[0041] based on Figure 1 A metadata construction method based on container technology receives a metadata construction request for raw data and determines parameter information of the raw data. A container for constructing metadata is called and runs, and the parameter information is input into the container, so that the container obtains the storage address of the raw data according to the input parameter information, obtains the raw data by accessing the storage address, and constructs corresponding metadata according to the raw data. According to the metadata constructed by the container, a metadata index table is updated according to a preset storage rule.

[0042] From the above method, it can be seen that the metadata construction tasks are isolated through containers, so that each metadata construction task has good resource isolation. Through the management and scheduling of containers, multi-threaded high-concurrency execution of metadata construction tasks is achieved, which improves the construction efficiency of metadata construction tasks while improving the availability and resource utilization of the system. It can be easily expanded horizontally by adding nodes to cope with sudden traffic.

[0043] In addition, when building metadata, different types of data will have different processing flows according to their types, and different processing flows require different processing steps. Therefore, when the parameter information contains the data source type, when building metadata, the server can input the parameter data into the called container, so that the client in the called container can select the corresponding metadata processor according to the data source type contained in the parameter information, thereby implementing metadata construction tasks for raw data of different data source types. Furthermore, when packaging the container image, the image contains the client and the environment that the client depends on, and the client contains different types of metadata processors, each type of processor is used to process different types of raw data. Figure 3 As shown, Figure 3 A schematic diagram of a container provided in this specification, wherein each container includes a client, and each client includes a metadata processor for processing different data.

[0044] That is, if the parameter information does not include the data source type, a metadata processor from a preset type of client is selected to process the metadata construction task. If the parameter information includes the data source type corresponding to the original data, the server can determine the metadata processor of the corresponding type of client based on the data source type and call it to process the metadata construction task of the data source type.

[0045] Furthermore, in order to meet the needs of metadata construction for certain data sets that require high dynamics, such as real-time transaction data, sensor data, etc., or data that requires real-time monitoring, such as system performance monitoring data, the server prevents the server from frequently calling and deleting containers for metadata construction tasks for raw data at the same storage address. The server can also determine the update frequency of the metadata carried in the metadata construction request, that is, regularly perform incremental construction of the metadata of the raw data at the same storage address through a container.

[0046] Specifically, the parameter information is input into the container, so that the container periodically obtains the original data from the storage address based on the update frequency of the metadata in the parameter information, and constructs the metadata of the original data obtained each time.

[0047] It should be noted that when determining the metadata update frequency, the server can, in addition to determining it based on the metadata construction request, also perform the metadata construction task based on the preset metadata update frequency. Of course, the server can also select the corresponding update frequency from the preset update frequencies as the update frequency of this metadata construction task based on the data source type of the original data.

[0048] When the container periodically obtains the original data from the storage address based on the update frequency and constructs the metadata of the original data, that is, when the metadata index table is updated, the metadata index table can also be updated periodically according to the constructed metadata and the preset storage rules according to the determined update frequency.

[0049] Of course, if the frequency of building metadata for the container is not limited, the server can obtain all the original data only once and complete the construction and storage of the metadata once, that is, a one-time full construction of the metadata.

[0050] Furthermore, when the metadata index table is updated according to the preset storage rules at the update frequency, in order to realize incremental metadata construction and reduce the storage occupancy and excessive noise caused by the number of expired metadata stored in the storage to subsequent metadata retrieval, the server updates the metadata stored in the metadata index table, which at least includes updating, inserting and deleting the constructed metadata, that is, modifying the existing metadata in the metadata index table, inserting metadata not stored in the metadata index table into the metadata index table, and deleting metadata stored in the metadata index table but not included in the metadata constructed this time.

[0051] Specifically, when updating metadata, the server can determine the unique identifier of the metadata corresponding to the original data according to each original data, and then determine the first identifier set according to the unique identifier of each metadata stored in the metadata index table. Then, the unique identifier of the metadata constructed this time is compared with each unique identifier in the first identifier set to update the original data index table.

[0052] It should be noted that when determining the unique identifier of the metadata corresponding to the original data, the unique identifier of the original data can be determined first, and then the unique identifier of the original data can be used as the unique identifier of the metadata of the original data. Of course, the unique identifier of the original data can also be encrypted, and the encrypted unique identifier can be used as the unique identifier of the metadata of the original data, and this manual does not limit this.

[0053] When deleting metadata, the server can determine each container used to construct metadata, and determine the second identifier set based on the unique identifier corresponding to the metadata constructed by each container. By comparing each unique identifier included in the second identifier set with each unique identifier in the first identifier set, each unique identifier included in the first identifier set but not included in the second identifier set is determined as the identifier to be deleted. The metadata corresponding to the identifier to be deleted is deleted from the metadata index table, and the first identifier set is updated according to the second identifier set, and the metadata corresponding to each unique identifier in the second identifier set is updated to the metadata index table according to the preset storage rules.

[0054] When there are unique identifiers that are not included in the first identifier set but are included in the second identifier set, the server can directly store the metadata corresponding to these identifiers in the metadata index table according to the preset storage rules, and add these identifiers to the first identifier set to update the first identifier set. The first identifier set and the metadata are stored separately to reduce the impact of network congestion on storage efficiency due to the different update frequencies of the two.

[0055] Furthermore, after the metadata update is completed, the server can search the stored metadata in the storage according to the input information, and output the retrieved key-value combination. It should be noted that in one or more embodiments of the present specification, the server is not limited to the specific retrieval method adopted by the server. It can be to convert the input information and each key-value combination into a vector, and determine the target key-value combination by calculating the matching degree between the vectors. It is also possible to perform keyword matching, fuzzy matching, or full-text retrieval on the input information and each key-value combination in the storage to determine the target key-value combination. It can be set according to actual needs. In addition, the above-mentioned input information refers to metadata information. Of course, if the retrieval information of the original data is not stored in the metadata index table, the server also outputs an empty or other preset identifier instead of outputting metadata.

[0056] Furthermore, in order to save storage space and protect data security, the server may also encode and compress the first identification set, and when the metadata in the storage needs to be updated, the first identification set is decoded and decompressed.

[0057] In addition, in order to facilitate subsequent indexing according to the metadata index table, the server can also unify the format of the metadata constructed by each container based on the preset storage rule. Specifically, after the server converts the metadata constructed by each container into the format of the preset storage rule, the converted metadata is added to the metadata index table according to the preset storage rule.

[0058] In addition, since the container may fail to build metadata due to network problems, configuration errors, dependency problems, etc., in order to reduce the impact of metadata construction failure on metadata retrieval, the server will provide feedback to the user and send the error log to the user after determining that the metadata construction has failed, so that the user can analyze and handle the cause of the failure.

[0059] It should be noted that in one or more embodiments of this specification, there is no restriction on the specific container technology used to implement metadata construction. It can be implemented through docker, kubernetes, argo workflows, etc. Of course, it is also possible to use existing container technologies in combination, or through other technologies. The specific settings can be based on actual needs.

[0060] This specification also provides an embodiment of constructing metadata based on container technology to describe the metadata construction method based on container technology.

[0061] Specifically, Figure 4 As shown, Figure 4 A schematic diagram of metadata construction and storage provided in this specification. The server responds to each metadata construction request, wherein each metadata construction request carries the data source type and storage address of the original data of the metadata to be constructed. Then, for each metadata construction request, the server sends the corresponding parameter information to the container according to the data source type of the original data of the metadata construction request, wakes up the client of the corresponding data source type in the called container, and loads the services required in the metadata construction task of the container through the awakened client in the container.

[0062] Then, the client of the container will pass the parameter information of the container as the task data required for the container to execute the metadata construction task, start running the metadata construction task, and then the container obtains the original data from the storage address based on the storage address of the original data to implement metadata construction based on the original data. If the parameter information also includes the data range of the original data, then when the container accesses the storage address to obtain the original data, it should only obtain the data within the data range as the original data.

[0063] If the server updates the metadata of each container to storage, the server can also compare the second identification set with the first identification set after decoding and decompression in the storage to determine the updated metadata and the metadata to be deleted. The metadata can be updated, inserted and deleted. The first identification set is updated, and the updated first identification set can be encoded and compressed.

[0064] If the parameter information includes an update frequency, the container periodically executes the metadata construction task based on the update frequency, and determines the unique identifier of the constructed metadata based on the unique identifier of the original data. Then, the constructed metadata and the corresponding unique identifier are updated to the storage.

[0065] The above is a metadata construction method based on container technology provided in the embodiments of this specification. Based on the same idea, this specification also provides a corresponding metadata construction device based on container technology, such as Figure 5 shown.

[0066] A determination module 400 is configured to receive a metadata construction request for original data and determine parameter information of the original data, wherein the parameter information at least includes a storage address of the original data; A construction module 401 is used to call a container for constructing metadata, input the parameter information into the container to wake up a client tool of the container, and based on a metadata processor called by the client tool, enable the container to obtain the original data and construct corresponding metadata according to the original data; The storage module 402 is used to update the metadata index table according to the metadata constructed by the container and according to preset storage rules.

[0067] Optionally, the parameter information also includes a data source type. A construction module 401 is used to input the parameter information into the container, so that the container determines the data source type of the original data based on the parameter information, and selects a metadata processor corresponding to the data source type from the client tool of the container; the metadata construction task corresponding to the container is started through the selected metadata processor, so that the container executes the metadata construction task to obtain the original data and construct corresponding metadata based on the original data.

[0068] Optionally, the metadata construction request carries the update frequency of the metadata of the original data; a construction module 401 is used to input the parameter information and the update frequency into the container, so that the container periodically obtains the original data from the storage address and constructs the metadata of the obtained original data according to the input parameter information and the update frequency; a storage module 402 is used to periodically update the metadata index table according to the constructed metadata and preset storage rules according to the update frequency.

[0069] Optionally, the storage module 402 is used to determine the unique identifier of the metadata constructed by the container based on the original data; determine the first identifier set based on the unique identifier of each metadata stored in the metadata index table; and update the metadata index table according to preset storage rules based on the unique identifier and the stored first identifier set.

[0070] Optionally, the storage module 402 is used to determine each container used to construct metadata; determine a second identifier set according to the unique identifier corresponding to the metadata constructed by each container; compare each unique identifier included in the second identifier set with each unique identifier in the first identifier set, and determine each unique identifier included in the first identifier set but not included in the second identifier set as an identifier to be deleted; delete the metadata corresponding to the identifier to be deleted from the metadata index table, update the first identifier set according to the second identifier set, and update the metadata corresponding to each unique identifier in the second identifier set to the metadata index table according to a preset storage rule.

[0071] Optionally, the storage module 402 is used to determine the metadata constructed by the container and determine whether the metadata conforms to a preset format; if not, convert the metadata into the preset format.

[0072] Optionally, the storage module 402 is used to respond to a query instruction, determine retrieval information of the original data to be retrieved according to the query instruction; search in the metadata index table according to the retrieval information, and output at least one key value combination according to the retrieval result.

[0073] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0074] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0075] This specification also provides a computer-readable storage medium, which stores a computer program, which can be used to execute the above Figure 1 A metadata construction method based on container technology is provided.

[0076] This manual also provides Figure 6As shown in Figure 6, at the hardware level, the electronic device includes a processor, an internal bus, a network interface, a memory, and a non-volatile memory, and may also include other hardware required for the business. The processor reads the corresponding computer program from the non-volatile memory into the memory and then runs it to achieve the above Figure 1 The metadata construction method based on container technology. Of course, in addition to the software implementation, this specification does not exclude other implementations, such as logic devices or a combination of software and hardware, etc., that is, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0077] In the 1990s, it was very clear whether the improvement of a technology was hardware improvement (for example, improvement of the circuit structure of diodes, transistors, switches, etc.) or software improvement (improvement of the method flow). However, with the development of technology, many improvements of the method flow today can be regarded as direct improvements of the hardware circuit structure. Designers almost always obtain the corresponding hardware circuit structure by programming the improved method flow into the hardware circuit. Therefore, it cannot be said that the improvement of a method flow cannot be implemented with hardware entity modules. For example, a programmable logic device (PLD) (such as a field programmable gate array (FPGA)) is such an integrated circuit whose logical function is determined by the user's programming of the device. Designers can "integrate" a digital system on a PLD by programming it themselves, without having to ask chip manufacturers to design and make dedicated integrated circuit chips. Moreover, nowadays, instead of manually making integrated circuit chips, this kind of programming is mostly implemented by "logic compiler" software, which is similar to the software compiler used when developing and writing programs, and the original code before compilation must also be written in a specific programming language, which is called hardware description language (HDL). There is not only one kind of HDL, but many kinds, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, RHDL (Ruby Hardware Description Language), etc. The most commonly used ones are VHDL (Very-High-Speed ​​Integrated Circuit Hardware Description Language) and Verilog. Those skilled in the art should also know that it is only necessary to program the method flow slightly in the above-mentioned hardware description languages ​​and program it into the integrated circuit, and then it is easy to obtain the hardware circuit that implements the logic method flow.

[0078] The controller may be implemented in any suitable manner, for example, the controller may take the form of a microprocessor or processor and a computer-readable medium storing a computer-readable program code (e.g., software or firmware) executable by the (micro)processor, a logic gate, a switch, an application-specific integrated circuit (ASIC), a programmable logic controller, and an embedded microcontroller, examples of which include but are not limited to the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicone Labs C8051F320, and the memory controller may also be implemented as part of the control logic of the memory. It is also known to those skilled in the art that, in addition to implementing the controller in a purely computer-readable program code manner, the controller may be implemented in the form of a logic gate, a switch, an application-specific integrated circuit, a programmable logic controller, and an embedded microcontroller by logically programming the method steps. Therefore, such a controller may be considered as a hardware component, and the devices for implementing various functions included therein may also be considered as structures within the hardware component. Or even, the devices for implementing various functions may be considered as both software modules for implementing the method and structures within the hardware component.

[0079] The systems, devices, modules or units described in the above embodiments may be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, the computer may be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0080] For the convenience of description, the above device is described in various units according to their functions. Of course, when implementing this specification, the functions of each unit can be implemented in the same or multiple software and / or hardware.

[0081] It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0082] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0083] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0084] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0085] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0086] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0087] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0088] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0089] It should be understood by those skilled in the art that the embodiments of this specification may be provided as methods, systems or computer program products. Therefore, this specification may take the form of a complete hardware embodiment, a complete software embodiment or an embodiment combining software and hardware. Moreover, this specification may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0090] This specification may be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. This specification may also be practiced in distributed computing environments where tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules may be located in local and remote computer storage media, including storage devices.

[0091] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0092] The above description is only an embodiment of this specification and is not intended to limit this specification. For those skilled in the art, this specification may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of this specification should be included in the scope of the claims of this application.

Claims

1. A metadata construction method based on container technology, characterized in that: include: Receiving a metadata construction request for original data, and determining parameter information of the original data, wherein the parameter information at least includes a storage address of the original data; Calling a container for constructing metadata, inputting the parameter information into the container to wake up a client tool of the container, and based on a metadata processor called by the client tool, enabling the container to obtain the original data and construct corresponding metadata according to the original data; According to the metadata constructed by the container, the metadata index table is updated according to the preset storage rules.

2. The method according to claim 1, characterized in that: The parameter information also includes the data source type, specifically including: Inputting the parameter information into the container to wake up the client tool of the container, and based on the metadata processor called by the client tool, enabling the container to obtain the original data and construct corresponding metadata according to the original data, specifically including: The parameter information is input into the container, so that the container determines the data source type of the original data according to the parameter information, and selects a metadata processor corresponding to the data source type from the client tool of the container; the metadata construction task corresponding to the container is started through the selected metadata processor, so that the container executes the metadata construction task to obtain the original data and construct corresponding metadata according to the original data.

3. The method according to claim 1, characterized in that: The metadata construction request carries the update frequency of the metadata of the original data, and the method further includes: Inputting the parameter information and the update frequency into the container, so that the container periodically obtains the original data from the storage address and constructs metadata of the obtained original data according to the input parameter information and the update frequency; According to the metadata constructed by the container, the metadata index table is updated according to the preset storage rules, specifically including: According to the update frequency, the metadata index table is updated regularly based on the constructed metadata and according to the preset storage rules.

4. The method according to claim 3, characterized in that: According to the metadata constructed by the container, the metadata index table is updated according to the preset storage rules, specifically including: Determine, based on the original data, a unique identifier of metadata constructed by the container; Determine a first identification set according to the unique identification of each metadata stored in the metadata index table; According to the unique identifier and the stored first identifier set, the metadata index table is updated according to a preset storage rule.

5. The method according to claim 4, characterized in that: According to the metadata constructed by the container, the metadata index table is updated according to the preset storage rules, specifically including: Determining containers for constructing metadata; Determine a second identifier set according to the unique identifier corresponding to the metadata constructed by each container; According to the comparison between each unique identifier included in the second identifier set and each unique identifier in the first identifier set, each unique identifier included in the first identifier set but not included in the second identifier set is determined as an identifier to be deleted; The metadata corresponding to the to-be-deleted identifier is deleted from the metadata index table, the first identifier set is updated according to the second identifier set, and the metadata corresponding to each unique identifier in the second identifier set is updated to the metadata index table according to a preset storage rule.

6. The method according to claim 1, characterized in that: According to the metadata constructed by the container, before updating the metadata index table according to the preset storage rule, the method further includes: Determining metadata constructed by the container, and judging whether the metadata conforms to a preset format; If not, the metadata is converted into the preset format.

7. The method according to claim 1, characterized in that The metadata is stored in the metadata index table in the form of key values, and the method further includes: In response to a query instruction, determining retrieval information of the original data to be retrieved according to the query instruction; A search is performed in the metadata index table according to the search information, and at least one key value combination is output according to the search result.

8. A metadata construction device based on container technology, characterized in that: include: A determination module, configured to receive a metadata construction request for original data, and determine parameter information of the original data, wherein the parameter information at least includes a storage address of the original data; A construction module, configured to call a container for constructing metadata, input the parameter information into the container to wake up a client tool of the container, and based on a metadata processor called by the client tool, enable the container to obtain the original data and construct corresponding metadata according to the original data; The storage module is used to update the metadata index table according to the metadata constructed by the container and according to preset storage rules.

9. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, and when the computer program is executed by a processor, the method described in any one of claims 1 to 7 is implemented.

10. An electronic device, characterized in that: The method comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor implements the method according to any one of claims 1 to 7 when executing the program.

Images