Metadata generation method and device, equipment and storage medium

Abstract

The invention provides a metadata generation method, and relates to the technical field of artificial intelligence, in particular to the technical field of big data processing and the like. The method comprises the steps of determining information of a preset position of a target file; in response to the fact that the file format of the target file is not determined based on the information, determining the file format of the target file based on the feature information of the data content in the target file; and analyzing the target file by adopting an analysis mode corresponding to the file format to generate target metadata of the target file. According to the method, the generation efficiency and accuracy of the file metadata are improved.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, specifically to technical fields such as big data processing, and in particular to a metadata generation method, device, equipment and storage medium. Background Art

[0002] File metadata is data that describes data, and is mainly used to indicate storage location, historical data, resource search, file records, and other functions. It helps users and management systems understand the attributes, sources, creation time, and other information of the data, thereby supporting the effective management and use of data. Summary of the invention

[0003] The present disclosure provides a metadata generation method, apparatus, device and storage medium.

[0004] According to a first aspect of the present disclosure, a metadata generation method is provided, including: determining information of a preset location of a target file; in response to the file format of the target file not being determined based on the information, determining the file format of the target file based on characteristic information of data content in the target file; and parsing the target file using a parsing method corresponding to the file format to generate target metadata for the target file.

[0005] According to a second aspect of the present disclosure, a metadata generating device is provided, including: an information determining module, configured to determine information of a preset position of a target file; a format determining module, configured to determine the file format of the target file based on characteristic information of data content in the target file in response to the file format of the target file not being determined based on the information; and a metadata generating module, configured to parse the target file in a parsing manner corresponding to the file format, and generate target metadata of the target file.

[0006] According to a third aspect of the present disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method described in any implementation manner in the first aspect.

[0007] According to a fourth aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, where the computer instructions are used to cause a computer to execute the method described in any implementation manner of the first aspect.

[0008] According to a fifth aspect of the present disclosure, a computer program product is provided, including a computer program, and when the computer program is executed by a processor, the computer program implements the method described in any implementation manner in the first aspect.

[0009] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0011] Figure 1 is an exemplary system architecture diagram in which the present disclosure may be applied;

[0012] Figure 2 is a flowchart of an embodiment of a metadata generation method according to the present disclosure;

[0013] Figure 3 is a flowchart of another embodiment of a metadata generation method according to the present disclosure;

[0014] Figure 4 is a flowchart of another embodiment of a metadata generation method according to the present disclosure;

[0015] Figure 5 is a flowchart of another embodiment of a metadata generation method according to the present disclosure;

[0016] Figure 6 is an application scenario diagram of the metadata generation method according to the present disclosure;

[0017] Figure 7 is a structural schematic diagram of an embodiment of a metadata generating device according to the present disclosure;

[0018] Figure 8 It is a block diagram of an electronic device used to implement the metadata generation method according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0019] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0020] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present disclosure may be combined with each other. The present disclosure will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0021] Figure 1An exemplary system architecture 100 is shown to which an embodiment of the metadata generation method or metadata generation apparatus of the present disclosure can be applied.

[0022] like Figure 1 As shown, the system architecture 100 may include terminal devices 101, 102, 103, a network 104 and a server 105. The network 104 is used to provide a medium for communication links between the terminal devices 101, 102, 103 and the server 105. The network 104 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0023] The user can use the terminal devices 101, 102, 103 to interact with the server 105 through the network 104 to receive or send information, etc. Various client applications can be installed on the terminal devices 101, 102, 103.

[0024] The terminal devices 101, 102, 103 may be hardware or software. When the terminal devices 101, 102, 103 are hardware, they may be various electronic devices, including but not limited to smart phones, tablet computers, laptop computers, desktop computers, etc. When the terminal devices 101, 102, 103 are software, they may be installed in the above electronic devices. They may be implemented as multiple software or software modules, or as a single software or software module. No specific limitation is made here.

[0025] The server 105 can provide various services. For example, the server 105 can analyze and process the target files obtained from the terminal devices 101, 102, and 103, and generate processing results (such as target metadata).

[0026] It should be noted that the server 105 can be hardware or software. When the server 105 is hardware, it can be implemented as a distributed server cluster consisting of multiple servers, or it can be implemented as a single server. When the server 105 is software, it can be implemented as multiple software or software modules (for example, for providing distributed services), or it can be implemented as a single software or software module. No specific limitation is made here.

[0027] It should be noted that the metadata generation method provided in the embodiment of the present disclosure is generally executed by the server 105 , and accordingly, the metadata generation device is generally disposed in the server 105 .

[0028] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0029] Continue to refer Figure 2, which shows a process 200 of an embodiment of a metadata generation method according to the present disclosure. The metadata generation method comprises the following steps:

[0030] Step 201, determining information of a preset location of a target file.

[0031] In this embodiment, the execution subject of the metadata generation method (eg Figure 1 The server 105 shown in the figure will first determine the information of the preset position of the target file. The target file is the file for which metadata needs to be generated, and it can be a file of any format. The preset position can be a file header or a file suffix. Accordingly, the information of the preset position can be information of the file header, such as byte information of the file header; the information of the preset position can be information of the file suffix, such as suffix information of the file suffix.

[0032] After determining the information of the preset location of the target file, the execution subject will also match the information with the information in the preset information set, and determine whether the file format of the target file can be determined based on the matching result. If the preset information corresponding to the above information is matched from the preset information set, the match is successful, and the file format corresponding to the information is determined as the file format of the target file.

[0033] Step 202: In response to the file format of the target file not being determined based on the information, the file format of the target file is determined based on characteristic information of data content in the target file.

[0034] In this embodiment, if the preset information corresponding to the above information cannot be matched from the preset information set, the match fails, and the file format of the target file cannot be determined based on the information. At this time, the execution subject will obtain the characteristic information of the data content in the target file, and determine the file format of the target file based on the characteristic information. Among them, the characteristic information of the data content can be referred to as data characteristics, which is used to characterize the internal data structure of the target file, such as which symbols are used to separate row data and column data; the key-value pair format; the specific representation of objects and arrays; whether nesting is supported, etc. Since the internal data structures of files of different file formats are different, the file format of the target file can be determined by the characteristics of the data content in the target file.

[0035] Step 203: parse the target file using a parsing method corresponding to the file format to generate target metadata of the target file.

[0036] In this embodiment, since different file formats correspond to different file parsing methods, the execution subject will first determine the parsing method corresponding to the file format, and use the parsing method to parse the content in the target file, thereby generating metadata of the target file based on the content parsing result, that is, target metadata. The metadata obtained by file parsing refers to basic information about the file, which is used to describe the content, structure, creation time, modification time, author, version, etc. of the file.

[0037] The metadata generation method provided by the embodiment of the present disclosure first determines the information of the preset position of the target file; then, in response to the file format of the target file not being determined based on the information, determines the file format of the target file based on the characteristic information of the data content in the target file; finally, the target file is parsed using a parsing method corresponding to the file format to generate target metadata for the target file. The metadata generation method in this embodiment first determines the information of the preset position of the target file. If the file format of the target file cannot be determined based on the information, the file format is determined based on the data characteristics of the target file, and the file format is parsed using a parsing method corresponding to the file format to generate metadata, so that files of any format can be parsed and metadata information of the file can be generated, thereby improving the efficiency and accuracy of metadata generation.

[0038] In addition, in the technical solution of the present disclosure, the acquisition, storage and application of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0039] Continue to refer Figure 3 , Figure 3 A process 300 of another embodiment of a metadata generation method according to the present disclosure is shown. The metadata generation method comprises the following steps:

[0040] Step 301, determining information of a preset location of a target file.

[0041] Step 301 is basically the same as step 201 of the aforementioned embodiment. The specific implementation method can refer to the aforementioned description of step 201, which will not be repeated here.

[0042] Step 302, in response to not determining the file format of the target file based on the information, determining data features of the data content.

[0043] In this embodiment, if the file format of the target file cannot be determined based on the information of the preset location, the execution subject of the metadata generation method (eg Figure 1The server 105 shown) will determine the data features of the data content, which are used to characterize the data structure inside the target file, such as which symbols separate row data and column data respectively; the key-value pair format; the specific representation of objects and arrays; whether nesting is supported, etc.

[0044] In some optional implementations of this embodiment, step 302 includes: reading the data structure inside the target file; analyzing the format of the data structure, and generating data features based on the analysis results.

[0045] In this implementation, the execution subject will first read the data structure inside the target file. The data structure is a format for data organization, management and storage. It is a collection of data elements that have one or more specific relationships with each other. Then, the format of the data structure is analyzed to generate the data features. The data features may specifically include the delimiter of row data, the delimiter of column data, the key-value pair format, whether nesting is supported, the table structure, etc., so as to accurately and quickly generate the data features corresponding to the data inside the target file.

[0046] Step 303: determine a target data feature that matches the data feature from a predefined data feature set.

[0047] In this embodiment, the execution subject will match the data feature with the data features in the predefined data feature set, thereby determining the target data feature that matches the data feature. The predefined data feature set defines the correspondence between various data features and file formats. For example, data feature 1: row data is separated by line breaks, and column data is separated by commas; the file format corresponding to data feature 1 is CSV (Comma-Separated Values). For another example, data feature 2: By label <tag>Organization, supports attributes, file content and nested elements; the file format corresponding to data feature 2 is XML (Extensible Markup Language).

[0048] Step 304: determine the file format based on the target data characteristics.

[0049] In this embodiment, the execution subject determines the file format based on the target data feature. Since the data feature set defines the correspondence between multiple data features and file formats, after determining the target data feature in the data feature set that corresponds to the data feature of the target file, the file format corresponding to the target data feature is used as the file format of the target file.

[0050] Therefore, the file format of the target file can be accurately and quickly determined through the correspondence between the predefined data features and the file format.

[0051] Step 305: determine a target parsing method corresponding to the file format.

[0052] In this embodiment, the execution subject will determine the parsing method corresponding to the file format. Here, the correspondence between different file formats and parsing methods is also predefined, for example, JSON (JavaScript Object Notation) corresponds to parsing method 1, and XML format corresponds to parsing method 2. Therefore, after determining the file format, the parsing method corresponding to the file format can be determined based on the correspondence. For example, if the file format of the target file is determined to be XML, the target parsing method is parsing method 2.

[0053] Step 306: parse the content in the target file using a target parsing method, and generate target metadata based on the content parsing result.

[0054] In this embodiment, the execution subject will parse the content in the target file using the target parsing method, thereby determining the target metadata based on the content parsing result. For example, if the file format of the target file is determined to be XML, the target file will be parsed using parsing method 2 to obtain the target metadata, which may include the content, structure, creation time, modification time, author, version, etc. used to describe the file.

[0055] from Figure 3 It can be seen that Figure 2 Compared with the corresponding embodiments, the metadata generation method in this embodiment determines the file format based on the data characteristics of the target file, and uses the target parsing method corresponding to the file format to parse it to generate metadata, so that files of any format can be parsed and metadata information of the file can be generated, thereby improving the efficiency and accuracy of metadata generation.

[0056] Continue to refer Figure 4 , Figure 4 A process 400 of another embodiment of a metadata generation method according to the present disclosure is shown. The metadata generation method comprises the following steps:

[0057] Step 401, determining information of a preset location of a target file.

[0058] Step 402: In response to the file format of the target file not being determined based on the information, the file format of the target file is determined based on characteristic information of data content in the target file.

[0059] Step 403: parse the target file using a parsing method corresponding to the file format to generate target metadata of the target file.

[0060] Steps 401-403 are basically consistent with steps 201-203 of the aforementioned embodiment. For specific implementation methods, reference may be made to the aforementioned description of steps 201-203, which will not be repeated here.

[0061] Step 404: compare the target metadata with the historical metadata stored in the database.

[0062] In this embodiment, the execution subject of the metadata generation method (eg Figure 1 The server 105 shown in the figure will compare the generated target metadata with the historical metadata stored in the database. Since each time the metadata is generated, it will be added to the metadata table in the database to manage the metadata in a unified manner. Therefore, after the target metadata of the target file is generated, it will be compared with the metadata table containing the historical metadata stored in the database.

[0063] In some optional implementations of this embodiment, step 404 includes: determining a file identifier of the target file; and comparing the target metadata with target historical metadata corresponding to the file identifier in the database.

[0064] In this implementation, the file identifier of the target file is determined first. Since the metadata is stored with the file identifier as identification information, after determining the file identifier of the target file, the execution subject will determine the target historical metadata corresponding to the file identifier in the database, and compare the target metadata with the target historical metadata. Thus, the historical metadata is maintained according to the generated metadata.

[0065] Step 405: Update the historical metadata based on the comparison result.

[0066] In this embodiment, the execution subject will update the historical metadata based on the comparison result, and the update operation may specifically include: adding, deleting or modifying.

[0067] In some optional implementations of this embodiment, step 405 includes: in response to the target metadata being different from the target historical metadata, modifying the target historical metadata using the target metadata.

[0068] In this implementation, if the target metadata is determined to be different from the target historical metadata through comparison, the execution subject will modify the target historical metadata using the target metadata, thereby updating the target historical metadata to the latest data and maintaining the historical metadata.

[0069] In some optional implementations of this embodiment, step 405 includes: in response to the file identifier not being stored in the database, adding the file identifier and the target metadata to the database.

[0070] In this implementation, if it is found through comparison that the file identifier is not stored in the database, it can be determined that the target file and the metadata of the target file are not stored in the database. At this time, the above-mentioned execution entity will add the file identifier and the target metadata to the database, thereby adding new metadata stored in the database, thereby realizing timely updating of historical metadata.

[0071] from Figure 4 It can be seen that Figure 2 Compared with the corresponding embodiment, the metadata generation method in this embodiment will compare the target metadata with the historical metadata stored in the database after generating the target metadata, so as to add or modify the historical metadata based on the comparison result, thereby realizing the maintenance of the historical metadata.

[0072] Continue to refer Figure 5 , Figure 5 A process 500 of another embodiment of a metadata generation method according to the present disclosure is shown. The metadata generation method comprises the following steps:

[0073] Step 501, determining information of a preset location of a target file.

[0074] In this embodiment, the execution subject of the metadata generation method (eg Figure 1 The server 105 shown in the figure will determine the information of the preset location of the target file. Step 501 is basically the same as step 201 in the above embodiment. The specific implementation method can refer to the above description of step 201, which will not be repeated here.

[0075] Step 502, matching the information with the information in the preset information set, and judging whether the file format has been determined based on the matching result.

[0076] In this embodiment, after determining the information of the preset location of the target file, the execution subject will also match the information with the information in the preset information set, and determine whether the file format of the target file can be determined based on the matching result. If the preset information corresponding to the above information is matched from the preset information set, the match is successful, and the file format corresponding to the information is determined as the file format of the target file.

[0077] In some optional implementations of this embodiment, the information includes byte information; and step 502 includes: matching the byte information in the file header of the target file with the byte information in a preset byte information set; in response to matching the target byte information corresponding to the byte information from the byte information set, determining the file format corresponding to the target byte information as the file format of the target file.

[0078] In this implementation, the preset position may be a file header. The file format is actually described by several bytes at the beginning of the file, and these file descriptions are called "file headers". Accordingly, the information at the preset position is the byte information of the file header. After determining the byte information of the file header of the target file, the above-mentioned execution subject will also match the byte information with the byte information in the preset byte information set. If the target byte information corresponding to the byte information is matched from the byte information set, the file format corresponding to the target byte information is determined as the file format of the target file. In this way, the file format of the target file can be accurately and quickly determined through the information of the file header of the target file.

[0079] In some optional implementations of this embodiment, the information includes suffix information; and step 502 includes: matching the suffix information of the file suffix of the target file with the suffix information in a preset suffix information set; in response to matching the target suffix information corresponding to the suffix information from the suffix information set, determining the file format corresponding to the target suffix information as the file format of the target file.

[0080] In this implementation, the preset position may be a file suffix, and a file extension (Filename Extension, or extended file name, suffix name) is a mechanism used by the operating system to mark the file format. Accordingly, the information at the preset position is the suffix information of the file suffix. After determining the suffix information of the file suffix of the target file, the above-mentioned execution subject will also match the suffix information with the suffix information in the preset suffix information set. If the target suffix information corresponding to the suffix information is matched from the suffix information set, the file format corresponding to the target suffix information is determined as the file format of the target file. Thus, the file format of the target file can be accurately and quickly determined through the information of the file suffix of the target file.

[0081] Step 503 , in response to the file format being determined based on the information, the target file is parsed using a parsing method corresponding to the file format to generate target metadata of the target file.

[0082] In this embodiment, the execution subject will parse the target file using a parsing method corresponding to the file format to generate target metadata for the target file. Since different file formats correspond to different file parsing methods, the execution subject will first determine the parsing method corresponding to the file format, and parse the content in the target file using the parsing method, thereby generating metadata for the target file based on the content parsing result, i.e., target metadata. The metadata obtained by file parsing refers to basic information about the file, which is used to describe the content, structure, creation time, modification time, author, version, etc. of the file.

[0083] from Figure 5 It can be seen that Figure 2 Compared with the corresponding embodiments, the metadata generation method in this embodiment determines the file format of the target file through the file header or file suffix of the target file, thereby improving the efficiency of determining the file format and further improving the efficiency of generating file metadata.

[0084] Continue to refer Figure 6 , Figure 6 An application scenario of the metadata generation method according to the present disclosure is shown. In this application scenario, the task is executed through the master-worker (concurrent design) mode. When the task starts, the task status is updated to RUNNING. The Reader / Writer plug-in mode is adopted, which is mainly divided into four stages:

[0085] 1. Preprocessing stage: Get all table names in the database, and then try to read the previous result summary from the specified path, that is, get summary information from the file system, scan the source file directory, split the record stream by table, and put the summary into the context of the record stream. If successful, execute the next stage; if the processing fails, update the task status to FAILED.

[0086] 2. Discovery and parsing phase: Scan the table file path, identify the file type and extract the table metadata. That is, obtain data from the file system, access the file directory, perform file identification and parsing, compare the extracted metadata with historical data, and perform addition, deletion or update operations. If successful, execute the next phase; if the parsing fails, update the task status to PartSucc.

[0087] 3. Metadata synchronization phase: Synchronize the results to the database and persist them, that is, pass the parsing results to metadata. After the synchronization is successful, a confirmation is returned, and the result summary and record flow operation results are appended to the local file. If the synchronization fails, the task status is updated to PartSucc.

[0088] 4. Post-processing stage: collect task status statistics and send them back to the service for storage. That is, update the summary and instance operation results in the file system. If the processing is successful, the task is terminated and the task status is updated to SUCCEED; if it fails, the task status is updated to FAILED.

[0089] Further references Figure 7 As an implementation of the methods shown in the above figures, the present disclosure provides an embodiment of a metadata generation device. Figure 2 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0090] like Figure 7 As shown, the metadata generating device 700 of this embodiment includes: an information determining module 701, a format determining module 702 and a metadata generating module 703. The information determining module 701 is configured to determine the information of the preset position of the target file; the format determining module 702 is configured to determine the file format of the target file based on the characteristic information of the data content in the target file in response to the file format of the target file not being determined based on the information; the metadata generating module 703 is configured to parse the target file in a parsing manner corresponding to the file format, and generate target metadata of the target file.

[0091] In the metadata generating device 700, the specific processing of the information determining module 701, the format determining module 702 and the metadata generating module 703 and the technical effects thereof can be referred to in Figure 2 The relevant descriptions of steps 201 - 203 in the corresponding embodiment are not repeated here.

[0092] In some optional implementations of this embodiment, the metadata generating device 700 further includes: a matching module, configured to match the information with information in a preset information set, and determine whether the file format has been determined based on the matching result; and a parsing module, configured to parse the target file in a parsing method corresponding to the file format in response to the file format being determined based on the information, and generate target metadata for the target file.

[0093] In some optional implementations of this embodiment, the information includes byte information; and the matching module is further configured to: match the byte information in the file header of the target file with the byte information in a preset byte information set; in response to matching the target byte information corresponding to the byte information from the byte information set, determine the file format corresponding to the target byte information as the file format of the target file.

[0094] In some optional implementations of this embodiment, the information includes suffix information; and the matching module is further configured to: match the suffix information of the file suffix of the target file with the suffix information in a preset suffix information set; in response to matching the target suffix information corresponding to the suffix information from the suffix information set, determine the file format corresponding to the target suffix information as the file format of the target file.

[0095] In some optional implementations of this embodiment, the format determination module includes: a first determination submodule, configured to determine data features of data content; a second determination submodule, configured to determine target data features that match the data features from a predefined data feature set; and a third determination submodule, configured to determine the file format based on the target data features.

[0096] In some optional implementations of this embodiment, the first determination submodule is further configured to: read the data structure inside the target file; analyze the format of the data structure, and generate data features based on the analysis results.

[0097] In some optional implementations of this embodiment, the metadata generation module is further configured to: determine a target parsing method corresponding to the file format; parse the content in the target file using the target parsing method, and generate target metadata based on the content parsing result.

[0098] In some optional implementations of this embodiment, the metadata generating device 700 further includes: a comparison module configured to compare the target metadata with the historical metadata stored in the database; and an update module configured to update the historical metadata based on the comparison result.

[0099] In some optional implementations of this embodiment, the comparison module is further configured to: determine a file identifier of the target file; and compare the target metadata with the target historical metadata corresponding to the file identifier in the database.

[0100] In some optional implementations of this embodiment, the update module is further configured to: in response to the target metadata being different from the target historical metadata, modify the target historical metadata using the target metadata.

[0101] In some optional implementations of this embodiment, the update module is further configured to: in response to the file identifier not being stored in the database, add the file identifier and the target metadata to the database.

[0102] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.

[0103] Figure 8 A schematic block diagram of an example electronic device 800 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0104] like Figure 8 As shown, the device 800 includes a computing unit 801, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 802 or a computer program loaded from a storage unit 808 into a random access memory (RAM) 803. In the RAM 803, various programs and data required for the operation of the device 800 can also be stored. The computing unit 801, the ROM 802, and the RAM 803 are connected to each other via a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.

[0105] Multiple components in the device 800 are connected to the I / O interface 805, including: an input unit 806, such as a keyboard, a mouse, etc.; an output unit 807, such as various types of displays, speakers, etc.; a storage unit 808, such as a disk, an optical disk, etc.; and a communication unit 809, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 809 allows the device 800 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0106] The computing unit 801 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 801 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The computing unit 801 performs the various methods and processes described above, such as the metadata generation method. For example, in some embodiments, the metadata generation method may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 808. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 800 via the ROM 802 and / or the communication unit 809. When the computer program is loaded into the RAM 803 and executed by the computing unit 801, one or more steps of the metadata generation method described above may be performed. Alternatively, in other embodiments, the computing unit 801 may be configured to perform the metadata generation method in any other appropriate manner (e.g., by means of firmware).

[0107] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0108] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. The program code may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable device, so that when the program code is executed by the processor or controller, the functions / operations specified in the flow chart and / or block diagram are implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0109] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0110] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0111] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0112] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship with each other. The server may be a cloud server, a server of a distributed system, or a server combined with a blockchain.

[0113] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0114] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.< / tag>

Claims

1. A metadata generation method, comprising: Information for determining a preset location of a target file; In response to not determining the file format of the target file based on the information, determining the file format of the target file based on characteristic information of data content in the target file; The target file is parsed using a parsing method corresponding to the file format to generate target metadata of the target file.

2. The method according to claim 1, further comprising: Matching the information with information in a preset information set, and determining whether the file format has been determined based on the matching result; In response to the file format being determined based on the information, the target file is parsed in a parsing manner corresponding to the file format to generate target metadata of the target file.

3. The method according to claim 2, wherein: The information includes byte information; as well as The step of matching the information with information in a preset information set and judging whether the file format has been determined based on the matching result includes: Matching the byte information in the file header of the target file with the byte information in a preset byte information set; In response to matching the target byte information corresponding to the byte information from the byte information set, the file format corresponding to the target byte information is determined as the file format of the target file.

4. The method according to claim 2, wherein: The information includes suffix information; as well as The step of matching the information with information in a preset information set and judging whether the file format has been determined based on the matching result includes: Matching the suffix information of the file suffix of the target file with the suffix information in a preset suffix information set; In response to matching the target suffix information corresponding to the suffix information from the suffix information set, the file format corresponding to the target suffix information is determined as the file format of the target file.

5. The method according to claim 1, wherein: Determining the file format of the target file based on characteristic information of data content in the target file includes: determining data characteristics of the data content; Determining a target data feature that matches the data feature from a predefined set of data features; The file format is determined based on the target data characteristics.

6. The method according to claim 5, wherein: The determining of the data features of the data content includes: Read the data structure inside the target file; The format of the data structure is analyzed, and the data feature is generated based on the analysis result.

7. The method according to claim 1, wherein: The step of parsing the target file in a parsing manner corresponding to the file format to generate target metadata of the target file includes: Determine a target parsing method corresponding to the file format; The target parsing method is used to parse the content in the target file, and the target metadata is generated based on the content parsing result.

8. The method according to any one of claims 1 to 7, further comprising: comparing the target metadata with historical metadata stored in a database; The historical metadata is updated based on the comparison result.

9. The method according to claim 8, wherein: The comparing the target metadata with the historical metadata stored in the database includes: Determine the file identifier of the target file; The target metadata is compared with the target historical metadata corresponding to the file identifier in the database.

10. The method according to claim 9, wherein: The updating of the historical metadata based on the comparison result includes: In response to the target metadata being different from the target history metadata, the target history metadata is modified using the target metadata.

11. The method according to claim 9, wherein: The updating of the historical metadata based on the comparison result includes: In response to the file identifier not being stored in the database, adding the file identifier and the target metadata to the database.

12. A metadata generating device, comprising: An information determination module, configured to determine information of a preset location of a target file; a format determination module, configured to, in response to not determining the file format of the target file based on the information, determine the file format of the target file based on characteristic information of data content in the target file; The metadata generation module is configured to parse the target file in a parsing manner corresponding to the file format to generate target metadata of the target file.

13. The apparatus according to claim 12, further comprising: a matching module, configured to match the information with information in a preset information set, and determine whether the file format has been determined based on the matching result; The parsing module is configured to parse the target file in a parsing manner corresponding to the file format in response to determining the file format based on the information, and generate target metadata of the target file.

14. The device according to claim 13, wherein: The information includes byte information; and The matching module is further configured to: Matching the byte information in the file header of the target file with the byte information in a preset byte information set; In response to matching the target byte information corresponding to the byte information from the byte information set, the file format corresponding to the target byte information is determined as the file format of the target file.

15. The device according to claim 13, wherein: The information includes suffix information; and The matching module is further configured to: Matching the suffix information of the file suffix of the target file with the suffix information in a preset suffix information set; In response to matching the target suffix information corresponding to the suffix information from the suffix information set, the file format corresponding to the target suffix information is determined as the file format of the target file.

16. The device according to claim 12, wherein: The format determination module comprises: A first determination submodule is configured to determine a data feature of the data content; A second determination submodule is configured to determine a target data feature that matches the data feature from a predefined data feature set; The third determination submodule is configured to determine the file format based on the target data feature.

17. The device according to claim 16, wherein: The first determining submodule is further configured to: Read the data structure inside the target file; The format of the data structure is analyzed, and the data feature is generated based on the analysis result.

18. The device according to claim 12, wherein: The metadata generation module is further configured to: Determine a target parsing method corresponding to the file format; The target parsing method is used to parse the content in the target file, and the target metadata is generated based on the content parsing result.

19. The apparatus according to any one of claims 12 to 18, further comprising: a comparison module, configured to compare the target metadata with historical metadata stored in a database; The updating module is configured to update the historical metadata based on the comparison result.

20. The device according to claim 19, wherein The comparison module is further configured to: Determine the file identifier of the target file; The target metadata is compared with the target historical metadata corresponding to the file identifier in the database.

21. The device according to claim 20, wherein: The update module is further configured to: In response to the target metadata being different from the target history metadata, the target history metadata is modified using the target metadata.

22. The device according to claim 20, wherein: The update module is further configured to: In response to the file identifier not being stored in the database, adding the file identifier and the target metadata to the database.

23. An electronic device, comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 11.

24. A non-transitory computer-readable storage medium storing computer instructions, the computer instructions being used to cause the computer to execute the method of any one of claims 1-11.

25. A computer program product comprising a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 11.

Images