Metadata scanning method and device, equipment, storage medium and program product

Abstract

The invention is suitable for the technical field of computers, and provides a metadata scanning method, device and equipment and a readable storage medium, the method comprises the steps that a scanning request is received, the scanning request comprises a target problem type and a metadata identifier, and the target problem type comprises a metadata record error type; searching N pieces of target metadata corresponding to the metadata identifier; and performing problem scanning on the N pieces of target metadata according to a target scanning scheme corresponding to the target problem type to obtain a scanning result of the N pieces of target metadata. Therefore, the metadata problem scanning efficiency can be improved by automatically performing problem scanning on the metadata.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of computers, and particularly relates to a metadata scanning method and device, equipment, a storage medium and a program product. BACKGROUND

[0002] Metadata is data used to describe data, and is mainly used to summarize and explain the characteristics, sources and locations of other data.

[0003] Currently, for any metadata corresponding to data, there may be content errors, content omissions and the like, and relevant staff can find the problems in the foregoing metadata through manual means. However, this problem checking method is low in efficiency and consumes time and labor costs. SUMMARY

[0004] The metadata scanning method, device, equipment, storage medium and program product provided by the embodiments of the present application can improve the efficiency and time of metadata problem scanning as much as possible.

[0005] In a first aspect, the embodiments of the present application provide a metadata scanning method, comprising:

[0006] receiving a scanning request, the scanning request comprising a target problem type and a metadata identifier, the target problem type comprising a metadata record error type; finding N target metadata corresponding to the metadata identifier, N being a positive integer greater than or equal to 1; performing problem scanning on the N target metadata according to a target scanning scheme corresponding to the target problem type to obtain scanning results of the N target metadata.

[0007] In a possible implementation manner, finding N target metadata corresponding to the metadata identifier comprises: loading N original metadata corresponding to the metadata identifier, each of the N original metadata comprising a plurality of sub-metadata; merging the plurality of sub-metadata corresponding to each original metadata to obtain the N target metadata.

[0008] In a possible implementation manner, the plurality of sub-metadata are stored in an inheritance tree, the inheritance tree comprising a plurality of levels, the plurality of levels comprising a top level, a next top level and levels lower than the next top level, and the plurality of sub-metadata being distributed on the plurality of levels; merging the plurality of sub-metadata corresponding to each original metadata to obtain the N target metadata comprises: for any one original metadata, taking a first sub-metadata of the top level in the plurality of levels as a bottom template, and on the basis of the bottom template, sequentially merging sub-metadata of levels other than the top level in the plurality of levels in descending order of the levels to obtain a target metadata.

[0009] In a possible implementation, in the plurality of sub-metadatas, the sub-metadatas in the top sub-level and the sub-levels below the top sub-level are associated with data tags, a first sub-metadata in the top sub-level of the plurality of sub-levels is taken as a bottom template, and the sub-metadatas in the sub-levels other than the top sub-level in the plurality of sub-levels are sequentially merged in a descending order of the sub-levels based on the bottom template to obtain target metadata, which includes:

[0010] A first sub-metadata in the top sub-level of the plurality of sub-levels is taken as a bottom template, and the sub-metadatas in the sub-levels other than the top sub-level in the plurality of sub-levels are sequentially merged in a descending order of the sub-levels based on the bottom template. In the process of merging, if a data tag of a second sub-metadata in a first sub-level is new addition, the second sub-metadata is added to a first metadata to obtain target metadata, the first sub-level is any one of the sub-levels other than the top sub-level in the plurality of sub-levels, and the first metadata is metadata obtained by merging before the current time; if the data tag of the second sub-metadata in the first sub-level is deletion, the second sub-metadata in the first metadata is deleted to obtain target metadata; if the data tag of the second sub-metadata in the first sub-level is modification, a sub-metadata corresponding to the second sub-metadata in the first metadata is replaced by the second sub-metadata to obtain target metadata.

[0011] In a possible implementation, after the plurality of sub-metadatas corresponding to each original metadata are merged to obtain N target metadatas, the method further includes:

[0012] The N target metadatas are divided to obtain M groups of target metadatas, and M is less than N. The N target metadatas are scanned according to a target scanning scheme corresponding to a target problem type to obtain scanning results of the N target metadatas, including: creating M threads through thread management; and scanning the M groups of target metadatas according to the target scanning scheme corresponding to the target problem type through the M threads to obtain M groups of scanning results.

[0013] In a possible implementation, after the N target metadatas are scanned according to the target scanning scheme corresponding to the target problem type through the M threads to obtain the M groups of scanning results, the method further includes:

[0014] The M groups of scanning results are written in the form of rows or columns into a data table through the M threads, and each row or each column of the data table is locked by using a thread lock in the process of writing to obtain a first summary table, the first summary table including M rows of scanning results or M columns of scanning results; and the first summary table is written into a local database.

[0015] In a possible implementation, the N target metadatas are divided to obtain M groups of target metadatas, including:

[0016] The N target metadata are filtered to obtain P target metadata, the P target metadata include metadata related to the target problem type in the N target metadata, and P is less than or equal to N; the P target metadata are divided to obtain M groups of target metadata.

[0017] In a possible implementation, the target problem type is any one of a plurality of problem types, the plurality of problem types include metadata content error, metadata dependency absence, and metadata definition inconsistency with an actual table definition; the target metadata are scanned according to a target scanning scheme corresponding to the target problem type to obtain scanning results of the N target metadata, including:

[0018] The target scanning scheme corresponding to the target problem type is filtered from a scanning scheme set corresponding to the plurality of problem types; and the N target metadata are scanned according to the target scanning scheme to obtain scanning results of the N target metadata.

[0019] In a second aspect, an embodiment of the present application provides a metadata scanning device, including:

[0020] A receiving module is configured to receive a scanning request, the scanning request including a target problem type and a metadata identifier, and the target problem type including a metadata record error type.

[0021] A processing module is configured to find N target metadata corresponding to the metadata identifier, N being a positive integer greater than or equal to 1.

[0022] The processing module is further configured to scan the N target metadata according to a target scanning scheme corresponding to the target problem type to obtain scanning results of the N target metadata.

[0023] In a third aspect, an embodiment of the present application provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor implements the method in any one of the first aspect when executing the computer program.

[0024] In a fourth aspect, an embodiment of the present application provides a computer readable storage medium, the computer readable storage medium storing a computer program, and the computer program is executable on a processor to implement the method in any one of the first aspect.

[0025] In a fifth aspect, an embodiment of the present application provides a computer program product, when the computer program product is executed on a computer device, the computer program product causes the computer device to execute the method in any one of the first aspect.

[0026] It can be understood that the beneficial effects of the second aspect to the fifth aspect can be referred to the related description in the first aspect, and will not be repeated here.

[0027] The beneficial effects of the embodiments of the present application compared with the prior art are:

[0028] The scanning request includes a target problem type and a metadata identifier. When receiving the scanning request, N target metadata corresponding to the metadata identifier can be searched, and problem scanning can be performed on the N target metadata according to a target scanning scheme corresponding to the target problem type, to obtain scanning results of the N target metadata. In this way, automatic problem scanning of target metadata can be implemented, so that the scanning results of the target metadata in a specified scene can be quickly obtained, and the problem that metadata is difficult to traverse completely and hidden problems of metadata are difficult to find in manual detection of a scene can be avoided, so that human resources and time cost can be saved. BRIEF DESCRIPTION OF DRAWINGS

[0029] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed to be used in the embodiments or prior art description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0030] Figure 1 is a flowchart of a metadata scanning method provided by an embodiment of the present application;

[0031] Figure 2 is a flowchart of a metadata scanning method provided by an embodiment of the present application;

[0032] Figure 3 is a schematic diagram of scanning schemes corresponding to different problem types provided by an embodiment of the present application;

[0033] Figure 4 is a flowchart of a metadata scanning method provided by an embodiment of the present application;

[0034] Figure 5 is a metadata merging diagram in an inheritance tree provided by an embodiment of the present application;

[0035] Figure 6 is a flowchart of a metadata scanning method provided by an embodiment of the present application;

[0036] Figure 7 is a schematic diagram of a metadata screening method provided by an embodiment of the present application;

[0037] Figure 8 is a structural schematic diagram of a metadata scanning device provided by an embodiment of the present application;

[0038] Figure 9Fig. 1 is a structural schematic diagram of a computer device according to an embodiment of the present application. DETAILED DESCRIPTION

[0039] In the following description, for purposes of explanation and not limitation, specific details are set forth, such as particular architectures, techniques, etc., in order to provide a thorough understanding of the embodiments of the present application. However, it will be apparent to those skilled in the art that the present application can be practiced in other embodiments that depart from these specific details. In other instances, detailed descriptions of well-known methods, devices, circuits, and

[0040] It will be understood that the terms "comprises" and / or "comprising," when used in this specification, specify the presence of stated features, integers, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0041] It is also to be understood that the term "and / or" as used herein refers to and encompasses any and all possible combinations of one or more of the associated listed items, and that the term "at least one of" followed by a list of two or more items means any single one of the listed items individually, as well as any combination of two or more of the listed items.

[0042] As used in the description of the application and the appended claims, the term "if' can be interpreted to mean "when" or "upon" or "in response to determining" or "in response to detecting" depending on the context. Similarly, the phrase "if it is determined" or "if [a described condition or event] is detected" can be interpreted to mean "upon determining" or "in response to determining" or "upon [the described condition or event] being detected" or "in response to [the described condition or event] being detected", depending on the context.

[0043] In addition, the terms "first", "second", "third", etc. as used in the description of the application and the appended claims are not used to denote or imply relative importance but are used to distinguish one element from another.

[0044] Reference throughout this specification to "one embodiment" or "an embodiment" means that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment of the present application. Thus, the appearances of the phrases "in one embodiment" or "in an embodiment" or "in some embodiments" or "in other embodiments" or "in still other embodiments", etc. in various places throughout this specification are not necessarily all referring to the same embodiment, unless otherwise specifically so stated. Furthermore, the term "comprising" or "containing" or "including" as used herein is specifically intended to mean "including, but not limited to".

[0045] For ease of understanding, the examples provided are for reference only and are related to the concepts in the embodiments of this application.

[0046] 1. Metadata

[0047] Metadata is data used to describe data. Metadata is an XML data body that includes information such as entity object styles, attributes, and rule validators. For example, the metadata of a file includes its name, size, and creation time; the metadata of a photo includes the shooting time, camera model, geographical location (latitude and longitude), aperture value, and shutter speed. For example, the metadata corresponding to a table definition includes the database table name, the field type of a certain field, and the field length.

[0048] 2. Differential storage mode

[0049] In differential storage mode, metadata is not stored in its entirety every time. Typically, only the parts that have changed compared to a certain "baseline" or "previous version" are stored.

[0050] For example, metadata is <person> <name>Zhang San< / name> <age> 30< / age> < / person> This can serve as a baseline, indicating that Zhang San's age is 30. If the age changes from 30 to 31, the differential storage mode only records... <age> 31< / age> (And include information explaining which part of which reference record this modification was made to).

[0051] 3. Inheritance tree

[0052] An inheritance tree is a hierarchical structure that can include multiple levels, each with at least one node. Multiple child metadata elements of a complete metadata set can be distributed across multiple nodes in the inheritance tree. Similar to a family tree, it has parent nodes (parent class metadata) and child nodes (child class metadata). Child nodes inherit some properties from their parent nodes, but can also have their own unique properties or modify the properties of their parent nodes.

[0053] This application provides a metadata scanning method, apparatus, device, storage medium, and program product. The method is applicable to various scenarios that require metadata problem scanning.

[0054] The metadata scanning method can be executed by a server. The metadata scanning method can also be executed by a terminal device, which can be a mobile phone, a tablet computer, a wearable device, a vehicle-mounted device, an augmented reality (AR) / virtual reality (VR) device, a notebook computer, an ultra-mobile personal computer (UMPC), a netbook, a personal digital assistant (PDA), or the like. The specific type of the terminal device is not limited in the embodiments of the present application.

[0055] The metadata scanning method is executed by the server when a scanning request is received. The scanning request can be sent by the control device or sent by the control device to a proxy server and then sent to the server by the proxy server.

[0056] It should be noted that the proxy server and the server executing the metadata scanning method can be different servers or the same server. When the proxy server and the server executing the metadata scanning method are the same server, the proxy server and the server executing the metadata scanning method are two service modules in the same server.

[0057] As shown in FIG. 1, the user can perform an operation of selecting or filling in related parameters on the control device, so that the control device can initiate scanning, that is, send a scanning request. The related parameters include a target question type and a metadata identifier, which will be described below and will not be described here in detail. Optionally, the related parameters can also include an account and a password (for logging into the server) for establishing a link with the server. Optionally, the control device can encapsulate the related parameters by a Kscript script and then initiate the scanning request. The scanning request can carry the related parameters. Figure 1 In some embodiments, the scanning request is a uniform resource locator (URL) request.

[0058] The proxy server can be referred to as an Nginx proxy, which can route the scanning request to the server.

[0059]

[0060] ​After receiving the query request, the server can perform the scanning scheme and store the scanning result to the local database. Specifically, the server can perform original metadata caching, metadata merging on the original metadata, metadata standardization on the merged metadata, i.e., converting the metadata from an extensible markup language (XML) format or a json format to a document object model (DOM) format, metadata parsing and extraction from the metadata to extract part of the data related to the problem scanning, and finally creating multiple threads through thread management, performing the scanning scheme on the part of the data through the multiple threads, and summarizing and storing (to the local database) the scanning result obtained through the scanning.

[0061] Based on the above scenario description, the metadata scanning method provided by the embodiments of the present application will be described in detail below, taking the server as the first device, the control device as the second device, and the proxy server as the third device as an example.

[0062] Please refer to Figure 2 , Figure 2 The flowchart of the metadata scanning method provided by an embodiment of the present application is shown. As shown in Figure 2 , the metadata scanning method provided by the present application can include:

[0063] S101, receiving a scanning request, the scanning request including a target problem type and a metadata identifier, the target problem type including a metadata record error type, and the scanning request being used to request problem scanning of the metadata corresponding to the metadata identifier in the target problem type.

[0064] The scanning request can be sent directly by the second device to the first device, or sent by the second device to the first device through the third device.

[0065] In some embodiments, if the scanning request is sent directly by the second device to the first device, the second device includes a target interface, and the target interface is used to fill in the metadata identifier. Thus, when the metadata identifier is received, the second device can trigger the sending of the scanning request.

[0066] In some embodiments, the target problem type is any one of a plurality of problem types, and the plurality of problem types include a metadata content error, a metadata dependency absence, and a metadata definition inconsistency with an actual table definition.

[0067] Based on this, the target interface can be an interface corresponding to a target problem type in a plurality of interfaces, and the plurality of interfaces correspond to a plurality of problem types in a one-to-one manner. Specifically, the second device can include an interface related to metadata content errors, an interface related to metadata dependency correctness, and an interface related to inconsistency between a definition of a metadata table and an actual table definition. For example, the target interface can be an interface related to metadata content errors.

[0068] In some embodiments, the target interface can include a search box in which a metadata identifier can be filled in, and a search control for triggering sending of a scanning request. Thus, when the second device receives the metadata identifier in the search box and an operation on the search control, the second device can trigger sending of the scanning request.

[0069] The metadata identifier can be used to indicate metadata of data of a plurality of application programs.

[0070] In some embodiments, the metadata identifier can include an identifier of an application program. For example, the metadata identifier can include a name of an application program 1, a name of an application program 2, and a name of an application program 3. If the metadata identifier includes the name of the application program 1, the name of the application program 2, and the name of the application program 3, the metadata corresponding to the metadata identifier includes metadata of data related to the application program 1, metadata of data related to the application program 2, and metadata of data related to the application program 3.

[0071] When the plurality of problem types include metadata content errors, a lack of dependency between metadata, and inconsistency between a table definition of metadata and a preset table definition, the target problem type can be metadata content errors, a lack of dependency between metadata, or inconsistency between a table definition of metadata and a preset table definition.

[0072] For metadata content errors: The metadata content errors can include a lack of keywords, spelling errors, sensitive words, and the like.

[0073] For example, a metadata includes time and date, and the metadata content errors can include spelling the English word time as tim and spelling the English word date as dete.

[0074] For a lack of dependency between metadata: There can be a certain dependency relationship between different metadata. The lack of dependency between metadata includes not correctly recording the dependency relationship between metadata.

[0075] The definition of the metadata pair table is inconsistent with the actual table definition: it can be understood that the name of the field, the type of the field, and the length of the field in the metadata record of the database table are inconsistent with the name of the field, the type of the field, and the length of the field in the actual database table in the database. For example, the metadata defines that the length of a field is 50, but the length of the field in the actual database table is 100.

[0076] S102, find N target metadata corresponding to the metadata identifier, N is a positive integer greater than or equal to 1.

[0077] Since the metadata identifier is included in the scan request, after receiving the scan request, the server can search in all metadata included in the server to obtain N target metadata corresponding to the metadata identifier.

[0078] For example, the metadata identifier is the name of application 1, and then the N target metadata includes all metadata corresponding to application 1 in the server.

[0079] S103, according to the target scanning scheme corresponding to the target problem type, the N target metadata is scanned for problem scanning, and the scanning result of the N target metadata is obtained.

[0080] The target scanning scheme corresponding to the target problem type can be stored in the first device, and after the first device finds the N target metadata, the N target metadata can be scanned for problem according to the target scanning scheme.

[0081] In some embodiments, the target problem type is any one of a plurality of problem types, and the plurality of problem types include metadata content error, metadata dependency missing, and metadata pair table definition inconsistent with actual table definition; the first device includes a scanning scheme set corresponding to a plurality of problem types, and the scanning scheme set includes a plurality of scanning schemes corresponding to the plurality of problem types one by one.

[0082] Based on this, in some embodiments, according to the target scanning scheme corresponding to the target problem type, the target metadata is scanned for problem, and the scanning result of the N target metadata is obtained, including: screening the target scanning scheme corresponding to the target problem type from the scanning scheme set corresponding to the plurality of problem types; according to the target scanning scheme, the N target metadata is scanned for problem, and the scanning result of the N target metadata is obtained.

[0083] If the plurality of problem types include metadata content error, metadata dependency missing, and metadata pair table definition inconsistent with actual table definition, the plurality of scanning schemes include a scanning scheme corresponding to metadata content error, a scanning scheme corresponding to metadata dependency missing, and a scanning scheme corresponding to metadata pair table definition inconsistent with actual table definition, as shown in Figure 3 .

[0084] The scanning scheme corresponding to the metadata content error can include: matching the error text keyword with the N target metadata, determining whether the error text keyword exists in the N target metadata, and obtaining the scanning result of the N target metadata.

[0085] For example, the target metadata A includes time and date, and the scanning result includes: there is an error, and the error is that the English word "time" is incorrectly spelled as "tim", and the English word "date" is incorrectly spelled as "dete".

[0086] The scanning scheme corresponding to the missing dependency between metadata includes a scanning scheme corresponding to missing basic information and a scanning scheme corresponding to missing rule reference entity described by metadata. The scanning scheme corresponding to missing basic information includes: traversing the N target metadata, determining whether the basic information is referenced in each target metadata, and if the basic information is referenced, determining whether the referenced basic information exists; for example, the existence of a field A needs to depend on another table B (the table B is the basic information of the field A), the field A references the table B, but the table B does not exist. The scanning scheme corresponding to missing rule reference entity described by metadata includes: traversing the N target metadata, determining whether each target metadata includes a business rule (for example, an approval rule, when the order amount is greater than 1000, the order needs to be approved), and if the business rule is included, determining whether the entity or field (such as order amount) referenced in the business rule exists.

[0087] The scanning scheme corresponding to the inconsistency between the metadata definition of the table and the actual table definition includes: traversing the N target metadata, and determining whether the metadata corresponding to the table definition is consistent with the actual table definition. For example, the target metadata A describes that the length of a certain field is 50, but the actual length of the field is 100.

[0088] In the embodiment of the application, the scanning request includes a target problem type and a metadata identifier. When receiving the scanning request, the first device can find the N target metadata corresponding to the metadata identifier, and perform problem scanning on the N target metadata according to the target scanning scheme corresponding to the target problem type, to obtain the scanning result of the N target metadata. In this way, the automatic problem scanning of the target metadata in the first device can be realized, so that the scanning result of the target metadata in a specified scene can be quickly obtained, and the problem that the metadata is difficult to be fully traversed and hidden in the manual detection scene can be avoided, thereby saving human resources and time cost.

[0089] Based on Figure 2In the description of the illustrated embodiments, the metadata found by the first device is original metadata, which can have problems such as incomplete information and inconsistent logic, and therefore, the plurality of sub-metadatas included in the original metadata need to be merged and then scanned for problems.

[0090] In some embodiments, finding the metadata identifies the corresponding N target metadatas includes: loading the metadata to identify the corresponding N original metadatas, each of the N original metadatas including a plurality of sub-metadatas; and merging the plurality of sub-metadatas corresponding to each of the original metadatas to obtain the N target metadatas.

[0091] The plurality of sub-metadatas are stored in a differential storage mode, specifically, in a manner of an inheritance tree, the inheritance tree including a plurality of levels, the plurality of levels including a top level, a next top level, and levels lower than the next top level, and the plurality of sub-metadatas being distributed on the plurality of levels; therefore, in some embodiments, merging the plurality of sub-metadatas corresponding to each of the original metadatas to obtain the N target metadatas includes: for any one of the original metadatas, taking a first sub-metadata of the top level in the plurality of levels as a bottom template, and on the basis of the bottom template, in order from high to low, merging sub-metadatas of levels other than the top level in the plurality of levels to obtain the target metadata.

[0092] Next, the specific implementation process of the metadata scanning method of the present application will be described in detail in combination with Figure 4 .

[0093] Please refer to Figure 4 , Figure 4 Fig. 1 shows a flowchart of a metadata scanning method provided by an embodiment of the present application. As shown in Figure 4 , the metadata scanning method provided by the present application includes:

[0094] S201, loading metadata to identify the corresponding N original metadatas, each of the N original metadatas including a plurality of sub-metadatas.

[0095] Generally, the metadata identifying the corresponding N original metadatas is usually stored in the local database of the first device, and the first device can pre-load the metadata identifying the corresponding N original metadatas from the database to the cache, so that the metadata is uniformly loaded through the cache, which can reduce the time cost of repeated reading of the original metadata and improve the efficiency and response speed of the subsequent problem scanning process.

[0096] Wherein, the number of sub-metadatas included in each of the N original metadatas can be the same or different, and the present application does not limit it.

[0097] For example, the metadata identifies that there are three original metadata (N=3), namely original metadata 1, original metadata 2 and original metadata 3, the original metadata 1 includes 100 sub-metadatas, the original metadata 2 includes 200 sub-metadatas, and the original metadata 3 includes 50 sub-metadatas.

[0098] S202, for any one original metadata, taking the first sub-metadata of the top layer in the multiple layers as a bottom template, and on the basis of the bottom template, sequentially merging the sub-metadatas of other layers in the multiple layers except the top layer in the order from high to low, to obtain the target metadata.

[0099] For example, the inheritance tree corresponding to the original metadata A includes three layers, namely the top layer, the middle layer and the end layer, the original metadata A includes the sub-metadata of the top layer (which can be used as a bottom template), the middle metadata of the middle layer and the end metadata of the end layer, and the first device can first merge the bottom template and the middle metadata, and then merge the obtained metadata and the end metadata to obtain the target metadata.

[0100] Among the multiple sub-metadatas, the sub-metadatas in the sub-top layer and the layers below the sub-top layer are associated with data tags. In combination with the above example and Figure 5 In addition to the bottom template of the top layer, other metadata such as the middle metadata recorded by the middle layer node and the end metadata recorded by the end layer node in the inheritance tree carry data tags, which may indicate that the data of the layer is newly added based on the sub-metadata of the upper layer, or may indicate that the data of the layer is modified based on the sub-metadata of the upper layer, or may indicate that the data of the layer is deleted from the sub-metadata of the upper layer, therefore, the specific merging mode needs to be determined according to the data tags of the middle layer or the end layer.

[0101] Based on this, in some embodiments, taking the first sub-metadata of the top layer in the multiple layers as a bottom template, and on the basis of the bottom template, sequentially merging the sub-metadatas of other layers in the multiple layers except the top layer in the order from high to low, to obtain the target metadata, including:

[0102] Step 2021, taking the first sub-metadata of the top layer in the multiple layers as a bottom template, and on the basis of the bottom template, sequentially merging the sub-metadatas of other layers in the multiple layers except the top layer in the order from high to low.

[0103] Step 2022, in the process of merging, if the data tag of the second sub-metadata of the first layer is new, the second sub-metadata is added on the basis of the first metadata to obtain the target metadata, the first layer is any one of the other layers in the multiple layers except the top layer, and the first metadata is the metadata obtained by merging before the current time.

[0104] For example, the inheritance tree corresponding to the original metadata A includes three layers, which are a top layer, an intermediate layer and a bottom layer. The original metadata A includes sub metadata of the top layer (which can be used as a bottom template), intermediate metadata of the intermediate layer and bottom layer metadata of the bottom layer. If the first layer is the bottom layer, the second sub metadata is the bottom layer metadata and the data tag of the bottom layer metadata is addition, the first device can first merge the bottom template and the intermediate metadata to obtain first metadata, and then merge the first metadata and the bottom layer metadata to obtain target metadata.

[0105] In step 2023, if the data tag of the second sub metadata of the first layer is deletion, the second sub metadata in the first metadata is deleted to obtain the target metadata.

[0106] For example, the inheritance tree corresponding to the original metadata A includes three layers, which are a top layer, an intermediate layer and a bottom layer. The original metadata A includes sub metadata of the top layer (which can be used as a bottom template), intermediate metadata of the intermediate layer and bottom layer metadata of the bottom layer. If the first layer is the bottom layer, the second sub metadata is the bottom layer metadata and the data tag of the bottom layer metadata is deletion, the first device can first merge the bottom template and the intermediate metadata to obtain first metadata, and then delete the second sub metadata in the first metadata to obtain target metadata.

[0107] In step 2024, if the data tag of the second sub metadata of the first layer is modification, the sub metadata corresponding to the second sub metadata in the first metadata is replaced by the second sub metadata to obtain the target metadata.

[0108] For example, the inheritance tree corresponding to the original metadata A includes three layers, which are a top layer, an intermediate layer and a bottom layer. The original metadata A includes sub metadata of the top layer (which can be used as a bottom template), intermediate metadata of the intermediate layer and bottom layer metadata of the bottom layer. If the first layer is the bottom layer, the second sub metadata is the bottom layer metadata and the data tag of the bottom layer metadata is modification, the first device can first merge the bottom template and the intermediate metadata to obtain first metadata, and then replace the sub metadata corresponding to the second sub metadata in the first metadata with the second sub metadata to obtain target metadata.

[0109] In this application, considering that the sub metadata of any one original metadata is distributed in different inheritance tree layers, a single sub metadata may not be able to provide complete description information. If direct problem scanning is performed, it is difficult to find hidden problems in subsequent problem scanning. Therefore, after the first device loads the metadata identifier corresponding to the N original metadata, the plurality of sub metadata included in any one original metadata can be merged to obtain complete target metadata. In this way, subsequent accurate problem scanning of the target metadata can be facilitated, and the problem of missing scanning in the problem scanning process can be avoided.

[0110] Based on Figure 4 In the description of the embodiment, after the first device merges the plurality of sub-metadatas corresponding to each original metadata to obtain the N target metadatas, the first device can further divide the N target metadatas to obtain M groups of target metadatas, so as to facilitate subsequent problem scanning on the M groups of target metadatas by using M threads simultaneously, so as to increase the efficiency of problem scanning.

[0111] In some embodiments, dividing the N target metadatas to obtain the M groups of target metadatas includes: screening the N target metadatas to obtain P target metadatas, the P target metadatas including target metadatas related to a target problem type in the N target metadatas, and P is less than or equal to N; and dividing the P target metadatas to obtain the M groups of target metadatas.

[0112] Next, the specific implementation process of the metadata scanning method of the present application will be described in detail in combination with Figure 6 .

[0113] Please refer to Figure 6 , Figure 6 Fig. 1 shows a flowchart of a metadata scanning method provided by an embodiment of the present application. As shown in Fig. 1, the metadata scanning method provided by the present application includes the following steps. Figure 6

[0114] S301, screening the N target metadatas to obtain P target metadatas, the P target metadatas including target metadatas related to a target problem type in the N target metadatas.

[0115] If the target problem type is metadata content error, the P target metadatas include the N target metadatas, that is, the value of P is the same as the value of N, and the P target metadatas include all target metadatas corresponding to the metadata identifier.

[0116] If the target problem type is metadata dependency missing, the P target metadatas include target metadatas that refer to target information in the N target metadatas, and the target information includes basic information and metadata rules.

[0117] If the target problem type is that the table definition of the metadata is inconsistent with the actual table definition, the P target metadatas include the N target metadatas, that is, the value of P is the same as the value of N, and the P target metadatas include all target metadatas corresponding to the metadata identifier.

[0118] ​It should be noted that the structure of the N target metadata can be in XML format or in json format, and the first device needs to standardize the N target metadata first, that is, convert the target metadata in XML format into DOM format, and convert the target metadata in json format into standard json format, and then perform screening. The screening method can be the above method, that is, the general extraction method, or other custom methods, or a combination of the above method and the custom method, such as shown in Figure 7 The present application does not limit this.

[0119] S302, divide the P target metadata to obtain M groups of target metadata, and M is less than N.

[0120] For example, the value of P is 1000, and the value of M is 10, that is, 1000 target metadata are divided into 10 groups of target metadata, and each group includes 100 target metadata.

[0121] For example, the value of P is 800, and the value of M is 10, that is, 800 target metadata are divided into 10 groups of target metadata, and each group includes 80 target metadata.

[0122] S303, create M threads through thread management.

[0123] The first device can determine the number of threads to be created according to the number of groups of target metadata, that is, M groups of target metadata can correspond to M threads.

[0124] For example, the value of M is 10, and the first device creates 10 threads through thread management, and each thread is used to process a group of target metadata.

[0125] S304, through the M threads, according to the target scanning scheme corresponding to the target problem type, the M groups of target metadata are scanned for problems to obtain M groups of scanning results.

[0126] If the target problem type is metadata content error, then the M threads scan the M groups of target metadata for problems according to the scanning scheme corresponding to the metadata content error to obtain M groups of scanning results.

[0127] If the target problem type is metadata dependency missing, then the M threads scan the M groups of target metadata for problems according to the scanning scheme corresponding to the metadata dependency missing to obtain M groups of scanning results.

[0128] If the target type is inconsistent with the actual table definition, the M threads perform problem scanning on the M groups of target metadata according to the scanning scheme corresponding to the inconsistency between the metadata table definition and the actual table definition, to obtain M groups of scanning results.

[0129] In S305, the M threads write the M groups of scanning results into the data table in the form of rows or columns, and lock each row or column of the data table using a thread lock during the writing process, to obtain a first summary table including M rows of scanning results or M columns of scanning results.

[0130] If the data table is written in the form of rows, any one of the M threads can write a group of scanning results obtained by the thread as a row, so that the M groups of scanning results are all written into the data table to obtain the first summary table including M rows of scanning results. Similarly, if the data table is written in the form of columns, the first device obtains the first summary table including M columns of scanning results.

[0131] The first device writes the M groups of scanning results simultaneously through the M threads, and the scanning results corresponding to different threads may be written into the same row or column, so that a thread lock is used to lock each row or column of the data table during the writing process, to ensure that the data writing of the same row or column does not compete, for example, when a thread is writing a row, other threads cannot simultaneously write the row, so as to ensure the consistency and integrity of the data.

[0132] Specifically, if the data table is written in the form of rows, a row lock is used, and when a thread needs to write its scanning results into a row of the data table, the lock of the row needs to be acquired. After the lock is acquired, the thread exclusively has the right to write the row, and other threads cannot simultaneously write the row until the lock is released.

[0133] If the data table is written in the form of rows, a column lock is used, and when a thread needs to write its scanning results into a column of the data table, the lock of the column needs to be acquired. After the lock is acquired, the thread exclusively has the right to write the column, and other threads cannot simultaneously write the column until the lock is released.

[0134] In S306, the first summary table is written into the local database of the first device.

[0135] In the embodiment, after the first device merges the multiple sub-metadatas corresponding to each original metadata to obtain N target metadatas, the N target metadatas can be further screened to obtain P target metadatas related to the target problem type, and then the problem scanning operation is performed, so that the data amount of problem scanning can be reduced, and the efficiency and accuracy of problem scanning can be improved.

[0136] And, before performing the problem scanning operation, the first device can also divide the P target metadata to obtain M groups of target metadata, and then perform problem scanning on the M groups of target metadata simultaneously through M threads, so that the efficiency of problem scanning can be increased and the time for performing problem scanning can be saved.

[0137] It should be understood that the size of the serial number of each step in the above embodiments does not mean the order of execution, and the execution order of each process should be determined according to its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0138] According to the metadata scanning method described in the above embodiments, Figure 8 A structure block diagram of the metadata scanning apparatus 400 provided by the embodiments of the present application is shown, and only the parts related to the embodiments of the present application are shown for the convenience of description.

[0139] With reference to Figure 8 The metadata scanning apparatus 400 includes:

[0140] The receiving module 401 is configured to receive a scanning request, the scanning request including a target problem type and a metadata identifier, and the target problem type including a metadata record error type.

[0141] The processing module 402 is configured to find N target metadata corresponding to the metadata identifier, N being a positive integer greater than or equal to 1.

[0142] The processing module 402 is further configured to perform problem scanning on the N target metadata according to a target scanning scheme corresponding to the target problem type, to obtain scanning results of the N target metadata.

[0143] In some embodiments, the processing module 402 is specifically configured to:

[0144] Load N original metadata corresponding to the metadata identifier, each of the N original metadata including a plurality of sub metadata; and merge the plurality of sub metadata corresponding to each original metadata to obtain the N target metadata.

[0145] In some embodiments, the plurality of sub metadata are stored in the form of an inheritance tree, the inheritance tree including a plurality of levels, the plurality of levels including a top level, a sub-top level, and levels lower than the sub-top level, and the plurality of sub metadata being distributed on the plurality of levels; and the processing module 402 is specifically configured to:

[0146] For any one original metadata, take a first sub metadata of the top level in the plurality of levels as a bottom template, and on the basis of the bottom template, sequentially merge sub metadata of levels other than the top level in the plurality of levels in order from high to low to obtain the target metadata.

[0147] In some embodiments, the sub-metadatas in the plurality of sub-metadatas in the top sub-metadatas and the sub-metadatas in the levels lower than the top sub-metadatas are associated with data tags, and the processing module 402 is specifically configured to:

[0148] taking the first sub-metadatas in the top level of the plurality of levels as a bottom template, and sequentially merging the sub-metadatas in the levels other than the top level in the plurality of levels in the order from high to low on the basis of the bottom template; in the process of merging, if the data tag of the second sub-metadatas in the first level is new addition, the second sub-metadatas are added on the basis of the first metadatas to obtain target metadatas, the first level is any one of the levels other than the top level in the plurality of levels, and the first metadatas are the metadatas obtained by merging before the current time; if the data tag of the second sub-metadatas in the first level is deletion, the second sub-metadatas in the first metadatas are deleted to obtain target metadatas; if the data tag of the second sub-metadatas in the first level is modification, the sub-metadatas corresponding to the second sub-metadatas in the first metadatas are replaced by the second sub-metadatas to obtain target metadatas.

[0149] In some embodiments, the processing module 402 is specifically configured to:

[0150] The N target metadatas are divided to obtain M groups of target metadatas, and M is less than N.

[0151] M threads are created through thread management; and M groups of target metadatas are scanned according to the target scanning scheme corresponding to the target problem type through the M threads to obtain M groups of scanning results.

[0152] In some embodiments, the processing module 402 is specifically configured to:

[0153] The M groups of scanning results are written in the form of rows or columns into a data table through the M threads, and each row or each column of the data table is locked by using a thread lock in the process of writing to obtain a first summary table, the first summary table including M rows of scanning results or M columns of scanning results; and the summary table is written into a local database.

[0154] In some embodiments, the processing module 402 is specifically configured to:

[0155] The N target metadatas are screened to obtain P target metadatas, the P target metadatas including the metadatas related to the target problem type in the N target metadatas, and P is less than or equal to N; and the P target metadatas are divided to obtain M groups of target metadatas.

[0156] In some embodiments, the target problem type is any one of a plurality of problem types, and the plurality of problem types include metadata content error, metadata dependency absence, and inconsistency between metadata definition and actual table definition; and the processing module 402 is specifically configured to:

[0157] Select the target scanning scheme corresponding to the target problem type from the set of scanning schemes corresponding to multiple problem types; according to the target scanning scheme, perform problem scanning on N target metadata to obtain the scanning results of N target metadata.

[0158] It should be noted that the information interaction and execution process between the above-mentioned devices / units are based on the same concept as the method embodiments of this application. For details on their specific functions and technical effects, please refer to the method embodiments section, and they will not be repeated here.

[0159] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0160] This application also provides a computer device, which includes: at least one processor, a memory, and a computer program stored in the memory and executable on the at least one processor, wherein the processor executes the computer program to implement the steps in any of the above method embodiments.

[0161] For example, this application provides a schematic diagram of the structure of a computer device. (As shown...) Figure 9 As shown, the computer device 500 includes a processor 501, a memory 502, a communication interface 503, and a bus 505. The processor 501, memory 502, and communication interface 503 communicate via the bus 505, or via other means such as wireless transmission. The memory 502 stores instructions, and the processor 501 executes the instructions stored in the memory 502. The memory 502 stores program code 5021, and the processor 501 can call the program code 5021 stored in the memory 502 to execute the methods described in the above embodiments.

[0162] It should be understood that, in this application, the processor 501 can be a CPU, and the processor 501 can also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor, etc.

[0163] The memory 502 can include read-only memory and random access memory, and provide instructions and data to the processor 501. The memory 502 can also include non-volatile random access memory. The memory 502 can be a volatile memory or a non-volatile memory, or can include both volatile and non-volatile memories. Among them, the non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM) or flash memory. The volatile memory can be random access memory (RAM) used as an external cache. By way of example but not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchlink dynamic random access memory (SLDRAM) and direct rambus random access memory (DR RAM).

[0164] The bus 505 can include not only a data bus, but also a power bus, a control bus, a status signal bus, etc. However, for the sake of clarity, all the buses are marked as bus 505 in Figure 9 .

[0165] The embodiments of the present application also provide a computer readable storage medium, which stores a computer program. The computer program is executed by a processor to implement the steps in each of the above-mentioned method embodiments.

[0166] The embodiment of the present application provides a computer program product, when the computer program product is executed on a mobile terminal, so that the computer device executes the steps in the above-mentioned various method embodiments.

[0167] The integrated unit, if in the form of a software function unit and sold or used as an independent product, can be stored in a computer-readable storage medium. Based on such understanding, all or part of the processes in the above-mentioned embodiment methods can be completed by a computer program instructing related hardware, and the computer program can be stored in a computer-readable storage medium. The computer program can implement the steps in the above-mentioned various method embodiments when executed by a processor. The computer program includes computer program code, which can be in the form of source code, object code, executable files or some intermediate forms. The computer-readable medium at least includes any entity or device capable of carrying the computer program code to the computer device, a recording medium, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunications signal and a software distribution medium. For example, a U disk, a mobile hard disk, a magnetic disk or an optical disk.

[0168] In the above-mentioned embodiments, the description of each embodiment has its own focus, and the parts not described or recorded in detail in a certain embodiment can be referred to the relevant description of other embodiments.

[0169] Those skilled in the art can appreciate that the units and algorithm steps of the examples described in combination with the embodiments disclosed herein can be realized by electronic hardware or a combination of computer software and electronic hardware. Whether the functions are realized in hardware or software depends on the specific application and design constraints of the technical solution. The skilled person can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0170] In the embodiments provided in the present application, it should be understood that the disclosed apparatus / computer device and method can be implemented in other ways. For example, the above-mentioned apparatus / computer device embodiments are only schematic, for example, the division of the modules or units is only a logical function division, and actual implementation can have another division manner, for example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the shown or discussed mutual elements can be indirect coupling or communication connection through some interfaces, devices or units, and can be electrical, mechanical or other forms.

[0171] The units described as separate components may or may not be physically separate, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed to multiple network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the embodiment scheme.

[0172] The above embodiments are only used to illustrate the technical solutions of the present application, but not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that the technical solutions recorded in the foregoing embodiments can still be modified, or some technical features can be replaced by equivalents; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should be included in the protection scope of the present application.

Claims

1. A metadata scanning method, characterized in that, include: Receive a scan request, the scan request including a target problem type and a metadata identifier, the target problem type including the metadata recording error type; Based on the scan request, find N target metadata corresponding to the metadata identifier, where N is a positive integer greater than or equal to 1; Based on the target scanning scheme corresponding to the target problem type, a problem scan is performed on the N target metadata to obtain the scan results of the N target metadata.

2. The method as described in claim 1, characterized in that, The process of finding the N target metadata corresponding to the metadata identifier includes: Load the N original metadata corresponding to the metadata identifier, where each of the N original metadata includes multiple sub-metadata; The multiple sub-metadata corresponding to each original metadata are merged to obtain the N target metadata.

3. The method as described in claim 2, characterized in that, The multiple sub-data elements are stored in an inheritance tree, which includes multiple levels, including a top level, a second-to-top level, and levels below the second-to-top level. The multiple sub-data elements are distributed across these multiple levels. The step of merging the multiple sub-metadata corresponding to each original metadata to obtain the N target metadata includes: For any given original metadata, the first sub-metadata of the top level among the multiple levels is used as the base template. Based on the base template, the sub-metadata of other levels in the multiple levels, except for the top level, are merged in order from high to low to obtain the target metadata.

4. The method as described in claim 3, characterized in that, In the plurality of sub-metadata, the sub-metadata in the second-to-top level and the levels below the second-to-top level are all associated with data tags. The first sub-metadata of the top level in the plurality of levels is used as a base template, and based on the base template, the sub-metadata of other levels in the plurality of levels, except the top level, are merged sequentially in descending order of hierarchy to obtain the target metadata, including: Using the first sub-metadata of the top level among the multiple levels as the base template, and based on the base template, in order from high to low, the sub-metadata of other levels among the multiple levels except the top level are merged in sequence; During the merging process, if the data tag of the second sub-metadata of the first level is newly added, the second sub-metadata is added on the basis of the first metadata to obtain the target metadata. The first level is any level other than the top level among the multiple levels, and the first metadata is the metadata obtained by merging before the current time. If the data tag of the second sub-metadata in the first level is "delete", then delete the second sub-metadata in the first metadata to obtain the target metadata; If the data tag of the second sub-metadata in the first level is modified, then the sub-metadata corresponding to the second sub-metadata in the first metadata is replaced with the second sub-metadata to obtain the target metadata.

5. The method as described in claim 2, characterized in that, After merging the multiple sub-metadata corresponding to each original metadata to obtain the N target metadata, the method further includes: The N target metadata are divided into M groups of target metadata, where M is less than N; The step of performing a problem scan on the N target metadata according to the target scanning scheme corresponding to the target problem type, and obtaining the scan results of the N target metadata, includes: Create M threads using thread management; Using the M threads, and based on the target scanning scheme corresponding to the target problem type, the M sets of target metadata are scanned to obtain the M sets of scanning results.

6. The method as described in claim 5, characterized in that, After obtaining M sets of scan results by scanning the N target metadata using the M threads according to the target scanning scheme corresponding to the target problem type, the method further includes: The M threads write the M sets of scan results into a data table in the form of rows or columns. During the writing process, thread locks are used to lock each row or column of the data table to obtain a first summary table, which includes M rows of scan results or M columns of scan results. Write the first summary table to the local database.

7. The method as described in claim 5, characterized in that, The step of dividing the N target metadata into M groups of target metadata includes: The N target metadata are filtered to obtain P target metadata, wherein the P target metadata include metadata related to the target problem type from the N target metadata, and P is less than or equal to N; The P target metadata are divided into M groups of target metadata.

8. The method according to any one of claims 1 to 7, characterized in that, The target problem type is any one of multiple problem types, including metadata content errors, missing dependencies between metadata, and inconsistencies between the metadata definition of the table and the actual table definition. The step of performing a problem scan on the target metadata according to the target problem type to obtain the scan results of the N target metadata includes: Select the target scanning scheme corresponding to the target problem type from the set of scanning schemes corresponding to the multiple problem types; According to the target scanning scheme, problem scanning is performed on all N target metadata to obtain the scanning results of the N target metadata.

9. A metadata scanning device, characterized in that, include: A receiving module is configured to receive a scan request, the scan request including a target problem type and a metadata identifier, the target problem type including a metadata recording error type; The processing module is used to find N target metadata corresponding to the metadata identifier, where N is a positive integer greater than or equal to 1; The processing module is also used to perform problem scanning on the target metadata according to the target scanning scheme corresponding to the target problem type, and obtain the scanning results of the N target metadata.

10. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the method as described in any one of claims 1 to 8.

11. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1 to 8.

12. A computer program product, characterized in that, The computer program product includes a computer program that, when run on a computer device, causes the computer device to perform the method as described in any one of claims 1 to 8.

Images