Intelligent guidance and anti-duplicate submission method and system for data asset registration

Abstract

The application relates to the technical field of computer information processing, and discloses an intelligent guidance and repeated submission prevention method and system for data asset registration, which comprises the following steps: receiving a questionnaire response, generating a scene identification vector, and matching a registration scene and a template; generating a registration form according to the template and checking form data; normalizing and serializing registration metadata, generating registration metadata fingerprints, and matching the registration metadata fingerprints with historical registration fingerprints to determine repetition; when no match is found, filtering a candidate set according to a subject type, a data type and a registration purpose in the scene vector, and sequentially performing semantic and structural similarity detection to determine suspected repetition or pass; generating a pre-trial state according to detection and checking results, and writing into a database and saving the registration metadata fingerprints when there is no blockage. Through scene adaptation and layered detection, the application reduces irrelevant filling and comparison, identifies repeated registration of expression or structural change, and reduces the risk of repeated writing.

Description

Technical Field

[0001] This invention relates to the field of computer information processing technology, and more specifically, to an intelligent guidance and anti-duplicate submission method and system for data asset registration. Background Technology

[0002] With the development of data element circulation and data asset management, data asset registration platforms typically receive electronic materials submitted by applicants, including basic data asset information, data source and ownership certificates, and registration certificates. These electronic materials undergo integrity verification, summary calculation, registration, and status query. Some data asset registration platforms employ technologies such as blockchain, distributed storage, and smart contracts to write registration certificates or information to multiple storage nodes and use on-chain transaction identifiers to achieve traceability, auditability, and tamper-proofing of the registration process.

[0003] For example, Chinese patent application CN119988495A discloses a method and apparatus for data asset registration and storage. Users upload data asset registration certificates to a blockchain management system. The system segments the uploaded electronic data, selects storage nodes based on node performance, establishes a mapping relationship between user information and the data to be accessed, and verifies the integrity of the electronic data. This solution mainly focuses on the storage reliability of registration certificates and the integrity of the storage results. It lacks further processing regarding the correspondence between application information and supporting materials under different registration items, as well as the correlation between historical registration records. When the registration information submitted by the applicant has missing fields, mismatched materials, or the same data asset is resubmitted due to different names, file formats, or descriptions, the platform struggles to detect these issues before acceptance, easily increasing the workload of material correction, duplicate acceptance, and subsequent review.

[0004] Therefore, it is necessary to design an intelligent guidance and anti-duplicate submission method and system for data asset registration to solve the problems existing in the current technology. Summary of the Invention

[0005] In view of this, the present invention proposes an intelligent guidance and anti-duplicate submission method and system for data asset registration, which aims to solve the problem that existing registration platforms have difficulty in determining whether the current application and the previous application constitute duplicate registration when the name, description, file format or field arrangement of the same data asset changes and is submitted again.

[0006] This invention proposes an intelligent guidance and anti-duplicate submission method for data asset registration, comprising: Receive scenario questionnaire responses submitted by registered users, generate scenario identifier vectors based on subject type, data type, registration purpose and industry field, match registration scenarios based on the vectors and obtain templates; The registration form is rendered to the terminal according to the template, the form data is received and integrity and consistency checks are performed. When a pre-submission detection request is received, the registration metadata in the form data is normalized and serialized, and the registration metadata fingerprint is calculated. The registration metadata fingerprint is then matched with the historical registration fingerprint in the historical registration fingerprint database. If the same historical registration fingerprint is matched, it is determined to be a duplicate. When no matching historical registration fingerprint is found, a historical registration candidate set is filtered according to the subject type, data type and registration purpose in the scene identifier vector. If the candidate set is empty, the detection is deemed passed. If the candidate set is not empty, semantic similarity detection is performed. If semantic similarity is detected, it is determined to be suspected duplication. If no semantic similarity is detected, structural feature similarity detection is performed, and the suspected duplication or detection is deemed passed based on the structural similarity. A preliminary review status is generated based on the detection and verification results. If the detection result is duplicate or suspected duplicate, it is marked as blocked. The form data is written to the registration database only when there are no blocked items, and the registration metadata fingerprint corresponding to this registration is saved.

[0007] Furthermore, when registering a scene and obtaining a template based on the scene identifier vector, the process includes: Retrieve the node path corresponding to the scene identifier vector in the pre-established scene knowledge graph, and obtain the registration template identifier associated with it based on the matched node path.

[0008] Further, when matching the registered scene in the scene knowledge graph, it includes: The scenario questionnaire responses are mapped to enumerated values ​​and concatenated in a preset order to form a scenario identifier vector. The scenario identifier vector is used to retrieve the node path in the scenario knowledge graph. If a unique matching path exists, the target path is determined. If multiple matching paths exist, the target path is determined according to the applicable relationship weight. If no complete matching path exists, the dimensions in the scenario identifier vector are deleted step by step according to the preset dimension priority and rematched. If no match is found, the general registration template is called.

[0009] Furthermore, when rendering the registration form to the terminal based on the template, the process includes: The system reads the field configuration and validation rules from the template, generates form controls on the user terminal based on the field configuration, receives field data submitted by the user terminal, performs field integrity and cross-field consistency validation according to the validation rules when the field data changes, and returns the validation status to the user terminal.

[0010] Furthermore, when calculating the registration metadata fingerprint, the following steps are included: Extract preset fields from the registration metadata, sort them by field name, perform character encoding unification, null value filling, space normalization and numerical unit format unification on the sorted fields, generate standardized serialization results and perform SHA-256 hash operation to obtain the registration metadata fingerprint.

[0011] Furthermore, performing the semantic similarity detection includes: The registration description texts of the current registration application and the historical registration candidate set are extracted, and the registration description texts are concatenated in a preset order. After word segmentation, stop word filtering and synonym normalization, the feature word weights are determined based on the TF-IDF parameters corresponding to the same version of the historical registration corpus. SimHash is used to generate semantic fingerprints of the historical registration records in the current registration application and the historical registration candidate set according to the feature word weights, and the Hamming distance between the semantic fingerprint of the current registration application and the semantic fingerprint of the historical registration candidate set is calculated. When the Hamming distance is less than the preset semantic similarity threshold, it is determined to be semantically similar.

[0012] Furthermore, when performing the structural feature similarity detection, the process includes: Extract field names, field types, number of fields, number of records, file size, and time coverage from the current registration application and the historical registration candidate set. Encode the field names and field types, normalize the remaining structural features, and concatenate the encoding results and normalization results to form a structural feature vector. Calculate the cosine similarity between the structural feature vector of the current registration application and the structural feature vector in the historical registration candidate set, and determine structural similarity when the cosine similarity is greater than a preset structural similarity threshold.

[0013] Furthermore, when filtering the historical registration candidate set, the following steps are included: Historical registration records are filtered sequentially according to subject type, data type, and registration purpose; when the number of filtered historical registration records exceeds the preset number, the preset number of historical registration records are retained in descending order of registration time.

[0014] Furthermore, generating the pre-approval status includes: Summarize the results of field integrity and consistency verification and anti-duplicate detection; when the anti-duplicate detection result is passed and there are no blocking items in the verification result, write the form data into the registration database, associate and save the registration metadata fingerprint corresponding to this registration with the scene identifier vector, and associate and save the generated semantic fingerprint and structural feature vector with the corresponding registration record and scene identifier vector respectively.

[0015] Compared with existing technologies, the beneficial effects of this invention are as follows: By receiving scenario questionnaire responses submitted by registered users and generating scenario identifier vectors based on subject type, data type, registration purpose, and industry field, the invention matches the corresponding registration scenario and registration template, enabling the registration form to adapt to the registration scenario and reducing repetitive filling caused by irrelevant fields in fixed forms; by performing integrity and consistency checks on form data, the invention detects missing fields or logical inconsistencies between fields before submission, reducing repeated submissions caused by incomplete or incorrect filling; by standardizing and serializing registration metadata and generating registration metadata fingerprints, the invention can identify duplicate submissions with identical registration metadata; and for For registration applications that do not match the same fingerprint, the historical registration candidate set is filtered based on the scene identifier vector, and semantic similarity detection and structural feature similarity detection are performed sequentially. This identifies suspected duplicate submissions that were not accurately matched due to changes in registration description, data format, or similar structural features. At the same time, candidate set filtering and layer-by-layer detection reduce unnecessary comparison processing. A pre-review status is generated based on the detection and verification results. When the detection result is duplicate or suspected duplicate, or when there are blocking items, the form data is prevented from being written to the registration database. Registration data is written and the corresponding fingerprint information is saved only when there are no blocking items, thereby reducing the risk of duplicate registration data entering the registration database.

[0016] On the other hand, this application also provides an intelligent guidance and anti-duplicate submission system for data asset registration, used to apply the above-mentioned intelligent guidance and anti-duplicate submission method for data asset registration, including: The scene recognition unit is used to receive the scene questionnaire responses submitted by registered users, generate scene identifier vectors based on subject type, data type, registration purpose and industry field, match the registered scene based on the vectors and obtain the template; The form processing unit is used to render a registration form to the terminal according to the template, receive form data and perform integrity and consistency checks. The fingerprint matching unit is used to, upon receiving a pre-submission detection request, normalize and serialize the registration metadata in the form data and calculate the registration metadata fingerprint, match the registration metadata fingerprint with the historical registration fingerprint in the historical registration fingerprint database, and determine that the same historical registration fingerprint is a duplicate when it is matched. The hierarchical detection unit is used to filter the historical registration candidate set according to the scene identifier vector when no matching historical registration fingerprint is found. When the candidate set is empty, the detection is determined to be passed. When the candidate set is not empty, semantic similarity detection is performed. When semantic similarity is detected, it is determined to be suspected duplication. When no semantic similarity is detected, structural feature similarity detection is performed, and the suspected duplication or detection is determined according to structural similarity. The pre-approval submission unit is used to generate a pre-approval status based on the detection and verification results. When the detection result is duplicate or suspected duplicate, it is marked as blocked. The form data is written to the registration database only when there are no blocked items, and the fingerprint information corresponding to this registration is saved.

[0017] It is understandable that the intelligent guidance and anti-duplicate submission methods and systems for the above-mentioned data asset registration have the same beneficial effects, and will not be elaborated further here. Attached Figure Description

[0018] Various other advantages and benefits will become apparent to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit the invention. Furthermore, the same reference numerals denote the same parts throughout the drawings. In the drawings: Figure 1 A flowchart illustrating the intelligent guidance and anti-duplicate submission method for data asset registration provided in this embodiment of the invention; Figure 2 This is a flowchart of semantic similarity and structural feature similarity detection provided in an embodiment of the present invention; Figure 3 A functional block diagram of the intelligent guidance and anti-duplicate submission system for data asset registration provided in this embodiment of the invention. Detailed Implementation

[0019] Exemplary embodiments of the present disclosure will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided to enable a more thorough understanding of the present disclosure and to fully convey the scope of the disclosure to those skilled in the art. It should be noted that, unless otherwise specified, embodiments and features in the embodiments of the present invention can be combined with each other. The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0020] In some embodiments of this application, see Figures 1-2 As shown, this application proposes an intelligent guidance and anti-duplicate submission method for data asset registration, including: S100: Receives scenario questionnaire responses submitted by registered users, generates scenario identifier vectors based on subject type, data type, registration purpose and industry field, matches registered scenarios based on vectors and obtains templates; S200: Renders the registration form to the terminal based on the template, receives the form data and performs integrity and consistency checks; S300: When a pre-submission detection request is received, the registration metadata in the form data is normalized and serialized, and the registration metadata fingerprint is calculated. The registration metadata fingerprint is matched with the historical registration fingerprint in the historical registration fingerprint database. If the same historical registration fingerprint is matched, it is determined to be a duplicate. S400: When no matching historical registration fingerprint is found, the historical registration candidate set is filtered according to the subject type, data type and registration purpose in the scene identifier vector. If the candidate set is empty, the detection is considered passed. If the candidate set is not empty, semantic similarity detection is performed. If semantic similarity is detected, it is determined to be suspected duplication. If no semantic similarity is detected, structural feature similarity detection is performed, and the suspected duplication or detection is determined according to the structural similarity. S500: Generate a pre-approval status based on the detection and verification results. If the detection result is duplicate or suspected duplicate, mark it as blocked. Only when there are no blocked items will the form data be written to the registration database and the registration metadata fingerprint corresponding to this registration be saved.

[0021] In one implementation, registered users access a data asset registration page via a terminal. The terminal displays a scenario questionnaire, which includes at least the subject type, data type, registration purpose, and industry sector. The system receives responses to the scenario questionnaire submitted by registered users, maps each response to a preset enumeration value, and concatenates them in a fixed order according to the subject type, data type, registration purpose, and industry sector to form a scenario identifier vector.

[0022] A scenario knowledge graph is pre-built before the method is executed. The scenario knowledge graph includes nodes for subject type, data type, registration purpose, and industry domain. Different nodes are connected according to their applicability relationships to form node paths. Each complete node path is associated with a corresponding registration template identifier. The applicability relationship indicates that adjacent nodes can be combined to form a valid registration scenario.

[0023] Configure application relationship weights for each application relationship. When determining application relationship weights, read historical registration records that have been processed within a preset statistical period. Based on the subject type, data type, registration purpose, and industry sector in the historical registration records, count the number of times each application relationship's corresponding node combination has been actually matched and registered. Then, normalize the registration counts of each application relationship according to the registration counts of each valid application relationship under the same node level, and use the normalized result as the initial weight of the corresponding application relationship. The application relationship weight ranges from 0 to 1; the more registration counts of a node combination, the greater the weight of the corresponding application relationship. For application relationships without historical registration records within the preset statistical period, set their weights to the pre-configured default weights. After a new registration record is added, update the registration counts and application relationship weights of the corresponding application relationship based on the node path corresponding to the new registration record.

[0024] After generating scene identifier vectors, the node paths corresponding to the scene identifier vectors are retrieved from the scene knowledge graph. When a unique, complete matching node path is found, it is designated as the target path, and the registration template identifier associated with the target path is read. When multiple complete matching node paths are found, the applicability weights of each applicable relationship in each node path are read. The applicability weights of all applicable relationships in a node path are added together, and then divided by the number of applicable relationships in that node path to obtain the path score. The path scores of each node path are compared, and the node path with the highest path score is designated as the target path. If the path scores are the same, the node path with the most historical calls to the registration templates associated with each node path is designated as the target path; if the historical call counts are still the same, the target path is determined according to the preset order of the registration template identifiers.

[0025] For example, the first node path contains three applicable relationships with weights of 0.82, 0.75, and 0.90. Averaging these weights gives the first node path a path score of 0.823. The second node path contains three applicable relationships with weights of 0.78, 0.80, and 0.85, giving it a path score of 0.81. Since the first node path has a higher path score than the second node path, it is designated as the target path.

[0026] When no node path completely matches the scene identifier vector, dimensions in the scene identifier vector are gradually deleted according to preset dimension priorities, and matching is performed again. In one implementation, the priority of the dimensions of subject type, data type, registration purpose, and industry domain decreases in that order. First, the industry domain dimension is deleted, and three-dimensional matching is performed using subject type, data type, and registration purpose; if no node path is obtained in three-dimensional matching, the registration purpose dimension is deleted, and two-dimensional matching is performed using subject type and data type; if no node path is obtained in two-dimensional matching, the data type dimension is deleted, and matching is performed only by subject type; if no node path is obtained using only subject type, the general registration template is invoked.

[0027] The registration template is read based on the registration template identifier associated with the target path, and the field configurations and field validation rules within the registration template are also retrieved. The field configurations record the field names, field types, whether they are required, and the format of their values. The field validation rules record the field value format, value range, and constraints between fields. Based on the field configurations, form controls corresponding to the field types are generated on the terminal. These form controls include one or more of the following: text input controls, selection controls, numeric input controls, and file upload controls.

[0028] The system receives field data submitted by the terminal. When any field data changes, it performs field integrity checks and cross-field consistency checks according to the field validation rules. Field integrity checks include checking whether required fields exist, whether field values ​​are empty, and whether file fields meet preset file type and size requirements. Cross-field consistency checks include checking whether related fields meet the dependency conditions configured in the registration template.

[0029] For example, when the source description field contains "external procurement," check whether the purchase invoice field has been submitted; if no purchase invoice has been submitted, the validation item is identified as an error and marked as a blocked item. When the data start time, data end time, and update frequency are all filled in, determine the time coverage range based on the data start time and data end time, and check whether the time coverage range matches the update frequency; if they do not match, a warning status is generated. After the validation is completed, the validation status and corresponding prompt information are returned to the terminal. In one implementation, the validation status includes a pass status, a warning status, and an error status; the warning status does not form a blocked item, while the error status does.

[0030] Upon receiving a pre-submission detection request, the registration metadata is extracted from the form data and then standardized and serialized. In one implementation, the following fields are fixed as preset fields: data_name, data_intro, data_source, data_format, and data_scale, and arranged in alphabetical order according to their key names: data_format, data_intro, data_name, data_scale, and data_source.

[0031] After sorting, the preset fields are subjected to unified character encoding, null padding, space normalization, and unified numerical unit format. Character encoding is unified to UTF-8 encoding; when a preset field is missing a value, it is filled with an empty string, and the filled field continues to participate in serialization; leading and trailing spaces in field values ​​are removed, and consecutive spaces and tabs are converted into a single space; for non-Chinese text fields, they are uniformly converted to lowercase; dates are uniformly formatted as YYYY-MM-DD; if a numerical value has a unit, it is uniformly split into "numerical value + standard unit code" (e.g., "500GB" is converted to "500_GB").

[0032] After normalization, a standard JSON string is generated according to the aforementioned fixed field order. The standard JSON string does not contain extra spaces or newlines. A SHA-256 hash operation is then performed on the standard JSON string to obtain a 256-bit registration metadata fingerprint.

[0033] After obtaining the registration metadata fingerprint, the historical registration fingerprint database is queried using the registration metadata fingerprint as the search criteria, and the registration metadata fingerprint is matched with the queried historical registration fingerprints. When a historical registration fingerprint identical to the registration metadata fingerprint exists, the current registration application is determined to be a duplicate, and subsequent semantic similarity detection and structural feature similarity detection are stopped.

[0034] After each registration application passes field validation and anti-duplicate detection and is written to the registration database, the generated registration metadata fingerprint is saved to the historical registration fingerprint database. At the same time, a semantic fingerprint is generated according to the same registration description text processing rules, and a structural feature vector is generated according to the same structural feature encoding rules. The semantic fingerprint and structural feature vector are then associated with and saved with the corresponding historical registration records for matching in subsequent registration applications.

[0035] When no matching historical registration fingerprint is found, the historical registration candidate set is filtered based on the subject type, data type, and registration purpose in the scene identifier vector corresponding to the current registration application. First, historical registration records are filtered by subject type; then, the results of the first filtering are filtered by data type; and finally, the results of the second filtering are filtered by registration purpose. The industry sector in the scene identifier vector is used for registration scene matching and registration template acquisition, but is not a necessary filtering condition for the historical registration candidate set.

[0036] When the number of historical registration records obtained from the third screening exceeds the preset number, the registration time of each historical registration record is read, sorted in descending order of registration time, and the preset number of historical registration records at the top of the sort are retained to form a historical registration candidate set.

[0037] The preset number is obtained through candidate number calibration. Historical registration samples containing known duplicate records are selected, and candidate set filtering is performed using multiple candidate numbers. The number of known duplicate records retained under each candidate number and the detection processing time are recorded. The candidate number with the smallest value that can retain all known duplicate records is selected as the preset number for the historical registration candidate set. For example, if 500 records fail to retain all known duplicate records, 1000 records do retain all known duplicate records, and increasing to 2000 records does not increase the number of known duplicate records retained, then 1000 records are determined as the preset number.

[0038] When the historical registration candidate set is empty, the current registration application is determined to have passed the detection, and semantic similarity detection and structural feature similarity detection between the current registration application and historical registration records are not performed. When the historical registration candidate set is not empty, semantic similarity detection is performed.

[0039] When performing semantic similarity detection, the registration description text of each historical registration record in the current registration application and the historical registration candidate set is extracted. In one implementation, the registration description text is extracted in a fixed order of data name, data summary, application scenario, and source description; if a corresponding field does not exist, the field is treated as a null value. The extracted registration description text is concatenated in the order of field name, separator, and field value, with a preset field separator inserted between adjacent fields.

[0040] The concatenated registration description text is segmented into Chinese, and stop words and words with a length of one character are removed. The retained words are identified as feature words. Feature words are then normalized using a pre-configured thesaurus. When a feature word matches any synonym in the thesaurus, it is converted into the corresponding standard word. For example, "collect," "gather," and "obtain" are converted into the same standard word.

[0041] Before method execution, a corpus version number is set for the historical registration corpus. For each corpus version, a fixed set of historical registration texts is selected. These texts are processed according to the aforementioned word segmentation, stop word filtering, and synonym normalization rules. An inverse document frequency (IVF) parameter table corresponding to that corpus version is generated based on the distribution of each feature word in the historical registration text set. Under the same corpus version, the current registration application and historical registration records in the historical registration candidate set use the same IVF parameter table to determine feature word weights and generate semantic fingerprints. The corpus version number is then associated with and saved with the semantic fingerprint. Adding a new registration record does not directly change the IVF parameter table of the currently valid version. When the historical registration corpus needs to be updated, a new corpus version is created, and the semantic fingerprints of the historical registration records are regenerated according to the new IVF parameter table. Once the historical registration records are regenerated, the new corpus version is activated, and semantic similarity detection only compares semantic fingerprints generated using the same corpus version.

[0042] The term frequency is determined based on the frequency of occurrence of the feature words in the current registered description text, and the inverse document frequency is determined based on the distribution of the feature words in the historical registered text set of the current valid version. The feature word weights are determined based on the term frequency and the inverse document frequency. When generating the semantic fingerprint, each feature word is hashed and encoded, and the hash bits are accumulated according to the corresponding feature word weights. The value of each hash bit is determined based on the accumulation result, and a 64-bit semantic fingerprint is formed from all hash bits.

[0043] Calculate the Hamming distance between the semantic fingerprint of the current registration application and the semantic fingerprints of each historical registration record in the historical registration candidate set. If any Hamming distance is less than a preset semantic similarity threshold, the current registration application is determined to be semantically similar, and the detection result is determined to be suspected duplication; structural feature similarity detection is no longer performed. If all Hamming distances are not less than the preset semantic similarity threshold, it is determined that no semantic similarity was detected, and structural feature similarity detection is performed.

[0044] The semantic similarity threshold is obtained through semantically labeled samples. Registration record pairs already recorded as duplicates in the historical registration database are selected as duplicate samples, and registration record pairs already recorded as passed are selected as non-duplicate samples. Semantic fingerprints of each registration record pair are generated under the same corpus version, and the Hamming distance between the semantic fingerprints of each duplicate and non-duplicate sample is calculated. Multiple candidate thresholds are set, and the precision and recall under each candidate threshold are calculated. The overall evaluation value is determined based on the harmonic average of precision and recall. The candidate threshold with the highest overall evaluation value is determined as the semantic similarity threshold. If multiple candidate thresholds have the same overall evaluation value, the candidate threshold with the smaller value is selected.

[0045] In one implementation, 1000 pairs of historical registration records are selected as semantic labeling samples, of which 500 pairs are duplicate samples and the other 500 pairs are non-duplicate samples. Under the same corpus version, thresholds 2, 3, and 4 are used as candidate thresholds for labeling. Threshold 3 corresponds to the highest comprehensive evaluation value, therefore 3 is determined as the semantic similarity threshold. A Hamming distance of 0, 1, or 2 indicates semantic similarity, while a Hamming distance of 3 or higher indicates semantic dissimilarity.

[0046] When performing structural feature similarity detection, extract the field names, field types, number of fields, number of records, file size, and time coverage of each historical registration record in the current registration application and historical registration candidate set.

[0047] Before method execution, a dictionary of field names and a dictionary of field types are pre-built, arranged in a fixed order. When building the field name dictionary, field names appearing in historical records are extracted, and normalization processes are performed sequentially, including removing spaces, converting full-width characters to half-width characters, converting English letters to lowercase characters, and removing version number suffixes. The normalized field names are then deduplicated and sorted, and each field name is assigned a fixed index position. When building the field type dictionary, text, numeric, date, and file field types are arranged in a fixed order, and each field type is assigned a fixed index position.

[0048] The same normalization process is performed on field names in both the current registration application and historical registration records, and the corresponding field names are retrieved from the field name dictionary. When a matching field name is found, the encoding value at the corresponding index position in the field name dictionary is set to 1, and the encoding value at other positions is set to 0. If no identical field name is found, the edit distance between the normalized field name and each field name in the field name dictionary is calculated. If the edit distance is no greater than 2, it is mapped to the field name with the smallest edit distance; if no field name with an edit distance no greater than 2 exists, it is mapped to another reserved field index position. Each field type is encoded according to the field type dictionary in the same manner.

[0049] The number of fields, records, file size, and time coverage are normalized. In one implementation, the number of fields is compared with the maximum allowed number of fields and converted to a value range of 0 to 1; the number of records is first logarithmically scaled, and then converted to a value range of 0 to 1 based on the maximum allowed number of records; the file size is converted to megabytes and then converted to a value range of 0 to 1 based on the maximum allowed file size; the time coverage is converted to coverage days and then converted to a value range of 0 to 1 based on the maximum allowed coverage days. When a value exceeds the corresponding maximum value, the corresponding normalization result is set to 1; when a value is missing, the corresponding normalization result is set to 0. The maximum number of fields, maximum number of records, maximum file size, and maximum coverage days are preset according to the field constraints of the registration template and the allowed range of registration business.

[0050] The field name encoding results, field type encoding results, and normalized results of field quantity, record quantity, file size, and time coverage are concatenated in a fixed order to form a fixed-dimensional structural feature vector. After the field name dictionary or field type dictionary is updated, the structural feature vectors of historical registration records are re-encoded according to the updated dictionary to ensure that the current registration application and historical registration records use the same vector dimension and index order.

[0051] Calculate the cosine similarity between the structural feature vector of the current registration application and the structural feature vectors of each historical registration record in the historical registration candidate set. If any cosine similarity is greater than a preset structural similarity threshold, the current registration application is determined to be structurally similar, and the detection result is determined to be a suspected duplicate; if all cosine similarities are not greater than the preset structural similarity threshold, the detection result is determined to be a successful detection.

[0052] The structural similarity threshold is obtained through structural calibration samples. Registration record pairs already recorded as duplicates are selected as duplicate samples, and registration record pairs already recorded as passed are selected as non-duplicate samples. The cosine similarity of structural features for each duplicate and non-duplicate sample is calculated. Multiple candidate thresholds are set, and the precision and recall at each candidate threshold are calculated. The overall evaluation value is determined based on the harmonic average of precision and recall. The candidate threshold with the highest overall evaluation value is determined as the structural similarity threshold. If multiple candidate thresholds have the same overall evaluation value, the candidate threshold with the larger value is selected.

[0053] In one implementation, 0.80, 0.85, and 0.90 were used as candidate thresholds for calibration. 0.85 yielded the highest overall evaluation value, therefore, 0.85 was determined as the structural similarity threshold. A cosine similarity greater than 0.85 was considered structurally similar, while a cosine similarity equal to or less than 0.85 was considered structurally dissimilar.

[0054] After completing the anti-duplicate detection, the results of field integrity verification, field consistency verification, and anti-duplicate detection are summarized, and a pre-approval status is generated based on the summarized results. When the anti-duplicate detection result is duplicate or suspected duplicate, the pre-approval status is marked as blocked; when there are error states in the field integrity verification or field consistency verification results, the corresponding verification item is identified as a blocked item, and the pre-approval status is marked as blocked. A prompt message is generated based on the specific reason for the duplicate, suspected duplicate, or field verification error, and the prompt message is returned to the terminal.

[0055] When the anti-duplicate detection result is passed, and there are no blocking items in the field integrity verification result and the field consistency verification result, the form data is written to the registration database. For the already generated registration metadata fingerprint and semantic fingerprint, they are associated and saved with the current registration record and scene identifier vector, respectively; for the already generated structural feature vector, they are associated and saved with the current registration record and scene identifier vector. For registration applications where the historical registration candidate set is empty and semantic similarity detection and structural feature similarity detection between the current registration application and historical registration records have not been performed, after the form data is written to the registration database, semantic fingerprints and structural feature vectors are still generated according to the preset registration description text processing rules and structural feature encoding rules, and are associated and saved with the registration metadata fingerprint and scene identifier vector.

[0056] In a complete embodiment, under the condition of pre-configured historical registration corpus (version V1), thesaurus, field name dictionary, field type dictionary, and feature normalization parameters, a registration application for the "National 5A Scenic Spot Real-Shot High-Definition Video Dataset" submitted by a video technology company is received. The registering user sequentially selects enterprise, audio / video data, first-time registration, and cultural tourism industry in the scenario questionnaire, concatenates the corresponding enumeration values ​​in a fixed order to form a scenario identifier vector, and matches the corresponding node path in the scenario knowledge graph to obtain the registration template TPL_AV_FIRST_ENT associated with that node path.

[0057] The system renders the registration form to the terminal based on the registration template, receives data from the terminal including video container format, resolution, total duration, and source description, and performs field integrity and consistency checks. All required fields in this registration application have been filled in, and there are no fields that do not comply with the validation rules; therefore, there are no blocking entries in the field validation results.

[0058] Upon receiving the pre-submission inspection request, the registration metadata in this registration application is standardized and serialized, and a registration metadata fingerprint is calculated. This fingerprint is then matched against historical registration fingerprints in the historical registration fingerprint database. If no matching historical registration fingerprint is found, historical registration records are filtered sequentially according to subject type, data type, and registration purpose to form a historical registration candidate set.

[0059] Extract the registration description text from the current registration application and the historical registration candidate set, and generate semantic fingerprints. Using the V1 version of the historical registration corpus and thesaurus used in this embodiment, the minimum Hamming distance between the current registration application and each historical registration record in the historical registration candidate set is calculated to be 15. Since 15 is not less than the semantic similarity threshold of 3, it is determined that no semantic similarity was detected, and structural feature similarity detection continues. Structural feature vectors are generated based on field name, field type, number of fields, number of records, file size, and time coverage. Under the condition of using the same field dictionary and normalization parameters, the maximum cosine similarity between the current registration application and each historical registration record in the historical registration candidate set is calculated to be 0.60. Since 0.60 is not greater than the structural similarity threshold of 0.85, the anti-duplicate detection result is determined to be a pass.

[0060] Based on the results of the detection and the field verification results of the absence of blocking items, a pre-approval status is generated. The form data of this registration application is written into the registration database, and the registration metadata fingerprint, semantic fingerprint, and structural feature vector generated during this detection process are associated with and saved with the scene identifier vector.

[0061] In another complete embodiment, using the same version of historical registration corpus, thesaurus, field name dictionary, field type dictionary, and feature processing parameters as in the aforementioned embodiment, the same video dataset is submitted again, with the data name changed to "Real-shot Material Library of Famous Scenic Spots Nationwide" and the data description changed to "Real-shot Video Resources Covering Famous Scenic Spots in China". The subject type, data type, registration purpose, and industry field of this registration application remain unchanged; therefore, the same scene identifier vector is generated as in the aforementioned embodiment, and the same registration template is obtained.

[0062] Because the data name and description have changed, the metadata fingerprint of this registration differs from the registration metadata fingerprint saved in the aforementioned embodiments, and no matching historical registration fingerprint was found in the historical registration fingerprint database. After filtering the historical registration candidate set based on the subject type, data type, and registration purpose in the scene identifier vector, the historical registration candidate set contains the registration records written in the aforementioned embodiments. The registration description text is extracted and a semantic fingerprint is generated. The Hamming distance between this registration application and the registration record in the aforementioned embodiments is calculated to be 2. Since 2 is less than the semantic similarity threshold of 3, this registration application is determined to be semantically similar, and the anti-duplicate detection result is determined to be suspected duplication, and structural feature similarity detection is no longer performed.

[0063] A notification message will be generated based on suspected duplicate test results, and the preliminary review status will be marked as blocked. The form data from this registration application will not be written to the registration database, nor will it be saved as a new historical registration record.

[0064] Based on another preferred embodiment described above, see [link to preferred embodiment]. Figure 3 As shown, this embodiment provides an intelligent guidance and anti-duplicate submission system for data asset registration, used to apply the above-mentioned intelligent guidance and anti-duplicate submission method for data asset registration, including: The scene recognition unit is used to receive the scene questionnaire responses submitted by registered users, generate scene identifier vectors based on subject type, data type, registration purpose and industry field, match the registered scene based on the vector and obtain the template; The form processing unit is used to render the registration form to the terminal according to the template, receive the form data and perform integrity and consistency checks. The fingerprint matching unit is used to normalize and serialize the registration metadata in the form data and calculate the registration metadata fingerprint when a pre-submission detection request is received. The registration metadata fingerprint is then matched with the historical registration fingerprint in the historical registration fingerprint database. If the same historical registration fingerprint is matched, it is determined to be a duplicate. The hierarchical detection unit is used to filter the historical registration candidate set according to the subject type, data type and registration purpose in the scene identifier vector when no matching historical registration fingerprint is found. When the candidate set is empty, the detection is determined to be passed. When the candidate set is not empty, semantic similarity detection is performed. When semantic similarity is detected, it is determined to be suspected duplication. When no semantic similarity is detected, structural feature similarity detection is performed, and the suspected duplication or detection is determined according to the structural similarity. The pre-approval submission unit is used to generate a pre-approval status based on the detection and verification results. When the detection result is duplicate or suspected duplicate, it is marked as blocked. The form data is written to the registration database only when there are no blocked items, and the registration metadata fingerprint corresponding to this registration is saved.

[0065] It is understandable that the intelligent guidance and anti-duplicate submission methods and systems for the above-mentioned data asset registration have the same beneficial effects, and will not be elaborated further here.

[0066] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the specific implementation of the present invention. Any modifications or equivalent substitutions that do not depart from the spirit and scope of the present invention should be covered within the protection scope of the present invention.

Claims

1. A method for intelligent guidance and anti-duplicate submission of data asset registration, characterized in that, include: Receive scenario questionnaire responses submitted by registered users, generate scenario identifier vectors based on subject type, data type, registration purpose and industry field, match registration scenarios based on the vectors and obtain templates; The registration form is rendered to the terminal according to the template, the form data is received and integrity and consistency checks are performed. When a pre-submission detection request is received, the registration metadata in the form data is normalized and serialized, and the registration metadata fingerprint is calculated. The registration metadata fingerprint is matched with the historical registration fingerprint in the historical registration fingerprint database. If the same historical registration fingerprint is matched, it is determined to be a duplicate. When no matching historical registration fingerprint is found, a historical registration candidate set is filtered according to the subject type, data type and registration purpose in the scene identifier vector. If the candidate set is empty, the detection is deemed passed. If the candidate set is not empty, semantic similarity detection is performed. If semantic similarity is detected, it is determined to be suspected duplication. If no semantic similarity is detected, structural feature similarity detection is performed, and the suspected duplication or detection is deemed passed based on the structural similarity. A preliminary review status is generated based on the detection and verification results. If the detection result is duplicate or suspected duplicate, it is marked as blocked. The form data is written to the registration database only when there are no blocked items, and the registration metadata fingerprint corresponding to this registration is saved.

2. The intelligent guidance and anti-duplicate submission method for data asset registration according to claim 1, characterized in that, When registering a scene and obtaining a template based on the scene identifier vector, the process includes: Retrieve the node path corresponding to the scene identifier vector in the pre-established scene knowledge graph, and obtain the registration template identifier associated with it based on the matched node path.

3. The intelligent guidance and anti-duplicate submission method for data asset registration according to claim 2, characterized in that, When matching the registered scene in the scene knowledge graph, the following steps are included: The scenario questionnaire responses are mapped to enumerated values ​​and concatenated in a preset order to form a scenario identifier vector. The scenario identifier vector is used to retrieve the node path in the scenario knowledge graph. If a unique matching path exists, the target path is determined. If multiple matching paths exist, the target path is determined according to the applicable relationship weight. If no complete matching path exists, the dimensions in the scenario identifier vector are deleted step by step according to the preset dimension priority and rematched. If no match is found, the general registration template is called.

4. The intelligent guidance and anti-duplicate submission method for data asset registration according to claim 3, characterized in that, When rendering the registration form to the terminal according to the template, the following are included: The system reads the field configuration and validation rules from the template, generates form controls on the user terminal based on the field configuration, receives field data submitted by the user terminal, performs field integrity and cross-field consistency validation according to the validation rules when the field data changes, and returns the validation status to the user terminal.

5. The intelligent guidance and anti-duplicate submission method for data asset registration according to claim 1, characterized in that, Calculating the registration metadata fingerprint includes: Extract preset fields from the registration metadata, sort them by field name, perform character encoding unification, null value filling, space normalization and numerical unit format unification on the sorted fields, generate standardized serialization results and perform SHA-256 hash operation to obtain the registration metadata fingerprint.

6. The intelligent guidance and anti-duplicate submission method for data asset registration according to claim 5, characterized in that, Performing the semantic similarity detection includes: The registration description texts of the current registration application and the historical registration candidate set are extracted, and the registration description texts are concatenated in a preset order. After word segmentation, stop word filtering and synonym normalization, the feature word weights are determined based on the TF-IDF parameters corresponding to the same version of the historical registration corpus. SimHash is used to generate semantic fingerprints of the historical registration records in the current registration application and the historical registration candidate set according to the feature word weights, and the Hamming distance between the semantic fingerprint of the current registration application and the semantic fingerprint of the historical registration candidate set is calculated. When the Hamming distance is less than the preset semantic similarity threshold, it is determined to be semantically similar.

7. The intelligent guidance and anti-duplicate submission method for data asset registration according to claim 6, characterized in that, When performing the structural feature similarity detection, the following are included: Extract field names, field types, number of fields, number of records, file size, and time coverage from the current registration application and the historical registration candidate set. Encode the field names and field types, normalize the remaining structural features, and concatenate the encoding results and normalization results to form a structural feature vector. Calculate the cosine similarity between the structural feature vector of the current registration application and the structural feature vector in the historical registration candidate set, and determine structural similarity when the cosine similarity is greater than a preset structural similarity threshold.

8. The intelligent guidance and anti-duplicate submission method for data asset registration according to claim 7, characterized in that, When filtering the historical registration candidate set, the following steps are included: Historical registration records are filtered sequentially according to subject type, data type, and registration purpose; when the number of filtered historical registration records exceeds the preset number, the preset number of historical registration records are retained in descending order of registration time.

9. The intelligent guidance and anti-duplicate submission method for data asset registration according to claim 8, characterized in that, When generating the aforementioned pre-approval status, the following steps are included: Summarize the results of field integrity and consistency verification and anti-duplicate detection; when the anti-duplicate detection result is passed and there are no blocking items in the verification result, write the form data into the registration database, associate and save the registration metadata fingerprint corresponding to this registration with the scene identifier vector, and associate and save the generated semantic fingerprint and structural feature vector with the corresponding registration record and scene identifier vector respectively.

10. A smart guidance and anti-duplicate submission system for data asset registration, used to apply the smart guidance and anti-duplicate submission method for data asset registration as described in any one of claims 1-9, characterized in that, include: The scene recognition unit is used to receive the scene questionnaire responses submitted by registered users, generate scene identifier vectors based on subject type, data type, registration purpose and industry field, match the registered scene based on the vectors and obtain the template; The form processing unit is used to render a registration form to the terminal according to the template, receive form data and perform integrity and consistency checks. The fingerprint matching unit is used to, upon receiving a pre-submission detection request, normalize and serialize the registration metadata in the form data and calculate the registration metadata fingerprint, match the registration metadata fingerprint with the historical registration fingerprint in the historical registration fingerprint database, and determine that the same historical registration fingerprint is a duplicate when it is matched. The hierarchical detection unit is used to filter the historical registration candidate set according to the subject type, data type and registration purpose in the scene identifier vector when no matching historical registration fingerprint is found. When the candidate set is empty, the detection is determined to be passed. When the candidate set is not empty, semantic similarity detection is performed. When semantic similarity is detected, it is determined to be suspected duplication. When no semantic similarity is detected, structural feature similarity detection is performed, and the suspected duplication or detection is determined according to the structural similarity. The pre-approval submission unit is used to generate a pre-approval status based on the detection and verification results. When the detection result is duplicate or suspected duplicate, it is marked as blocked. The form data is written to the registration database only when there are no blocked items, and the registration metadata fingerprint corresponding to this registration is saved.

Images