Field sensitivity grading method, apparatus, device, and storage medium

Abstract

This application relates to the field of data security technology, and in particular to a method, apparatus, device, and storage medium for classifying field sensitivity. The method includes: acquiring at least two fields to be classified, and comparing the semantic sensitivity of each field pairwise to obtain an initial preference relationship; sorting the fields according to the initial preference relationship to obtain a sensitivity ranking, and determining the sensitivity level corresponding to each field based on the sensitivity ranking. This application obtains an initial preference relationship by acquiring at least two fields to be classified and comparing their semantic sensitivity pairwise, then sorts them according to the initial preference relationship to obtain a sensitivity ranking, and finally determines the sensitivity level based on the sensitivity ranking. This ensures that the sensitivity level is determined by the relative high or low relationship between the pairs of fields, avoiding the problem of the same semantic field being assigned different levels under different input orders when a single field is classified independently.

Description

Technical Field

[0001] This application relates to the field of data security technology, and in particular to a method, apparatus, device and storage medium for classifying field sensitivity. Background Technology

[0002] Under data security compliance requirements, enterprises need to classify the sensitivity of massive fields in the database (such as L1–L5) to support governance measures such as data anonymization and access control.

[0003] Current mainstream solutions employ field-by-field grading: the metadata (field name, comments, sample values, etc.) of a single field is input into a large language model, which then directly outputs a discrete sensitivity level. However, due to the contextual memory effect in large language models, when multiple fields are input sequentially or in batches, the model's grading of the current field is influenced by previously input fields. This results in fields with the same semantic meaning being assigned different sensitivity levels under different input orders or batches, leading to semantic drift. Summary of the Invention

[0004] The main purpose of this application is to provide a method, apparatus, device and storage medium for classifying field sensitivity, which aims to solve the problem that the classification of the current field by the model is affected by the input fields, resulting in fields with the same semantics being assigned different sensitivity levels under different input orders or different batches.

[0005] To achieve the above objectives, this application proposes a field sensitivity classification method, the method comprising: Obtain at least two fields to be classified, and compare the semantic sensitivity of each field to be classified pairwise to obtain the original preference relationship; The fields to be classified are sorted according to the original preference relationship to obtain a sensitivity ranking, and the sensitivity level corresponding to each field to be classified is determined according to the sensitivity ranking.

[0006] In one embodiment, the step of obtaining at least two fields to be classified and comparing the semantic sensitivity of each field to obtain the original preference relationship includes: Obtain the field metadata of at least two fields to be classified, validate each field metadata, and map the validated field metadata to a field descriptor; The field descriptors are vectorized to generate the embedding vectors corresponding to the fields to be classified. Based on the semantic similarity between the embedded vectors, at least one pair of fields to be compared is determined from the fields to be classified. The field names and annotation information of the corresponding fields to be graded in each pair of fields to be compared are assembled into comparison prompt words to instruct the preset language model to perform sensitivity comparison. Each of the comparison prompt words is input into the preset language model, and the preset language model is used to compare each pair of fields to be compared to obtain the original preference relationship.

[0007] In one embodiment, the step of sorting each of the fields to be classified according to the original preference relationship to obtain a sensitivity ranking includes: Obtain at least one nearest neighbor field of the first field to be classified in the original preference relation, and determine the proximity semantic similarity between the nearest neighbor field and the first field to be classified; When the semantic similarity of the nearest neighbor exceeds a preset similarity threshold, a derived preference relationship is generated between the nearest neighbor field and the second field to be classified in the original preference relationship, wherein the sensitivity of the first field to be classified is greater than the sensitivity of the second field to be classified. The fields to be classified are sorted according to the original preference relationship and the derived preference relationship to obtain the sensitivity ranking.

[0008] In one embodiment, the step of sorting each of the fields to be classified according to the original preference relationship and the derived preference relationship to obtain a sensitivity ranking includes: Based on the original preference relationship and the derived preference relationship, the win-loss frequency between each of the fields to be classified is determined, and an initial preference statistics matrix is ​​constructed based on each win-loss frequency. Normalize each row of the initial preference statistics matrix to obtain the target preference statistics matrix; The sensitivity ranking is obtained by sorting each of the fields to be classified according to the initial sensitivity score vector of each field and the target preference statistics matrix.

[0009] In one embodiment, the step of sorting each of the fields to be classified according to its initial sensitivity score vector and the target preference statistics matrix to obtain a sensitivity ranking includes: The initial sensitivity score vectors of each of the fields to be classified are weighted and summed according to the weights of the corresponding rows in the target preference statistics matrix to obtain the intermediate sensitivity score vectors of each of the fields to be classified. Normalize each of the intermediate sensitivity score vectors to obtain the target sensitivity score vector corresponding to each of the fields to be classified. Set the target sensitivity score vector as the new initial sensitivity score vector, and return to the step of performing a weighted summation of the initial sensitivity score vectors of each field to be classified according to the weight of the corresponding row in the target preference statistics matrix, until the sum of the score changes of each field to be classified in two adjacent iterations is less than a preset iteration threshold. The target sensitivity score vector of each field to be classified in the last iteration is used as the sensitivity score of each field to be classified. The corresponding fields to be classified are sorted according to the sensitivity scores to obtain the sensitivity ranking.

[0010] In one embodiment, after the step of generating the derived preference relationship between the nearest neighbor field and the second field to be classified in the original preference relationship, the method further includes: Based on the historical original preference relation set and the historical derived preference relation set, loop detection is performed on the derived preference relation to obtain the detection result; If the detection result is a non-contradictory cycle, the derived preference relation is retained and added to the historical derived preference relation set.

[0011] In one embodiment, before the step of obtaining at least one nearest neighbor field of the first field to be classified in the original preference relation, the method further includes: Obtain the field metadata of at least two fields to be classified, validate each field metadata, and map the validated field metadata to a field descriptor; The field descriptors are vectorized to generate the embedding vectors corresponding to the fields to be classified. Based on the semantic similarity between the embedding vectors, determine the nearest neighbor set of each field to be classified; The step of obtaining at least one nearest neighbor field of the first field to be classified in the original preference relation includes: Obtain at least one nearest neighbor field from the nearest neighbor set of the first field to be classified.

[0012] Furthermore, to achieve the above objectives, this application also proposes a field sensitivity grading device, the device comprising: The relationship acquisition module is used to acquire at least two fields to be classified and to compare the semantic sensitivity of each field to be classified pairwise to obtain the original preference relationship. The level determination module is used to sort each of the fields to be classified according to the original preference relationship to obtain a sensitivity ranking, and to determine the sensitivity level corresponding to each of the fields to be classified according to the sensitivity ranking.

[0013] In addition, to achieve the above objectives, this application also proposes a field sensitivity classification device, the device comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the field sensitivity classification method as described above.

[0014] In addition, to achieve the above objectives, this application also proposes a storage medium, which is a computer-readable storage medium, on which a computer program is stored, and which, when executed by a processor, implements the steps of the field sensitivity classification method as described above.

[0015] This application provides a method, apparatus, device, and storage medium for classifying field sensitivity. The method includes: acquiring at least two fields to be classified, and comparing the semantic sensitivity of each field to be classified pairwise to obtain an original preference relationship; sorting each field to be classified according to the original preference relationship to obtain a sensitivity ranking, and determining the sensitivity level corresponding to each field to be classified based on the sensitivity ranking.

[0016] The field sensitivity grading method of this application obtains at least two fields to be graded and compares the semantic sensitivity of each field pairwise to obtain the original preference relationship. The model then judges the relative level of semantic sensitivity between the two fields each time and sorts the fields to be graded according to the original preference relationship to obtain the sensitivity ranking. Based on the sensitivity ranking, the sensitivity level corresponding to each field to be graded is determined. Since the pairwise comparison yields the relative level of sensitivity between the two fields, this relative level does not depend on the input order of the fields, and the ranking result is determined by all pairwise comparison results. This allows the same field to obtain a consistent sensitivity level under different input orders or different batches, solving the problem that the existing model's classification of the current field is affected by the input fields, resulting in the same semantic field being assigned different sensitivity levels under different input orders or different batches. Attached Figure Description

[0017] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0018] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1This is a schematic diagram of the device structure of the hardware operating environment involved in the embodiments of this application; Figure 2 This is a flowchart illustrating the first embodiment of the field sensitivity classification method of this application; Figure 3 This is a flowchart illustrating the second embodiment of the field sensitivity classification method of this application; Figure 4 This is a flowchart illustrating the third embodiment of the field sensitivity grading method of this application; Figure 5 This is a schematic diagram of the overall process of the field sensitivity classification method in this application; Figure 6 This is a structural block diagram of the field sensitivity grading device in this application.

[0020] The purpose, features, and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0021] It should be understood that the specific embodiments described herein are merely illustrative of the technical solutions of this application and are not intended to limit this application.

[0022] Reference Figure 1 , Figure 1 This is a schematic diagram of the device structure of the hardware operating environment involved in the embodiments of this application.

[0023] like Figure 1 As shown, the sensitivity grading device may include: a processor 1001, such as a central processing unit (CPU), a communication bus 1002, a user interface 1003, a network interface 1004, and a memory 1005. The communication bus 1002 is used to establish communication between these components. The user interface 1003 may be connected to a display screen; optionally, the user interface 1003 may include a standard wired interface or a wireless interface. In this application, the wired interface of the user interface 1003 may be a USB interface. The network interface 1004 may optionally include a standard wired interface or a wireless interface (such as a Wireless-Fidelity (Wi-Fi) interface). The memory 1005 may be high-speed random access memory (RAM) or non-volatile memory (NVM), such as a disk drive. The memory 1005 may also optionally be a storage device independent of the aforementioned processor 1001.

[0024] Those skilled in the art will understand that Figure 1The structure shown does not constitute a limitation on the device and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0025] like Figure 1 As shown, the memory 1005, which is identified as a computer storage medium, may include an operating system, a network communication module, a user interface module, and a field sensitivity classification program.

[0026] exist Figure 1 In the device shown, the network interface 1004 is mainly used to connect to the backend server and communicate with the backend server; the user interface 1003 is mainly used to connect to the user equipment; the device calls the field sensitivity classification program stored in the memory 1005 through the processor 1001 and executes the steps of the field sensitivity classification method provided in the embodiments of this application.

[0027] It should be noted that, under data security compliance requirements, enterprises need to classify the massive fields in the database into sensitivity levels (such as L1–L5) to support governance measures such as data anonymization and access control.

[0028] Current mainstream solutions employ field-by-field grading: the metadata (field name, comments, sample values, etc.) of a single field is input into a large language model, which then directly outputs a discrete sensitivity level. However, due to the contextual memory effect in large language models, when multiple fields are input sequentially or in batches, the model's grading of the current field is influenced by previously input fields. This results in fields with the same semantic meaning being assigned different sensitivity levels under different input orders or batches, leading to semantic drift.

[0029] Therefore, in order to solve the above-mentioned defects, this embodiment provides a method, apparatus, device and storage medium for classifying field sensitivity. The method includes: obtaining at least two fields to be classified, and comparing the semantic sensitivity of each field to be classified pairwise to obtain an original preference relationship; sorting each field to be classified according to the original preference relationship to obtain a sensitivity ranking, and determining the sensitivity level corresponding to each field to be classified according to the sensitivity ranking.

[0030] The field sensitivity grading method in this embodiment obtains at least two fields to be graded and compares the semantic sensitivity of each field pairwise to obtain the original preference relationship. The model then judges the relative level of semantic sensitivity between the two fields each time and sorts the fields to be graded according to the original preference relationship to obtain the sensitivity ranking. Based on the sensitivity ranking, the sensitivity level corresponding to each field to be graded is determined. Since the pairwise comparison yields the relative level of sensitivity between the two fields, this relative level does not depend on the input order of the fields, and the ranking result is determined by all pairwise comparison results. This allows the same field to obtain a consistent sensitivity level under different input orders or different batches, solving the problem that the existing model's classification of the current field is affected by the input fields, resulting in the same semantic field being assigned different sensitivity levels under different input orders or different batches.

[0031] For ease of understanding, the following is combined with Figures 2 to 6 The field sensitivity classification method provided in the embodiments of this application will be described in detail.

[0032] Reference Figure 2 , Figure 2 This is a flowchart illustrating the first embodiment of the field sensitivity classification method of this application. The first embodiment of the field sensitivity classification method of this application is presented as follows: Figure 2 As shown, in this embodiment, the method includes: Step S10: Obtain at least two fields to be classified, and compare the semantic sensitivity of each field to be classified pairwise to obtain the original preference relationship.

[0033] It is understood that the method in this embodiment can be applied to the above-mentioned field sensitivity classification device (hereinafter referred to as classification device). The classification device can be any device with data processing, program running and data acquisition functions, such as a computer, server, etc., and there is no limitation in this embodiment.

[0034] It should be noted that the fields to be classified can be fields in the database that need to be classified into sensitivity levels, such as columns in a database table. There can be one or more fields to be classified, and there are no specific restrictions on this.

[0035] It should also be noted that the aforementioned semantic sensitivity can be the degree of harm that the data carried by the field may cause after leakage or misuse. This degree of harm can be determined by the semantic information contained in the field's metadata, which may include, for example, field name, field comments, data type, and desensitized sample values.

[0036] It is understood that the aforementioned original preference relation can be a relative judgment result of the relative sensitivity between two fields to be classified, obtained by comparing the semantic sensitivity of at least two fields to be classified pairwise. The original preference relation can indicate the field with higher sensitivity and the field with lower sensitivity among the two fields to be classified. For example, the original preference relation can indicate that the semantic sensitivity of the first field to be classified is higher than that of the second field to be classified, or it can indicate that the semantic sensitivity of the second field to be classified is higher than that of the first field to be classified, depending on the actual comparison result.

[0037] In its implementation, the aforementioned classification device acquires at least two fields to be classified and performs pairwise comparisons of the semantic sensitivity of each field to obtain the original preference relationship. This pairwise comparison can involve forming any two fields from the at least two fields to be classified into a field pair and comparing the semantic sensitivity of the two fields in that pair. This pairwise comparison can be performed between multiple field pairs. When the number of fields to be classified is N, N(N-1) / 2 field pairs can be formed. The classification device then compares the semantic sensitivity of the two fields in each field pair to obtain the original preference relationship for each field pair. This original preference relationship is used to characterize the relative levels of semantic sensitivity between the two fields to be classified in the corresponding field pair.

[0038] Step S20: Sort each of the fields to be classified according to the original preference relationship to obtain a sensitivity ranking, and determine the sensitivity level corresponding to each of the fields to be classified according to the sensitivity ranking.

[0039] It should be understood that the above sorting can be a process of arranging the fields to be classified according to their semantic sensitivity from high to low or from low to high, based on the original preference relationships between them. The sensitivity sorting can be the result of the order of the fields to be classified after the sorting process.

[0040] It should also be understood that the above sensitivity level can be a level identifier obtained after sorting and dividing the sensitivity. This level identifier is used to characterize the sensitivity level of the field. The number of sensitivity levels can be set according to the actual application scenario, such as five levels from L1 to L5. This embodiment does not impose specific restrictions on this.

[0041] In its implementation, the aforementioned classification device sorts each field to be classified according to its original preference relationships to obtain a sensitivity ranking, and then determines the sensitivity level corresponding to each field to be classified based on this ranking. Specifically, after obtaining the original preference relationships corresponding to each field pair, the classification device sorts all the fields to be classified according to the semantic sensitivity relationship between the two fields to be classified in each field pair indicated by the original preference relationships. This results in the fields to be classified being arranged in descending or ascending order of semantic sensitivity, forming a sensitivity ranking. Subsequently, the classification device determines the sensitivity level corresponding to each field to be classified based on this sensitivity ranking.

[0042] In one specific implementation, the grading device, after obtaining the sensitivity ranking, acquires a preset number of levels. This number of levels can be pre-set according to the actual application scenario, for example, five levels. Based on this number of levels, the grading device divides each field to be graded in the sensitivity ranking into several level intervals corresponding to the number of levels. The division method can be to assign levels to the sorted fields in order from front to back, so that fields ranked earlier correspond to higher sensitivity levels and fields ranked later correspond to lower sensitivity levels, or vice versa. The grading device determines the sensitivity level corresponding to each field based on its position within the level interval in the sensitivity ranking.

[0043] In another specific implementation, the grading device can obtain at least one pre-defined anchor field and the reference sensitivity level corresponding to each anchor field. The anchor field is a field with a pre-defined reference sensitivity level. The grading device compares and sorts each anchor field and each field to be graded pairwise, obtaining an overall sensitivity ranking that includes both the anchor fields and the fields to be graded. Subsequently, the grading device determines the sensitivity level division boundary based on the ranking position of each anchor field in this overall sensitivity ranking and its corresponding reference sensitivity level. Then, based on the position of each field to be graded relative to this division boundary, it determines the sensitivity level corresponding to each field to be graded. In this way, the grading of each field to be graded is constrained by the reference sensitivity level of the anchor field, avoiding the forced allocation of fields that should all be in low-level ranges to high-level ranges.

[0044] This embodiment obtains at least two fields to be classified and performs pairwise comparisons of the semantic sensitivity of each field to obtain the original preference relationship. The sensitivity ranking is obtained based on the original preference relationship, and the sensitivity level is determined based on the sensitivity ranking. Since the pairwise comparisons yield the relative high and low relationship of sensitivity between the two fields, this relative high and low relationship does not depend on the input order of the fields, and the ranking result is determined by all pairwise comparison results. This can eliminate the single-time ranking drift caused by the model context memory effect, and thus enable the same field to obtain a consistent sensitivity level under different input orders or different batches. This solves the problem in the existing scheme that the same semantic field is assigned different sensitivity levels under different input orders or different batches due to the context memory effect.

[0045] Reference Figure 3 , Figure 3 This is a flowchart illustrating the second embodiment of the field sensitivity classification method of this application. Based on the first embodiment described above, a second embodiment of the field sensitivity classification method of this application is proposed.

[0046] To further improve the accuracy of semantic sensitivity comparison and address the difficulty of direct global pairwise comparisons in multi-field, complex metadata scenarios, this embodiment includes the following steps: obtaining at least two fields to be classified and performing pairwise comparisons of the semantic sensitivity of each field to obtain the original preference relationship. Step S101: Obtain the field metadata of at least two fields to be classified, verify the field metadata of each field, and map the field metadata that passes the verification to field descriptors.

[0047] It should be noted that the aforementioned field metadata can be data describing field attributes. This field metadata may include at least one of the following: field name, field comment, data type, and desensitized sample value. For example, the field name is "user_id", the field comment is "user unique identifier", the data type is "VARCHAR(64)", and the desensitized sample value is "U123". "wait.

[0048] It is understandable that the above validation can be a validity check on field metadata. This validity check can include checking whether the field metadata is complete, conforms to preset format requirements, and contains valid content. For example, the validation can check whether the field name is empty or whether the field comment exceeds the preset length limit. The above field descriptor can be a standardized data object obtained by mapping the validated field metadata according to a preset data structure. For example, the field descriptor can contain structured data fields such as field identifier, field name, comment, type code, and sample value.

[0049] In its implementation, the aforementioned hierarchical device acquires the field metadata of at least two fields to be hierarchically classified, verifies each field's metadata, and maps the verified field metadata to field descriptors. Specifically, the hierarchical device reads the field metadata of each field to be hierarchically classified from a database system or metadata repository. After acquiring the field metadata, the hierarchical device verifies each field's metadata according to preset verification rules. These rules may include checking whether the field metadata contains a valid field name, checking whether field comments contain abnormal characters, and checking whether the data type is a supported type. The hierarchical device marks the field to be hierarchically classified corresponding to the field metadata that fails verification as an abnormal field and excludes the abnormal field from subsequent processing. The hierarchical device maps the verified field metadata according to a preset field descriptor format to obtain the field descriptors corresponding to each field to be hierarchically classified.

[0050] Step S102: Perform vectorization encoding on each of the field descriptors to generate the embedding vector corresponding to each of the fields to be classified.

[0051] It is important to understand that the aforementioned vectorization encoding can be a process of converting text data into a numerical vector representation, which is used to characterize the semantic information of the text data. The aforementioned embedding vector can be a numerical vector obtained through vectorization encoding, which is used to represent the semantic features of the corresponding field to be classified in the vector space.

[0052] In its implementation, the aforementioned grading device performs vectorized encoding on each field descriptor, generating an embedding vector corresponding to each field to be graded. Specifically, the grading device obtains the field descriptor corresponding to each field to be graded, which contains text information such as field name and field comments. The grading device combines the text information in the field descriptors to obtain the text string corresponding to each field to be graded. The grading device inputs the text string into a preset embedding model, which performs semantic understanding on the text string and maps it into a numerical vector. The grading device receives the numerical vector output by the embedding model and uses this numerical vector as the embedding vector for the corresponding field to be graded.

[0053] Step S103: Based on the semantic similarity between the embedded vectors, determine at least one pair of fields to be compared from the fields to be classified.

[0054] It should be understood that the semantic similarity mentioned above can be the distance or angle between two embedding vectors in the vector space. This distance or angle is used to measure the semantic similarity between the fields to be classified corresponding to the two embedding vectors. Semantic similarity can be calculated using methods such as cosine similarity, Euclidean distance, or dot product. This embodiment does not impose specific limitations on this. The field pair to be compared mentioned above can be a pair of fields selected from the fields to be classified.

[0055] In its implementation, the aforementioned grading device determines at least one pair of fields to be compared from the fields to be graded based on the semantic similarity between each embedding vector. Specifically, after obtaining the embedding vectors corresponding to each field to be graded, the grading device calculates the semantic similarity between each pair of embedding vectors. Based on the calculated semantic similarity, the grading device selects two fields to be graded that meet preset conditions to form a pair of fields to be compared. The preset conditions may include selecting the top N field pairs with the highest semantic similarity as the pair of fields to be compared, or selecting field pairs with semantic similarity exceeding a preset similarity threshold as the pair of fields to be compared, or selecting a subset of field pairs from field pairs with a certain degree of semantic similarity according to a preset sampling strategy.

[0056] Step S104: Assemble the field names and annotation information of the corresponding fields to be graded in each pair of fields to be compared into comparison prompt words to instruct the preset language model to perform sensitivity comparison.

[0057] It should be noted that the comparison prompts mentioned above can be text inputs used to instruct the preset language model to compare the semantic sensitivity of two fields to be classified. The comparison prompts can include the field names and annotation information corresponding to the two fields to be classified, so that the preset language model can determine the sensitivity relationship between the two fields to be classified based on the semantics implied by the field names and annotation information.

[0058] It should also be noted that the aforementioned preset language model can be a pre-trained large language model with natural language understanding and generation capabilities, which can receive text input and output corresponding judgment results.

[0059] In its implementation, the aforementioned classification device assembles the field names and annotation information of the corresponding fields to be classified in each pair of fields to be compared into comparison prompt words to instruct the preset language model to compare their sensitivity levels. Specifically, for each pair of fields to be compared, the classification device obtains the field names and annotation information corresponding to the two fields to be classified in that pair. Following a preset prompt word template, the classification device fills the corresponding positions in the prompt word template with the field names and annotation information of the two fields to be classified, generating comparison prompt words to instruct the preset language model to compare the sensitivity levels of the two fields to be classified. These comparison prompt words instruct the preset language model to determine which of the two fields to be classified has higher semantic sensitivity based on the semantic information implied in the field names and annotation information.

[0060] Step S105: Input each of the comparison prompt words into the preset language model, and compare each pair of fields to be compared through the preset language model to obtain the original preference relationship.

[0061] In its implementation, the aforementioned classification device inputs each comparison prompt word into a preset language model. The preset language model then compares each pair of fields to be compared to obtain the original preference relationship. Specifically, the classification device inputs the comparison prompt words generated for each pair of fields to be compared into the preset language model. Upon receiving the comparison prompt words, the preset language model performs semantic understanding on the field names and annotation information of the two fields to be classified carried in the comparison prompt words. Based on the semantic understanding results, it determines which of the two fields to be classified has higher semantic sensitivity and then outputs the corresponding judgment result. The classification device receives the judgment result returned by the preset language model and uses this judgment result as the original preference relationship corresponding to the pair of fields to be compared.

[0062] This embodiment acquires the metadata of at least two fields to be classified, validates each field's metadata, maps the validated metadata to field descriptors, vectorizes each field descriptor to generate embedding vectors, and determines at least one pair of fields to be compared from the fields to be classified based on the semantic similarity between the embedding vectors. The corresponding field names and annotation information in each pair are assembled into comparison prompts, which are then input into a preset language model. The preset language model compares the pairs of fields to be compared to obtain the original preference relationships, thus achieving automated filtering of field pairs with comparative value from a large number of fields to be classified. Since it is unnecessary to compare all possible field pairs pairwise, only those whose semantic similarity meets the preset conditions need to be compared, the number of pairwise comparisons is significantly reduced while ensuring that the comparison results cover the relative ranking relationships between semantically similar fields. Simultaneously, validation excludes abnormal fields from subsequent processing, avoiding interference from invalid or abnormal field metadata, further improving the accuracy and efficiency of the comparison.

[0063] Furthermore, to improve the accuracy of sensitivity ranking, in this embodiment, the step of ranking each of the fields to be classified according to the original preference relationship includes: Step S201: Obtain at least one nearest neighbor field of the first field to be classified in the original preference relation, and determine the proximity semantic similarity between the nearest neighbor field and the first field to be classified.

[0064] It should be noted that the aforementioned nearest neighbor field can be a field to be classified whose semantic similarity with the first field to be classified meets a preset condition. For example, the nearest neighbor field can be the top K fields to be classified with the highest semantic similarity with the first field to be classified, or it can be a field to be classified whose semantic similarity with the first field to be classified exceeds a preset similarity threshold. This embodiment does not impose specific restrictions on this.

[0065] It should also be noted that the aforementioned proximity semantic similarity can be the semantic similarity between the first field to be classified and the nearest neighbor field. This proximity semantic similarity is used to determine the degree of semantic similarity between the nearest neighbor field and the first field to be classified.

[0066] In its specific implementation, the aforementioned grading device acquires at least one nearest neighbor field of the first field to be graded in the original preference relation and determines the semantic similarity between the nearest neighbor field and the first field to be graded. The grading device acquires the original preference relation, which contains the comparison results of at least one pair of fields, where the two fields to be graded in the pair have a semantic sensitivity relationship. The grading device determines the first field to be graded from the original preference relation; this first field to be graded is the field in the original preference relation with a higher sensitivity than the second field to be graded. Based on the embedding vectors of each field to be graded, the grading device calculates the semantic similarity between the first field to be graded and all other fields to be graded. Based on the calculated semantic similarity, the grading device acquires at least one field to be graded that meets a preset nearest neighbor condition from the other fields to be graded as the nearest neighbor field of the first field to be graded, and determines the semantic similarity between this nearest neighbor field and the first field to be graded. This preset nearest neighbor condition may include a semantic similarity ranking in the top K positions, or a semantic similarity exceeding a preset similarity threshold.

[0067] Step S202: When the semantic similarity of the nearest neighbor exceeds a preset similarity threshold, a derived preference relationship is generated between the nearest neighbor field and the second field to be classified in the original preference relationship, wherein the sensitivity of the first field to be classified is greater than the sensitivity of the second field to be classified.

[0068] It is understood that the aforementioned preset similarity threshold can be a pre-set threshold for determining whether two fields are semantic neighbors. This preset similarity threshold can be flexibly set according to the actual application scenario, for example, it can be set to 0.70 or 0.75. This embodiment does not impose any specific restrictions on this.

[0069] It can also be understood that the aforementioned derived preference relationship can be a preference relationship of sensitivity between the neighboring field and the second field to be classified, derived from the existing original preference relationship between the first field to be classified and the second field to be classified, as well as the semantic similarity relationship between the first field to be classified and the neighboring field. This derived preference relationship is used to expand the coverage of the original preference relationship between semantically similar fields, and this derived preference relationship indicates that the semantic sensitivity of the neighboring field is higher than that of the second field to be classified.

[0070] In its implementation, when the aforementioned classification device determines that the semantic similarity of neighboring fields exceeds a preset similarity threshold, it generates a derived preference relationship between the neighboring field and the second field to be classified in the original preference relationship. The sensitivity of the first field to be classified is greater than the sensitivity of the second field to be classified. Specifically, the classification device compares the semantic similarity of neighboring fields with the preset similarity threshold. When the semantic similarity exceeds the preset similarity threshold, it determines that the neighboring field and the first field to be classified are semantically similar. Based on the fact that the semantic sensitivity of the first field to be classified is higher than that of the second field to be classified, and the semantic similarity relationship between the neighboring field and the first field to be classified, the classification device deduces that the semantic sensitivity of the neighboring field is higher than that of the second field to be classified, and generates a derived preference relationship between the neighboring field and the second field to be classified. This derived preference relationship indicates that the semantic sensitivity of the neighboring field is higher than that of the second field to be classified.

[0071] For ease of understanding, the following explanation uses examples, but does not impose specific limitations on this embodiment. The classification device obtains an initial preference relation indicating that the semantic sensitivity of field A (id_card_number) is higher than that of field B (user_name), where field A is the first field to be classified, and field B is the second field to be classified. The classification device determines that field C (user_id) is a nearest neighbor field of field A and determines that the semantic similarity between field C and field A is 0.78. The classification device obtains a preset similarity threshold of 0.70 and compares the semantic similarity of 0.78 with the preset similarity threshold. Since 0.78 is greater than 0.70, field C and field A are determined to be semantically similar. Based on the fact that the semantic sensitivity of field A is higher than that of field B, and that field C is semantically similar to field A, the classification device deduces that the semantic sensitivity of field C is higher than that of field B, generating a derived preference relation between field C and field B. This derived preference relation indicates that the semantic sensitivity of field C is higher than that of field B.

[0072] Step S203: Sort each of the fields to be classified according to the original preference relationship and the derived preference relationship to obtain the sensitivity ranking.

[0073] In its implementation, the aforementioned hierarchical device sorts each field to be classified based on its original preference relationships and derived preference relationships. Specifically, the hierarchical device obtains the original preference relationships corresponding to each field pair, as well as the derived preference relationships between each nearest neighbor field and its corresponding second field to be classified. The hierarchical device uses both the original and derived preference relationships as the sorting criteria, and sorts all the fields to be classified as a whole according to the semantic sensitivity relationship between the two fields to be classified in each field pair indicated by each preference relationship. This results in the fields to be classified being arranged in descending or ascending order of semantic sensitivity, thus obtaining a sensitivity ranking.

[0074] This embodiment obtains the nearest neighbor fields of the first field to be classified in the original preference relation, determines the semantic similarity between the nearest neighbor field and the first field to be classified, and generates a derived preference relation between the nearest neighbor field and the second field to be classified when the semantic similarity exceeds a preset similarity threshold. The fields to be classified are then sorted according to the original preference relation and the derived preference relation. This expands the original preference relation to more fields along the semantic proximity relation without increasing the number of calls to the preset language model. Since the nearest neighbor field of the first field to be classified is semantically similar to the first field to be classified, the sensitivity relationship between the first and second fields to be classified can be passed to the nearest neighbor field. This allows the preference relation between the nearest neighbor field and the second field to be classified to be obtained without actual comparison using the preset language model, significantly expanding the coverage of the preference relation among semantically similar fields, making the sorting basis more sufficient, and thus improving the accuracy of sensitivity sorting.

[0075] To improve the accuracy and reliability of sensitivity ranking, in this embodiment, the step of sorting each field to be classified according to the original preference relationship and the derived preference relationship to obtain the sensitivity ranking includes: Step S2031: Determine the win / loss frequency between each of the fields to be classified based on the original preference relationship and the derived preference relationship, and construct an initial preference statistics matrix based on each win / loss frequency.

[0076] It should be noted that the aforementioned win-loss frequency can be the cumulative number of times that one of the two fields to be classified is better than the other in the original preference relationship and the derived preference relationship. For example, in multiple preference relationships, if field A is better than field B a certain number of times, then the win-loss frequency of field A relative to field B is that certain number of times.

[0077] It should also be noted that the aforementioned initial preference statistics matrix can be a matrix used to store win-loss frequency information between each field to be classified. Each element in the matrix represents the win-loss frequency of the field to be classified indicated by the corresponding row relative to the field to be classified indicated by the corresponding column.

[0078] In its implementation, the aforementioned grading device determines the win-loss frequency between each field to be graded based on the original and derived preference relationships, and constructs an initial preference statistics matrix based on these frequencies. The grading device obtains the original and derived preference relationships, each indicating the relative semantic sensitivity between the corresponding two fields to be graded. The grading device calculates the cumulative number of times one field is superior to the other in any two fields across all preference relationships, obtaining the win-loss frequency between each field. The grading device establishes an initial preference statistics matrix according to the numbering order of the fields to be graded, with rows and columns corresponding to each field. The grading device then fills the win-loss frequencies between each field into the corresponding rows and columns of this matrix, completing the construction of the initial preference statistics matrix.

[0079] Step S2032: Normalize each row of the initial preference statistics matrix to obtain the target preference statistics matrix.

[0080] It should be understood that the above normalization can be a process of proportionally transforming the elements of each row in the matrix according to the sum of the elements in that row, so that the value of each element in that row after transformation is between 0 and 1 and the sum of the elements in that row is equal to 1. For example, dividing each element in a row by the sum of all the elements in that row yields the normalized value of each element.

[0081] It should also be understood that the aforementioned target preference statistics matrix can be a matrix obtained by normalizing each row of the initial preference statistics matrix. The element values ​​of each row in the target preference statistics matrix represent the preference intensity distribution of the corresponding field to be classified relative to other fields to be classified.

[0082] In its implementation, the aforementioned grading device normalizes each row of the initial preference statistics matrix to obtain the target preference statistics matrix. The grading device obtains the initial preference statistics matrix, where each row corresponds to a field to be graded, and each element in the row represents the win / loss frequency of that field relative to other fields to be graded. For each row, the grading device calculates the sum of all elements in that row and divides each element in that row by the sum, making the values ​​of each element in that row proportional values ​​between 0 and 1, and ensuring that the sum of all elements in that row equals 1. The grading device performs the above normalization operation on each row of the initial preference statistics matrix and combines the normalized rows into the target preference statistics matrix.

[0083] Step S2033: Sort each of the fields to be classified according to the initial sensitivity score vector of each field to be classified and the target preference statistical matrix to obtain the sensitivity ranking.

[0084] It should be understood that the aforementioned initial sensitivity score vector can be a vector composed of the initial sensitivity scores of each field to be classified. Each element in the initial sensitivity score vector corresponds to the initial sensitivity score of a field to be classified. The initial sensitivity score can be an initial value assigned to each field to be classified before the iterative calculation begins. This initial value can be used to characterize the sensitivity estimate of the field to be classified in the initial state.

[0085] In its implementation, the aforementioned grading device sorts each field to be graded based on its initial sensitivity score vector and the target preference statistical matrix to obtain a sensitivity ranking. Specifically, the grading device obtains the initial sensitivity score vector for each field to be graded. Each element in this initial sensitivity score vector can be assigned according to a preset method, such as setting all initial sensitivity scores for each field to be graded to the same value, or differentiating the initial sensitivity scores of some fields according to their semantic features. This embodiment does not impose specific limitations on this. The grading device performs a matrix-vector multiplication operation between the initial sensitivity score vector and the target preference statistical matrix to obtain the updated sensitivity scores for each field to be graded. The grading device sorts each field to be graded from high to low or from low to high based on its updated sensitivity scores to obtain a sensitivity ranking, where fields with higher updated sensitivity scores are ranked higher in the sensitivity ranking.

[0086] This embodiment determines the win / loss frequency between each field to be classified based on the original and derived preference relationships and constructs an initial preference statistics matrix. Each row of the initial preference statistics matrix is ​​normalized to obtain a target preference statistics matrix. The fields to be classified are then sorted according to their initial sensitivity score vectors and the target preference statistics matrix to obtain a sensitivity ranking. Since each row of the target preference statistics matrix, after normalization, forms a preference intensity distribution for that field relative to other fields, this distribution reflects the accumulated preference information of that field across all preference relationships. By applying this preference intensity distribution to the sensitivity scores of each field through matrix-vector multiplication, preference information can be transmitted and accumulated between fields according to the direction of the preference relationships. This allows the ranking result to comprehensively reflect the global preference information contained in all original and derived preference relationships, improving the accuracy and reliability of the sensitivity ranking.

[0087] To further improve the accuracy and reliability of sensitivity ranking, in this embodiment, the step of ranking each field to be classified based on its initial sensitivity score vector and the target preference statistical matrix to obtain the sensitivity ranking includes: Step S20331: The initial sensitivity score vectors of each field to be classified are weighted and summed according to the weights of the corresponding rows in the target preference statistics matrix to obtain the intermediate sensitivity score vectors of each field to be classified.

[0088] It should be noted that the aforementioned intermediate sensitivity score vector can be a vector composed of the sensitivity scores of each field to be classified, obtained by weighting and summing the initial sensitivity score vector according to the weights of the corresponding rows in the target preference statistical matrix. Each element in the intermediate sensitivity score vector corresponds to the intermediate sensitivity score of a field to be classified. The intermediate sensitivity score is used to characterize the sensitivity estimate of the field to be classified after the current round update during the iteration process.

[0089] In its implementation, the aforementioned grading device performs a weighted summation of the initial sensitivity score vectors of each field to be graded, based on the weights of the corresponding rows in the target preference statistical matrix, to obtain the intermediate sensitivity score vectors of each field. The grading device obtains the initial sensitivity score vectors of each field, where each element corresponds to the initial sensitivity score of that field. The grading device obtains the target preference statistical matrix, where the number of rows and columns equals the number of fields to be graded. For each row in the target preference statistical matrix, the grading device multiplies each weight of that row by the corresponding element in the initial sensitivity score vector and sums the products to obtain the intermediate sensitivity score of the corresponding field. The grading device performs the aforementioned weighted summation operation on each field to be graded to obtain the intermediate sensitivity score corresponding to each field, and these intermediate sensitivity scores constitute the intermediate sensitivity score vector.

[0090] Step S20332: Normalize each of the intermediate sensitivity score vectors to obtain the target sensitivity score vector corresponding to each of the fields to be classified.

[0091] It should be noted that the above-mentioned target sensitivity score vector can be a vector composed of the sensitivity scores of each field to be classified after normalizing the intermediate sensitivity score vector. Each element in the target sensitivity score vector corresponds to the target sensitivity score of a field to be classified, and the target sensitivity score is used as the input for the next iteration in the iteration process.

[0092] In its implementation, the aforementioned grading device normalizes the intermediate sensitivity score vectors of each field to be graded, obtaining the target sensitivity score vector corresponding to each field. The grading device obtains the intermediate sensitivity score vector, where each element corresponds to the intermediate sensitivity score of each field to be graded. The grading device calculates the sum of all elements in the intermediate sensitivity score vector and divides each element by the sum, ensuring that the normalized values ​​of each element are between 0 and 1 and the sum of all elements equals 1. The grading device then combines the normalized elements into the target sensitivity score vector.

[0093] Step S20333: Set the target sensitivity score vector as the new initial sensitivity score vector, and return to the step of weighted summation of the initial sensitivity score vectors of each field to be classified according to the weight of the corresponding row in the target preference statistics matrix, until the sum of the score changes of each field to be classified in two adjacent iterations is less than the preset iteration threshold.

[0094] It is understood that the aforementioned preset iteration threshold can be a pre-set critical value used to determine whether the iteration process has converged. This preset iteration threshold is used to compare with the sum of the score changes of each field to be graded in two adjacent iterations. When the sum of the score changes is less than the preset iteration threshold, the iteration is determined to have converged and the iteration is stopped.

[0095] In its implementation, the aforementioned grading device sets the target sensitivity score vector as the new initial sensitivity score vector and returns to the step of weighted summation of the initial sensitivity score vectors of each field to be graded according to the weights of the corresponding rows in the target preference statistical matrix, until the sum of the score changes of each field to be graded in two adjacent iterations is less than a preset iteration threshold. In each iteration, the grading device uses the target sensitivity score vector of the current round as the initial sensitivity score vector for the next round and re-executes the weighted summation and normalization operations to obtain a new target sensitivity score vector. After each iteration, the grading device calculates the absolute value of the difference between the corresponding elements of the target sensitivity score vector of the current round and the target sensitivity score vector of the previous round, and adds these absolute values ​​to obtain the sum of the score changes of each field to be graded in two adjacent iterations. The grading device compares this sum of score changes with a preset iteration threshold. If the sum of score changes is less than the preset iteration threshold, the iteration is considered converged and stopped; if the sum of score changes is greater than or equal to the preset iteration threshold, the next iteration continues.

[0096] Step S20334: Use the target sensitivity score vector of each field to be classified in the last iteration as the sensitivity score of each field to be classified.

[0097] It is understandable that the last iteration mentioned above can be determined to be the last iteration when the sum of the score changes of each field to be graded in two adjacent iterations is less than the preset iteration threshold.

[0098] It is also understood that the above sensitivity score can be obtained after the iteration converges, and the target sensitivity score vector of each field to be classified obtained in the last iteration can be used as the final sensitivity score of each field to be classified. This sensitivity score is used to characterize the semantic sensitivity estimate of each field to be classified under the combined effect of all preference relations.

[0099] In its implementation, the aforementioned grading device uses the target sensitivity score vector of each field to be graded from the last iteration as the sensitivity score of each field. After each iteration, the grading device determines whether the current iteration meets the iteration stopping condition, i.e., whether the sum of the score changes of each field to be graded in two adjacent iterations is less than a preset iteration threshold. When the sum of the score changes is determined to be less than the preset iteration threshold, the grading device stops iterating and designates the current iteration as the last iteration. The grading device obtains the target sensitivity score vector of each field to be graded obtained in the last iteration and uses each element of the target sensitivity score vector as the sensitivity score of the corresponding field to be graded. This sensitivity score reflects the semantic sensitivity estimate of each field to be graded under the joint constraints of all original preference relations and derived preference relations; a higher sensitivity score indicates a higher semantic sensitivity of the corresponding field to be graded.

[0100] Step S20335: Sort the corresponding fields to be classified according to each sensitivity score to obtain the sensitivity ranking.

[0101] In its implementation, the aforementioned grading device sorts the corresponding fields to be graded based on their sensitivity scores to obtain a sensitivity ranking. The grading device acquires the sensitivity score corresponding to each field to be graded and arranges them sequentially from highest to lowest sensitivity score. In this sensitivity ranking, fields with higher sensitivity scores are listed first, indicating that the semantic sensitivity of that field is higher than that of the fields listed after it. The grading device can also arrange the fields in ascending order of sensitivity scores; the specific sorting direction can be set according to the actual application scenario, and this embodiment does not impose specific restrictions on this.

[0102] This embodiment obtains intermediate sensitivity score vectors by weighted summation of the initial sensitivity score vectors of each field to be classified according to the weights of the corresponding rows in the target preference statistical matrix. These intermediate sensitivity score vectors are then normalized to obtain the target sensitivity score vector. The target sensitivity score vector is set as the new initial sensitivity score vector, and the weighted summation step is repeated until the sum of the score changes between two adjacent iterations is less than a preset iteration threshold. The target sensitivity score vector of the last iteration is used as the sensitivity score of each field to be classified. The corresponding fields to be classified are then sorted according to their sensitivity scores to obtain a sensitivity ranking. This achieves full transmission and global accumulation of preference information among all fields to be classified. Because the sensitivity scores of each field to be classified are transmitted to other fields according to the weight distribution of the corresponding rows in the target preference statistical matrix during each iteration, after multiple iterations, the sensitivity score of each field to be classified can comprehensively reflect the preference information from all other fields. This makes the ranking result no longer dependent on local information from a single comparison, but rather reflects a globally consistent ranking jointly determined by all original preference relationships and derived preference relationships, thereby improving the accuracy and reliability of the sensitivity ranking.

[0103] Reference Figure 4 , Figure 4 This is a flowchart illustrating the third embodiment of the field sensitivity classification method of this application. Based on the above embodiments, the third embodiment of the field sensitivity classification method of this application is proposed.

[0104] Considering that during semantic propagation, derived preference relations derived from semantic similarity may logically conflict with existing original preference relations or other derived preference relations, forming a contradiction loop, this contradiction loop can lead to self-contradictory preference information in the preference relation set, thus affecting the accuracy and reliability of subsequent sorting processing. To ensure the logical consistency of derived preference relations and avoid including contradictory preference information in subsequent processing, in this embodiment, after the step of generating the derived preference relation between the nearest neighbor field and the second field to be graded in the original preference relation, the following is also included: Step S212: Perform loop detection on the derived preference relations based on the historical original preference relation set and the historical derived preference relation set to obtain the detection result.

[0105] It should be noted that the aforementioned set of historical original preference relations can be the set of all original preference relations obtained by comparing each pair of fields to be compared using a preset language model before the current derived preference relation is generated. The aforementioned set of historical derived preference relations can also be the set of all derived preference relations generated and preserved through semantic propagation before the current derived preference relation is generated.

[0106] It should also be noted that the aforementioned cycle detection can be a process of checking the logical consistency between the candidate derived preference relation and the set of historical original preference relations, as well as the preference relations in the set of historical derived preference relations. This cycle detection is used to determine whether adding the candidate derived preference relation to the existing set of preference relations will create a contradictory cycle. A contradictory cycle refers to a cyclic pointing relationship in the preference relation set where field A is superior to field B, field B is superior to field C, and field C is superior to field A. The detection result can be the judgment result obtained after cycle detection, indicating whether the derived preference relation will create a contradictory cycle.

[0107] In its implementation, the aforementioned grading device performs loop detection on derived preference relationships based on the historical original preference relationship set and the historical derived preference relationship set, obtaining the detection result. The grading device acquires the currently generated derived preference relationship and the stored historical original preference relationship set and historical derived preference relationship set, which contain all preference relationships determined before the current derived preference relationship was generated. After adding the currently generated derived preference relationship to the historical original preference relationship set and the historical derived preference relationship set, the grading device checks for contradictory loops, i.e., checks for cyclic pointing relationships consisting of at least three fields to be graded, such as the first field to be graded having a higher sensitivity than the second field to be graded, the second field to be graded having a higher sensitivity than the third field to be graded, and the third field to be graded having a higher sensitivity than the first field to be graded. The grading device generates a detection result based on the check result, which indicates whether contradictory loops exist.

[0108] Step S222: If the detection result is a non-contradictory cycle, retain the derived preference relation and add the derived preference relation to the historical derived preference relation set.

[0109] It should be understood that the above retention can be a processing method that determines the derived preference relation as a valid preference relation and stores it when the detection result is a conflict-free cycle.

[0110] In its implementation, when the detection result obtained by the aforementioned hierarchical device indicates the absence of contradictory cycles, the hierarchical device retains the derived preference relation and adds it to the historical derived preference relation set. The hierarchical device obtains the cycle detection results; when the detection results indicate that there are no contradictory cycles between the current derived preference relation and the preferences in the historical original preference relation set and the historical derived preference relation set, the hierarchical device determines the derived preference relation as a valid preference relation. The hierarchical device stores this derived preference relation in the historical derived preference relation set, making it part of subsequent cycle detection and the historical derived preference relation set, for use in the cycle detection and sorting processing of newly generated derived preference relations.

[0111] This embodiment achieves effective verification of the logical consistency of derived preference relations by performing cycle detection on the derived preference relations after generation, based on the historical original preference relation set and the historical derived preference relation set. If the detection result shows no contradictory cycles, the derived preference relation is retained and added to the historical derived preference relation set. Since derived preference relations are derived based on semantically similar relationships, this derivation process may logically conflict with existing original or derived preference relations under certain circumstances, forming contradictory cycles. Directly including derived preference relations with contradictory cycles in subsequent ranking processes would lead to self-contradictory preference information in the preference relation set, thus affecting the accuracy and reliability of sensitivity ranking. Verifying derived preference relations through cycle detection, and retaining them only when there are no contradictory cycles, ensures that all preference relations participating in subsequent ranking are logically consistent, avoiding interference from contradictory preference information in the ranking results, thereby improving the accuracy and reliability of sensitivity ranking.

[0112] Considering that when obtaining the nearest neighbor fields of the first field to be classified in the original preference relation, it is necessary to determine the semantic similarity between each field to be classified in order to select the nearest neighbor fields, recalculating the semantic similarity every time the nearest neighbor fields are obtained will result in large computational overhead and low processing efficiency. In order to reduce the consumption of computational resources and enable the acquisition of nearest neighbor fields to be completed quickly, in this embodiment, before the step of obtaining at least one nearest neighbor field of the first field to be classified in the original preference relation, the following step is further included: Step S021: Obtain the field metadata of at least two fields to be classified, verify the field metadata of each field, and map the field metadata that passes the verification to field descriptors.

[0113] Step S022: Perform vectorization encoding on each of the field descriptors to generate the embedding vector corresponding to each of the fields to be classified.

[0114] Step S023: Determine the nearest neighbor set of each field to be classified based on the semantic similarity between the embedded vectors.

[0115] It should be noted that the aforementioned nearest neighbor set can be a set of multiple fields to be classified whose semantic similarity with a certain field to be classified meets a preset condition. For example, the nearest neighbor set may include the top K fields to be classified with the highest semantic similarity, or it may include fields to be classified whose semantic similarity exceeds a preset similarity threshold.

[0116] In its implementation, the aforementioned hierarchical device acquires the field metadata of at least two fields to be hierarchically classified, verifies each field's metadata, and maps the verified field metadata to field descriptors. The hierarchical device reads the field metadata of each field to be hierarchically classified from a database system or metadata repository, and verifies each field's metadata according to preset verification rules. These verification rules may include checking whether the field metadata contains valid field names, checking whether field comments contain abnormal characters, and checking whether the data type is a supported type.

[0117] Subsequently, the grading device marks the fields to be graded corresponding to the field metadata that failed verification as abnormal fields and excludes them from subsequent processing. The grading device maps the verified field metadata according to a preset field descriptor format to obtain the field descriptors corresponding to each field to be graded. Then, the grading device performs vectorization encoding on each field descriptor to generate an embedding vector for each field to be graded. The grading device combines the text information in the field descriptors to obtain the text string corresponding to each field to be graded. This text string is input into a preset embedding model, which performs semantic understanding of the text string and maps it into a numerical vector. The grading device receives the numerical vector output by the embedding model and uses this numerical vector as the embedding vector for the corresponding field to be graded.

[0118] Next, the classification device determines the nearest neighbor set of each field to be classified based on the semantic similarity between each embedding vector. The classification device calculates the semantic similarity between each pair of embedding vectors, and for each field to be classified, it selects other fields to be classified that meet the preset nearest neighbor conditions to form the nearest neighbor set of the field to be classified based on the semantic similarity between the field to be classified and other fields to be classified.

[0119] The step of obtaining at least one nearest neighbor field of the first field to be classified in the original preference relation includes: Step S2011: Obtain at least one nearest neighbor field from the nearest neighbor set of the first field to be classified.

[0120] In a specific implementation, the aforementioned classification device obtains at least one nearest neighbor field from the nearest neighbor set of the first field to be classified. The classification device obtains the nearest neighbor set corresponding to the first field to be classified. This nearest neighbor set is determined and stored in advance in the aforementioned steps based on the semantic similarity between each embedding vector. This nearest neighbor set includes other fields to be classified that are semantically similar to the first field to be classified. The classification device selects at least one field to be classified as a nearest neighbor field from this nearest neighbor set. The selection method can be to select sequentially from the nearest neighbor set according to semantic similarity from high to low, or to select randomly from the nearest neighbor set, or to select according to a preset sampling strategy. This embodiment does not impose specific limitations on this.

[0121] This embodiment obtains the field metadata of at least two fields to be classified before acquiring the nearest neighbor fields of the first field to be classified in the original preference relation, verifies and maps them to field descriptors, vectorizes each field descriptor to generate embedding vectors, determines the nearest neighbor set of each field to be classified based on the semantic similarity between the embedding vectors, and then obtains at least one nearest neighbor field from the nearest neighbor set of the first field to be classified. This achieves the pre-determination and storage of the nearest neighbor set of each field to be classified. Since the construction of the nearest neighbor set is completed independently before acquiring the nearest neighbor fields, when it is necessary to acquire the nearest neighbor field of any field to be classified later, it is only necessary to read it from the already stored nearest neighbor set of that field. There is no need to recalculate the semantic similarity between that field and all other fields for each acquisition of a nearest neighbor field, thereby significantly reducing the consumption of computing resources and improving the efficiency of acquiring nearest neighbor fields. Especially when there are a large number of fields to be classified, the above-mentioned pre-construction of the nearest neighbor set can effectively avoid the performance overhead caused by repeated calculations.

[0122] Reference Figure 5 , Figure 5 This is a schematic diagram of the overall process of the field sensitivity classification method in this application, as shown below. Figure 5 As shown, this application divides all processing steps into three layers according to different data processing stages: a priori layer, a pairwise inference layer, and a consistency aggregation layer. In the priori layer, firstly, the metadata of each field to be classified in the database is collected. Then, the metadata text of each field is converted into semantic vectors through vectorized embedding. Subsequently, a Top-K similarity graph is constructed based on the similarity between semantic vectors, providing semantic nearest neighbor references for subsequent comparison scheduling and semantic propagation. In the pairwise inference layer, the pairs of fields to be compared are extracted from the similarity graph according to the scheduling strategy. The large language model is called through pairwise comparison scheduling to perform pairwise comparisons of semantic sensitivity for each pair of fields to obtain the original preference relationship. Then, the original preference relationship is propagated to adjacent fields along the semantic nearest neighbor path through semantic propagation extension to generate derived preference relationships. In the consistency aggregation layer, all original and derived preference relations are aggregated to construct a transition matrix M. The matrix is ​​then solved using Rank-One Matrix Value Iteration (ROMVI) with exponential iteration. The global sensitivity vector obtained after convergence is used as the final sensitivity score for each field. Finally, the sensitivity scores are sorted, and an L1-L5 level mapping is performed to output the sensitivity level for each field to be graded. The entire process extends from field metadata input to sensitivity level output.

[0123] In addition, refer to Figure 6 , Figure 6 This is a structural block diagram of the field sensitivity grading device in this application. Figure 6As shown in the embodiments of this application, a field sensitivity classification device is also proposed, which includes: The relationship acquisition module 601 is used to acquire at least two fields to be classified and to compare the semantic sensitivity of each field to be classified pairwise to obtain the original preference relationship. The level determination module 602 is used to sort each of the fields to be classified according to the original preference relationship to obtain a sensitivity ranking, and to determine the sensitivity level corresponding to each of the fields to be classified according to the sensitivity ranking.

[0124] Furthermore, this application also proposes a storage medium, which is a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, it implements the steps of the field sensitivity classification method described above.

[0125] Other embodiments or specific implementations of the field sensitivity grading device described in this example can be found in the above-described method embodiments, and will not be repeated here.

[0126] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or system that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or system. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or system that includes that element.

[0127] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0128] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as a read-only memory image (ROM) / random access memory (RAM), magnetic disk, optical disk), and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0129] The above are merely preferred embodiments of this application and do not limit the patent scope of this application. Any equivalent structural or procedural transformations made using the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.

Claims

1. A method for classifying field sensitivity, characterized in that, The method includes: Obtain at least two fields to be classified, and compare the semantic sensitivity of each field to be classified pairwise to obtain the original preference relationship; The fields to be classified are sorted according to the original preference relationship to obtain a sensitivity ranking, and the sensitivity level corresponding to each field to be classified is determined according to the sensitivity ranking.

2. The method as described in claim 1, characterized in that, The step of obtaining at least two fields to be classified and comparing the semantic sensitivity of each field to obtain the original preference relationship includes: Obtain the field metadata of at least two fields to be classified, validate each field metadata, and map the validated field metadata to field descriptors; The field descriptors are vectorized to generate the embedding vectors corresponding to the fields to be classified. Based on the semantic similarity between the embedded vectors, at least one pair of fields to be compared is determined from the fields to be classified. The field names and annotation information of the corresponding fields to be graded in each pair of fields to be compared are assembled into comparison prompt words to instruct the preset language model to perform sensitivity comparison. Each of the comparison prompt words is input into the preset language model, and the preset language model is used to compare each pair of fields to be compared to obtain the original preference relationship.

3. The method as described in claim 1, characterized in that, The step of sorting each of the fields to be classified according to the original preference relationship to obtain the sensitivity ranking includes: Obtain at least one nearest neighbor field of the first field to be classified in the original preference relation, and determine the proximity semantic similarity between the nearest neighbor field and the first field to be classified; When the semantic similarity of the nearest neighbor exceeds a preset similarity threshold, a derived preference relationship is generated between the nearest neighbor field and the second field to be classified in the original preference relationship, wherein the sensitivity of the first field to be classified is greater than the sensitivity of the second field to be classified. The fields to be classified are sorted according to the original preference relationship and the derived preference relationship to obtain the sensitivity ranking.

4. The method as described in claim 3, characterized in that, The step of sorting each of the fields to be classified according to the original preference relationship and the derived preference relationship to obtain the sensitivity ranking includes: Based on the original preference relationship and the derived preference relationship, the win-loss frequency between each of the fields to be classified is determined, and an initial preference statistics matrix is ​​constructed based on each win-loss frequency. Normalize each row of the initial preference statistics matrix to obtain the target preference statistics matrix; The sensitivity ranking is obtained by sorting each of the fields to be classified according to the initial sensitivity score vector of each field and the target preference statistics matrix.

5. The method as described in claim 4, characterized in that, The step of sorting each of the fields to be classified according to its initial sensitivity score vector and the target preference statistics matrix to obtain a sensitivity ranking includes: The initial sensitivity score vectors of each of the fields to be classified are weighted and summed according to the weights of the corresponding rows in the target preference statistics matrix to obtain the intermediate sensitivity score vectors of each of the fields to be classified. Normalize each of the intermediate sensitivity score vectors to obtain the target sensitivity score vector corresponding to each of the fields to be classified. Set the target sensitivity score vector as the new initial sensitivity score vector, and return to the step of performing a weighted summation of the initial sensitivity score vectors of each field to be classified according to the weight of the corresponding row in the target preference statistics matrix, until the sum of the score changes of each field to be classified in two adjacent iterations is less than a preset iteration threshold. The target sensitivity score vector of each field to be classified in the last iteration is used as the sensitivity score of each field to be classified. The corresponding fields to be classified are sorted according to the sensitivity scores to obtain the sensitivity ranking.

6. The method as described in claim 3, characterized in that, After the step of generating the derived preference relationship between the nearest neighbor field and the second field to be classified in the original preference relationship, the method further includes: Based on the historical original preference relation set and the historical derived preference relation set, loop detection is performed on the derived preference relation to obtain the detection result; If the detection result is a non-contradictory cycle, the derived preference relation is retained and added to the historical derived preference relation set.

7. The method as described in claim 3, characterized in that, Before the step of obtaining at least one nearest neighbor field of the first field to be classified in the original preference relation, the method further includes: Obtain the field metadata of at least two fields to be classified, validate each field metadata, and map the validated field metadata to field descriptors; The field descriptors are vectorized to generate the embedding vectors corresponding to the fields to be classified. Based on the semantic similarity between the embedding vectors, determine the nearest neighbor set of each field to be classified; The step of obtaining at least one nearest neighbor field of the first field to be classified in the original preference relation includes: Obtain at least one nearest neighbor field from the nearest neighbor set of the first field to be classified.

8. A field sensitivity grading device, characterized in that, The device includes: The relationship acquisition module is used to acquire at least two fields to be classified and to compare the semantic sensitivity of each field to be classified pairwise to obtain the original preference relationship. The level determination module is used to sort each of the fields to be classified according to the original preference relationship to obtain a sensitivity ranking, and to determine the sensitivity level corresponding to each of the fields to be classified according to the sensitivity ranking.

9. A field sensitivity grading device, characterized in that, The device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the field sensitivity grading method as described in any one of claims 1 to 7.

10. A storage medium, characterized in that, The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, it implements the steps of the field sensitivity classification method as described in any one of claims 1 to 7.

Images