Marking data deduplication method and device and storage medium

Abstract

The invention provides a beacon data deduplication method and device and a storage medium, relates to the technical field of data processing, and can improve the deduplication accuracy of beacon data. The method comprises the following steps: acquiring original beacon data, wherein the original beacon data comprises a plurality of first fields; based on the importance weight of each first field in the plurality of first fields and de-weighting the original beacon data, determining beacon data after preliminary de-weighting, the beacon data after preliminary de-weighting comprising a plurality of second fields; and based on the feature vector of each second field in the plurality of second fields and the cross-stage semantic migration network model, de-duplicating the beacon data after preliminary de-duplication.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a method, apparatus and storage medium for deduplicating tag data. Background Technology

[0002] With the rapid development of data processing technology, data deduplication is becoming increasingly widespread. In particular, bidding information data generated during the bidding process often contains duplicate fields, increasing the difficulty of managing this data. Therefore, a reasonable method for deduplicating bidding information data is crucial for the bidding process.

[0003] Currently, traditional deduplication methods for bidding information data mainly rely on static rule matching or shallow feature matching to remove duplicate fields. However, due to the increasing complexity and diversity of bidding information data, these traditional methods suffer from low deduplication accuracy. Summary of the Invention

[0004] This application provides a method, apparatus, and storage medium for deduplicating bidding data, which can improve the deduplication accuracy of bidding data.

[0005] To achieve the above objectives, this application adopts the following technical solution: In a first aspect, this application provides a method for deduplicating bidding information data. The method includes: acquiring original bidding information data, which includes multiple first fields; deduplicating the original bidding information data based on the importance weight of each of the multiple first fields to determine the initially deduplicated bidding information data, which includes multiple second fields; and deduplicating the initially deduplicated bidding information data based on the feature vector of each of the multiple second fields and a cross-stage semantic transfer network model.

[0006] The above technical solution brings at least the following beneficial effects: The deduplication method for bidding data provided in this application performs preliminary deduplication on the original bidding data by weighting the importance of each field in the original bidding data, determines the preliminary merged bidding data, and further deduplicates the bidding data based on the feature vector of each field in the bidding data and the cross-stage semantic transfer network model. Through two deduplications, duplicate fields in the original bidding data can be fully screened, improving the deduplication accuracy. In addition, the cross-stage semantic transfer network model provided in this application has a high recognition accuracy for duplicate field pairs, thereby improving the deduplication accuracy of the bidding data.

[0007] In one possible implementation, the original bidding data is deduplicated based on the importance weight of each of the multiple first fields to determine the initially deduplicated bidding data. This includes: constructing a hierarchical representation model of the original bidding data based on the importance weight of each field in the original bidding data, whereby the hierarchical representation model is used to determine the importance level of each field in the original bidding data; determining the comprehensive similarity between multiple fields of the first importance level in the hierarchical representation model based on a comprehensive similarity formula, where the first importance level is one of the multiple importance levels in the hierarchical representation model; and merging the first fields in the original bidding data with a comprehensive similarity greater than a first threshold into one field to determine the initially deduplicated bidding data.

[0008] In one possible implementation, the method further includes: determining the correlation between the third field and the fourth field based on the number of times the third field appears in multiple first fields and the number of times the fourth field appears in multiple first fields, wherein the third field and the fourth field are two fields at different importance levels in the hierarchical representation model; and determining whether to merge the first field and the second field into the same field based on the correlation and similarity between the first field and the second field.

[0009] In one possible implementation, based on the feature vectors of each of the multiple second fields and a cross-stage semantic transfer network model, the initially deduplicated tagging data is de-duplicated, including: determining the semantic similarity between the fifth and sixth fields based on the feature vectors of the fifth and sixth fields, and calculating the new value index of the fifth and sixth fields, where the fifth field is one of the multiple second fields and the sixth field is one of the multiple second fields; if the semantic similarity between the third and fourth fields is greater than a second threshold and the new value index of the third and fourth fields is less than a third threshold, the third and fourth fields are identified as a duplicate field pair; the duplicate field pairs in the initially deduplicated tagging data are merged into one field.

[0010] In one possible implementation, the method further includes: acquiring historical tag data; adding labels to fields in the historical tag data, the labels being used to indicate whether the fields in the historical tag data are duplicate fields; training an initial model based on the labeled historical tag data and the feature vectors of the historical tag data, and determining a cross-stage semantic transfer network model.

[0011] In one possible implementation, the comprehensive similarity formula satisfies the following formula:

[0012] in, For fields and The overall similarity between them This represents the total number of similarity measurement methods. For the first Weights of various similarity metrics For the first Fields under various measurement methods and Similarity score, and These are two different fields.

[0013] In one possible implementation, the field's new value index satisfies the following formula:

[0014] in, This represents the importance coefficient of the field in the industry. For fields semantic sparsity, This represents the time-based activity coefficient.

[0015] Secondly, this application provides a deduplication device for bidding information data, the device comprising: a communication unit and a processing unit; the communication unit is used to acquire original bidding information data, the original bidding information data including a plurality of first fields; the processing unit is used to deduplicate the original bidding information data based on the importance weight of each of the plurality of first fields, and determine the initially deduplicated bidding information data, the initially deduplicated bidding information data including a plurality of second fields; the processing unit is further used to deduplicate the initially deduplicated bidding information data based on the feature vector of each of the plurality of second fields and a cross-stage semantic transfer network model.

[0016] In one possible implementation, the processing unit is specifically used to: construct a hierarchical representation model of the original bidding data based on the importance weights of each field in the original bidding data, the hierarchical representation model being used to determine the importance level of each field in the original bidding data; determine the comprehensive similarity between multiple fields of the first importance level in the hierarchical representation model based on a comprehensive similarity formula, the first importance level being one of the multiple importance levels in the hierarchical representation model; and merge the first field in the original bidding data with a comprehensive similarity greater than a first threshold into one field to determine the bidding data after preliminary deduplication.

[0017] In one possible implementation, the processing unit is specifically used to: determine the correlation between the third field and the fourth field based on the number of times the third field appears in multiple first fields and the number of times the fourth field appears in multiple first fields, wherein the third field and the fourth field are two fields at different importance levels in the hierarchical representation model; and determine whether to merge the first field and the second field into the same field based on the correlation and similarity between the first field and the second field.

[0018] In one possible implementation, the processing unit is specifically used to: determine the semantic similarity between the fifth and sixth fields based on the feature vectors of the fifth and sixth fields, and calculate the new value index of the fifth field and the new value index of the sixth field, wherein the fifth field is one of multiple second fields and the sixth field is one of multiple second fields; if the semantic similarity between the third and fourth fields is greater than a second threshold and the new value index of the third field and the new value index of the fourth field are both less than a third threshold, then the third and fourth fields are identified as a duplicate field pair; and the duplicate field pairs in the initially deduplicated tag data are merged into one field.

[0019] In one possible implementation, the communication unit is further configured to acquire historical tag data; the processing unit is further configured to add labels to the fields of the historical tag data, the labels being used to indicate whether the fields of the historical tag data are duplicate fields; the processing unit is further configured to train an initial model based on the labeled historical tag data and the feature vectors of the historical tag data, and determine the cross-stage semantic transfer network model.

[0020] In one possible implementation, the comprehensive similarity formula satisfies the following formula:

[0021] in, For fields and The overall similarity between them This represents the total number of similarity measurement methods. For the first Weights of various similarity metrics For the first Fields under various measurement methods and Similarity score, and These are two different fields.

[0022] In one possible implementation, the field's new value index satisfies the following formula:

[0023] in, This represents the importance coefficient of the field in the industry. For fields semantic sparsity, This represents the time-based activity coefficient.

[0024] Thirdly, this application provides a deduplication device for bidding data, the device comprising: a processor and a communication interface; the communication interface and the processor are coupled, the processor being used to run computer programs or instructions to implement the deduplication method for bidding data as described in the first aspect and any possible implementation thereof.

[0025] Fourthly, this application provides a computer-readable storage medium storing instructions that, when executed on a terminal, cause the terminal to perform the deduplication method for identifier data as described in the first aspect and any possible implementation thereof.

[0026] Fifthly, this application provides a computer program product containing instructions that, when run on a deduplication device for flag data, causes the deduplication device to perform the deduplication method for flag data as described in the first aspect and any possible implementation thereof.

[0027] In a sixth aspect, this application provides a chip including a processor and a communication interface, the communication interface being coupled to the processor, the processor being used to run computer programs or instructions to implement the deduplication method for identifier data as described in the first aspect and any possible implementation thereof.

[0028] Specifically, the chip provided in this application also includes a memory for storing computer programs or instructions. Attached Figure Description

[0029] Figure 1 This application provides a schematic diagram of the structure of a tag data deduplication system. Figure 2 A schematic diagram illustrating the composition of a tag data deduplication device provided in this application embodiment; Figure 3 A flowchart illustrating a method for deduplicating tag data provided in this application embodiment; Figure 4 A flowchart illustrating another method for deduplicating tag data provided in this application embodiment; Figure 5 This is a schematic diagram of a deduplication device for tag data provided in an embodiment of this application. Detailed Implementation

[0030] The method, apparatus, and storage medium for deduplicating tag data provided in the embodiments of this application will now be described in detail with reference to the accompanying drawings.

[0031] In this article, the term "and / or" is merely a description of the relationship between related objects, indicating that there can be three relationships. For example, A and / or B can represent three situations: A exists alone, A and B exist simultaneously, and B exists alone.

[0032] The terms "first" and "second," etc., used in the specification and drawings of this application are used to distinguish different objects or to distinguish different treatments of the same object, rather than to describe a specific order of objects.

[0033] Furthermore, the terms "comprising" and "having," and any variations thereof, used in the description of this application are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not limited to the steps or units listed, but may optionally include other steps or units not listed, or may optionally include other steps or units inherent to such process, method, product, or apparatus.

[0034] It should be noted that in the embodiments of this application, the words "exemplary" or "for example" are used to indicate examples, illustrations, or explanations. Any embodiment or design scheme described as "exemplary" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design schemes. Specifically, the use of the words "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.

[0035] In the description of this application, unless otherwise stated, "a plurality of" means two or more.

[0036] With the rapid development of data processing technology, data deduplication is becoming increasingly widespread. In particular, bidding information data generated during the bidding process often contains duplicate fields, increasing the difficulty of managing this data. Therefore, a reasonable method for deduplicating bidding information data is crucial for the bidding process.

[0037] As described above regarding methods for deduplicating bidding data, traditional methods mainly use static rule matching or shallow feature matching to remove duplicate fields from the bidding data. However, due to the increasing complexity and diversity of bidding data, traditional methods suffer from low deduplication accuracy.

[0038] Tender documents, as key data in bidding activities, span multiple stages, including the tender announcement, supplementary announcements, bid results, and the announcement of the winning bid. The content of tender documents at these stages not only exhibits periodic repetition but also involves dynamic incremental changes in fields and semantic evolution. For example, the tender announcement and supplementary announcement may have partial overlap in fields such as budget amount and bid deadline, while the bid evaluation results may add specific information such as supplier qualifications and scoring criteria. This multi-stage dynamic evolution characteristic means that tender documents may contain both repetitive content and new information with stage-specific value across different stages, thus placing higher demands on deduplication efforts.

[0039] Meanwhile, different industries exhibit significant differences in their focus and priority regarding tender information fields, and the importance of fields within the same industry also dynamically adjusts with different stages. For example, the construction industry primarily focuses on the budget amount during the bidding stage, while the contract terms and performance cycle become more important during the winning bid stage; the medical industry prioritizes technical parameters and equipment brands. These dynamic changes in field priority further complicate the challenge of deduplicating tender information, requiring not only the differentiation between duplicate content across stages and newly added information, but also adaptation to dynamic field adjustments to ensure the accuracy of the deduplication results.

[0040] As described above regarding methods for deduplicating bidding data, traditional methods mainly use static rule matching or shallow feature matching to remove duplicate fields from the bidding data. However, due to the increasing complexity and diversity of bidding data, traditional methods suffer from low deduplication accuracy.

[0041] In view of this, this application proposes a method for deduplicating bidding data. This method performs initial deduplication of the original bidding data by assigning importance weights to each field, determining the initially merged bidding data. Then, it further deduplicates the bidding data based on the feature vectors of each field and a cross-stage semantic transfer network model. This two-stage deduplication process effectively filters out duplicate fields in the original bidding data, improving the deduplication accuracy. Furthermore, the cross-stage semantic transfer network model provided in this application has a high accuracy rate in identifying duplicate field pairs, thereby further improving the deduplication accuracy of the bidding data. The technical solutions provided in this application can be applied to various communication systems, such as New Radio (NR) communication systems using 5G, future evolution systems, or multiple communication convergence systems.

[0042] For example, Figure 1The diagram shows a schematic representation of a tag data deduplication system according to an embodiment of this application. The tag data deduplication system may include at least one data acquisition device 101 and at least one computing device 102.

[0043] In one possible implementation, the data acquisition device 101 is used to acquire raw tag data and send the raw tag data to the computing device 102.

[0044] In one possible implementation, computing device 102 is used to acquire original tag information data, which includes multiple first fields; based on the importance weight of each of the multiple first fields, the original tag information data is deduplicated to determine the initially deduplicated tag information data, which includes multiple second fields; based on the feature vector of each of the multiple second fields and a cross-stage semantic transfer network model, the initially deduplicated tag information data is deduplicated.

[0045] In one possible implementation, the data acquisition device 101 can be a dedicated device with data acquisition and transmission capabilities. It can be installed in various locations in terrestrial environments, including indoor spaces such as industrial plants and laboratories, or outdoor spaces such as field monitoring stations and farmland; it can be a handheld device easy for personnel to carry and operate, or it can be mounted on a vehicle for onboard data collection. It can also be placed on water, such as on a ship for marine environmental data collection. Furthermore, it can be carried on aerial platforms, such as on an aircraft, for data collection and long-distance data acquisition.

[0046] In one possible implementation, the computing device 102 can be a high-capacity device with wireless communication capabilities (e.g., a server integrating a large number of boards), which can be deployed on land, including indoors or outdoors, handheld or vehicle-mounted. It can also be deployed on water (such as on ships). It can also be deployed in the air (e.g., on airplanes, balloons, and satellites).

[0047] It should be noted that, Figure 1 This is just an example framework diagram. Figure 1 The number of nodes included and the names of the devices are unlimited, except for... Figure 1 In addition to the functional nodes shown, the deduplication system for beacon data may also include other nodes, such as core network equipment, and this application makes no restrictions on this.

[0048] The application scenarios of the embodiments in this application are not limited. The system architecture and business scenarios described in the embodiments of this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided by the embodiments of this application. As those skilled in the art will know, with the evolution of network architecture and the emergence of new business scenarios, the technical solutions provided by the embodiments of this application are also applicable to similar technical problems.

[0049] In practical implementation, Figure 1 All the equipment in the middle can be adopted Figure 2 The shown composition structure, or including Figure 2 The components shown. Figure 2 This is a schematic diagram illustrating the composition of a tag data deduplication device 20 provided in an embodiment of this application. The tag data deduplication device 20 can be a terminal device 101 or a chip or system-on-a-chip within the terminal device 101. Alternatively, the tag data deduplication device 20 can be an edge-side intelligent agent 102 or a chip or system-on-a-chip within the edge-side intelligent agent 102. For example... Figure 2 As shown, the deduplication device 20 for flag data may include a processor 201, a bus 202, a communication interface 203, and a memory 204.

[0050] Furthermore, the deduplication device 20 for flag data may also include a communication interface 203 and a memory 204. The processor 201, memory 204, and communication interface 203 can be connected via a bus 202.

[0051] The processor 201 can be a CPU, a general-purpose processor, a network processor (NP), a digital signal processor (DSP), a microprocessor, a microcontroller, a programmable logic device (PLD), or any combination thereof. The processor 201 can also be other devices with processing capabilities, such as circuits, devices, or software modules, without limitation.

[0052] Bus 202 is used to transmit information between the components included in the mark data deduplication device 20.

[0053] Communication interface 203 is used to communicate with other devices or other communication networks. These other communication networks can be Ethernet, radio access network (RAN), wireless local area networks (WLAN), etc. Communication interface 203 can be a module, circuit, communication interface, or any device capable of enabling communication.

[0054] Memory 204 is used to store instructions. These instructions can be computer programs.

[0055] The memory 204 can be a read-only memory (ROM) or other type of static storage device that can store static information and / or instructions; it can also be a random access memory (RAM) or other type of dynamic storage device that can store information and / or instructions; it can also be an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.), magnetic disk storage media or other magnetic storage devices, etc., without limitation.

[0056] It should be noted that the memory 204 can exist independently of the processor 201 or can be integrated with the processor 201. The memory 204 can be used to store instructions, program code, or some data. The memory 204 can be located inside or outside the flag data deduplication device 20, without restriction.

[0057] In one example, processor 201 may include one or more CPUs.

[0058] As an optional implementation, the deduplication device 20 for flag data includes multiple processors.

[0059] As an optional implementation, the deduplication device 20 for flag data may also include an output device and an input device. For example, the input device is a keyboard, mouse, microphone, or joystick, and the output device is a display screen, speaker, or other similar device.

[0060] It should be noted that the data deduplication device 20 can be a desktop computer, laptop computer, network server, mobile phone, tablet computer, wireless terminal, embedded device, chip system, or other device. Figure 3 Equipment with a similar structure. Furthermore... Figure 2 The composition shown does not constitute a basis for the interpretation of this invention. Figure 1 as well as Figure 2 The limitations of each device in the process, except Figure 2 In addition to the components shown, Figure 1 as well as Figure 2 The various devices may include more or fewer components than illustrated, or combine certain components, or have different component arrangements.

[0061] In this embodiment of the application, the chip system may be composed of chips or may include chips and other discrete devices.

[0062] Furthermore, the actions, terms, etc., involved in the various embodiments of this application can be referenced interchangeably without limitation. The message names or parameter names in the messages exchanged between the various devices in the embodiments of this application are merely examples, and other names may be used in specific implementations without limitation.

[0063] The following is combined with Figure 1 The illustrated tag data deduplication system describes the tag data deduplication method provided in the embodiments of this application. The actions and terminology used in the various embodiments of this application can be referenced interchangeably without limitation. The message names or parameter names in the messages exchanged between devices in the embodiments of this application are merely examples; other names may be used in specific implementations without limitation. The actions involved in the various embodiments of this application are merely examples; other names may be used in specific implementations. For example, "included in" in the embodiments of this application can be replaced with "carried on" or "carried in," etc.

[0064] In order to solve the problems existing in the above-mentioned prior art, such as Figure 3 As shown in the embodiments of this application, a method for deduplicating bidding data is proposed, which can improve the deduplication accuracy of bidding data. The method includes: S301, The computing device acquires the original bill of lading data.

[0065] The original bidding information data includes several first fields.

[0066] In one possible implementation, the computing device can acquire raw data from multiple stages of the tender documents, including the tender announcement, supplementary announcement, results announcement, and winning bid announcement, and construct an initial dataset. ,in The total number of tender documents. Represents the first in the initial dataset Data. Computing devices can process the initial dataset. Each data in Perform format standardization processing.

[0067] By standardizing the format of the initial dataset, data noise caused by inconsistent formats can be eliminated, and data file consistency can be maintained.

[0068] In one possible implementation, the computing device can extract all possible fields from the normalized dataset to form a complete set of fields. ,in, This represents the total number of fields, specifically the total number of fields in the first field, which is the complete set of the original tag data.

[0069] In one possible implementation, the computing device can record each field. Establish an initial mapping relationship between fields and stages for different occurrences in the bidding process.

[0070] For example, the different stages of the bidding process may include: Stage 1: Bidding Announcement Stage: Initial release of project information, including project name, bidding number, budget amount, bid deadline, and the bidding entity. Stage 2: Bid Evaluation Results Stage: After bid submission, the bid evaluation results are announced, providing detailed information such as project name, winning bidder, evaluation criteria, evaluation date, and bid result announcement. Stage 3: Contract Signing Stage: After the successful bidder, the final contract is signed, including data such as project name, contract amount, contract signing date, contract terms, and the contracting entity.

[0071] For example, based on the complete set of fields Complete set of fields Each field in the document is divided into stages. For example, in stage 1, there are fields such as project name, tender number, budget amount, bid deadline, and tendering entity. In stage 2, there are fields such as project name, winning bidder, evaluation criteria, evaluation date, and announcement of evaluation results. In stage 3, there are fields such as project name, contract amount, contract signing date, contract terms, and contracting entity.

[0072] S302. The computing device deduplicates the original bidding information data based on the importance weight of each of the multiple first fields, and determines the bidding information data after preliminary deduplication.

[0073] The initial deduplication of the bidding information data includes several second fields.

[0074] In one possible implementation, the importance weight of a field satisfies the following formula 1: Formula 1 in, The total number of industries, The total number of stages, For the industry In the stage The weighting coefficients are determined by business requirements and must meet [the following conditions]. , For fields In the industry stage The importance metric, which takes a value of 0 or 1, indicates whether the field is of interest at this stage in the industry.

[0075] In one possible implementation, the computing device deduplicates the original bidding data based on the importance weight of each of the multiple first fields. The specific process for determining the initially deduplicated bidding data is as follows: The computing device constructs a hierarchical representation model of the original bidding data based on the importance weight of each field in the original bidding data. The hierarchical representation model is used to determine the importance level of each field in the original bidding data. The computing device determines the comprehensive similarity between multiple fields of the first importance level in the hierarchical representation model based on a comprehensive similarity formula. The first importance level is one of the multiple importance levels in the hierarchical representation model. The first fields in the original bidding data with a comprehensive similarity greater than a first threshold are merged into one field to determine the initially deduplicated first bidding data.

[0076] In one possible implementation, the specific process by which the computing device constructs a hierarchical representation model of the original bidding data based on the importance weights of each field in the original bidding data is as follows: the computing device assigns importance weights to each field. Sort each first field from highest to lowest to form an ordered list of fields. ,in The computing device calculates based on preset hierarchical thresholds. Divide multiple first fields into There are several levels, and the specific rules for dividing them are as follows: when... When, field Classified to the top level ,when When, field Classified to the layer .

[0077] In one possible implementation, the computing device constructs a hierarchical mapping tree structure of the fields based on the multiple first fields after hierarchical partitioning. The tree nodes represent fields at each level. The parent-child node relationship is determined based on the importance weight and relevance of the fields, according to the degree of relevance. Values ​​determine the parent-child relationships between fields, and a hierarchical mapping tree is constructed. This hierarchical mapping tree structure This is used to reflect the importance hierarchy and relationships of fields. The computing device is based on a hierarchical mapping tree structure. Establish a hierarchical representation model for the original bidding data. The hierarchical representation model of the original tag data has the following structure: ,in, For the first Hierarchy hierarchical The field collections in the data are mapped between levels through a hierarchical tree structure. The father-son relationship is connected.

[0078] For example, the computing device can organize the first field into the following hierarchy: The first level contains core information, including: project name and tender number. The second level contains time information, including: bid deadline, bid evaluation date, and contract signing date. The third level contains financial information, including: budget amount and contract amount. The fourth level contains participant information, including: the tendering party, the winning bidder, and the contracting party. The fifth level contains detailed content, including: scoring criteria, bid evaluation result announcement, and contract terms. Each field is assigned an importance weight based on business needs and experience. For example, "project name" and "tender number" are key fields and can be assigned higher weights; "scoring criteria" and "contract terms" are also important, thus allowing for analysis of different emphases within the fields.

[0079] In one possible implementation, the degree of association between fields The following formula 2 is satisfied: Formula 2 in, Representation field In the Individual bidding data The occurrence of a variable is indicated by a value of 1 if it appears, and 0 if it does not. This represents the total number of beacon data points. i is an integer.

[0080] In one possible implementation, the comprehensive similarity formula satisfies the following formula 3: Formula 3 in, For fields and The overall similarity between them This represents the total number of similarity measurement methods. For the first Weights of various similarity metrics For the first Fields under various measurement methods and Similarity score, and These are two different fields.

[0081] For example, in Greater than or equal to the first threshold In this case, the field can be considered and For synonymous fields, you can choose the name of one of the fields as the common field name for both fields, or you can set a standard field name by establishing an alias mapping relationship. This means mapping the field names of two fields to standard field names. This application does not impose any limitations on this.

[0082] For example, for a set of synonym fields that have alias mapping relationships. Based on format constraint functions Replace them with standard field names. And impose consistency constraints on its data format.

[0083] In one possible implementation, the format constraint function The following formula 4 is satisfied: Formula 4 in, The original field value, The field value after format constraints.

[0084] In one possible implementation, the deduplication process between fields of different importance levels by the computing device is as follows: based on the number of times the third field appears in multiple first fields and the number of times the fourth field appears in multiple first fields, the correlation between the third field and the fourth field is determined. The third field and the fourth field are two fields at different importance levels in the hierarchical representation model. Based on the correlation and similarity between the first field and the second field, it is determined whether to merge the first field and the second field into the same field.

[0085] For example, for hierarchy The next level New or semantically evolved field collections in It can be calculated using the comprehensive similarity formula. New or semantically evolved fields and levels in The overall similarity between fields in the dataset is calculated, and the correlation between fields at two levels is calculated based on the following correlation formula.

[0086] In one possible implementation, the degree of association satisfies the following formula 5: Formula 5 in, For fields and Relevance, Count For fields and The number of times they appear simultaneously in the same bidding data. For fields Total number of occurrences.

[0087] For example, in the field and new fields If the correlation between the fields is greater than a preset correlation threshold, the computing device can add new fields. Determined as a field The evolution or expansion of computing devices can Perform label annotation and record its source field. Field and new fields The correlation between them is greater than the preset correlation threshold and the fields and new fields If the overall similarity between the fields is greater than the first threshold, then the fields can be... and new fields The field pair has been identified as a duplicate.

[0088] For example, "bidder" might sometimes be called "purchaser," so "purchaser" is mapped to "bidder." Similarly, "successful bidder" might be listed as "successful bidder," which is mapped to "successful bidder." Constructing a synonym mapping table ensures that fields referring to the same information are correctly identified, which is crucial for accurate data merging and comparison. After establishing the synonym mapping, the hierarchical representation model is updated accordingly to ensure that all synonymous fields are correctly categorized in the hierarchical structure, and that the model accurately reflects the relationships between fields. For example, the process by which the computing device performs preliminary deduplication on the initial bidding data and generates preliminary deduplicated bidding data can be as follows: the computing device extracts the hierarchical representation model after preliminary deduplication. The fields in the code generate preliminary deduplicated tag data. , This includes fields with unified naming and format constraints, as well as newly added fields that have been labeled and merged. Among them, duplicate fields can be merged to remove duplicates from the label information data.

[0089] In one possible implementation, the initially deduplicated tag information data satisfies the following formula 6: Formula 6 in, For the processed fields, For fields In data The value in To obtain from raw data Extract fields A function of the value of , The hierarchical representation model after initial deduplication The set of fields in.

[0090] In one possible implementation, the hierarchical representation model after initial deduplication... The set of fields in the data satisfies the following formula 7: Formula 7 in, h represents the total number of fields across all levels, where h is an integer.

[0091] In one possible implementation, the computing device can also construct a field alignment mapping matrix to adaptively adjust the field alignment strategy for different industries and stages. ,in This represents the total number of fields after processing. This represents the total number of scenarios determined by combining different industries and bidding stages.

[0092] In one possible implementation, the metatable in the matrix The following formula 8 is satisfied: Formula 8 in, Let be the element in the j-th row and s-th column of the matrix, where j and s are integers.

[0093] Based on mapping matrix The initially aligned data is then adaptively adjusted for specific scenarios, filtering out irrelevant fields to obtain a representation of the target information data for each specific scenario.

[0094] S303, The computing device uses the feature vector of each of the multiple second fields and the cross-stage semantic transfer network model to deduplicate the initial deduplicated tag information data.

[0095] In one possible implementation, the computing device performs the following process to deduplicate the initially deduplicated bidding data based on the feature vectors of each of the multiple second fields and a cross-stage semantic transfer network model: The computing device determines the semantic similarity between the fifth and sixth fields based on the feature vectors of the fifth and sixth fields, and calculates the added value index of the fifth and sixth fields. The fifth and sixth fields are each one of the multiple second fields. If the semantic similarity between the third and fourth fields is greater than a second threshold, and both the added value index of the third and fourth fields are less than the third threshold, the computing device identifies the third and fourth fields as a duplicate field pair. The computing device then merges the duplicate field pairs in the initially deduplicated bidding data into a single field.

[0096] In one possible implementation, the cross-stage semantic transfer network model can also be called a cross-stage semantic transfer network (CSTN). The CSTN model includes an alignment module, a latent semantic learning module, and an output correction module. The alignment module is used to embed features into the initially deduplicated tag data. The latent semantic learning module is used to learn representations of the embedded features based on a multi-layer deep neural network. The output correction module is used to correct the semantic features extracted from the initially deduplicated tag data and output the semantic feature results.

[0097] For example, the computing device will process the initially deduplicated tag data. The input alignment module is fed into the CSTN, and the input alignment module aligns each beacon data point. Extract its field feature vector Construct the input feature matrix ,in, This represents the total number of fields after processing. For bidding data Field feature vectors ,in, For bidding data In the field The value that can be taken on.

[0098] For example, the input alignment module can define adaptive embedding functions for different industries and stages. , input feature matrix Mapping to the embedding space, where the methods for mapping to the embedding space can be word embedding algorithms (Word2Vec) and vector representation techniques (Embedding).

[0099] In one possible implementation, the adaptive embedding function Satisfy the following formula 9: Formula 9 in, To embed the feature matrix, For embedded dimensions, The set of parameters for the embedded function, the embedded function By adjusting the parameters It adapts to the characteristics of different industries and stages to achieve input feature alignment.

[0100] In one possible implementation, CSTN's implicit semantic learning module is based on a cross-stage semantic transfer function. Learn about the dynamic changes in bidding information data at different stages and across industries.

[0101] In one possible implementation, a cross-stage semantic transfer function The following formula 10 is satisfied: Formula 10 in, For the first Layer feature representation, For the first The network parameters of the layer, including the weight matrix and bias vector , It includes cross-stage transfer components, enabling the network to learn shared features and stage-specific features.

[0102] In one possible implementation, a stage embedding matrix representing stage information is used. ,in For the stage embedding dimension, each data point is encoded according to its stage, and a stage attention weight matrix is ​​calculated. .

[0103] In one possible implementation, the stage attention weight matrix Satisfy the following formula 11: Formula 11 in, Here is the stage weight matrix, and softmax is the row-by-row softmax function.

[0104] In one possible implementation, the computing device weights and combines the features of the previous layer based on the stage attention weights to obtain the stage-weighted feature representation of the current layer.

[0105] In one possible implementation, the stage-weighted feature representation of the current layer satisfies the following formula 12: Formula 12 in, This indicates element-wise multiplication.

[0106] In one possible implementation, the computing device is based on a nonlinear activation function. Transform the stage-weighted feature representation of the current layer to determine the transformed stage-weighted feature representation.

[0107] In one possible implementation, the transformed stage-weighted feature representation satisfies the following formula 13: Formula 13 in, This represents the weighted feature representation of the transformed stage.

[0108] In one possible implementation, CSTN's implicit semantic learning module can utilize a multi-task loss function. Simultaneously learn about industry characteristics and stage characteristics.

[0109] In one possible implementation, the multi-task loss function The following formula 14 is satisfied: Formula 14 Where α, β, and γ are weighting coefficients. For repeated detection loss, To identify the loss of newly added information, For semantic transfer regularization, Used to maintain the consistency of feature distribution and reduce the differences in feature distribution between different stages and industries.

[0110] In one possible implementation, repeated detection loss The following formula 15 is satisfied: Formula 15 in, This is the set of positive sample pairs (repeated data pairs). It is a set of negative sample pairs (non-repeating data pairs). For data The final characteristic representation, Distance threshold In one possible implementation, additional information recognition loss is added. The following formula 16 is satisfied: Formula 16 in, For data Does the annotation include a transition function for newly added information? To predict probabilities, and These are the classifier parameters.

[0111] In one possible implementation, the gradient descent algorithm can be used to optimize the parameters of the cross-stage semantic transfer network model. Optimize.

[0112] In one possible implementation, the optimized cross-stage semantic transfer network model parameters The following formula 17 is satisfied: Formula 17 in, Here are the parameters of the optimized cross-stage semantic transfer network model, and L is the multi-task loss function.

[0113] During training, the cross-stage semantic transfer network learns deep feature representations that can adapt to both stage dynamics and industry characteristics. This improves recognition performance through repeated detection and the addition of new information.

[0114] In one possible implementation, a deep feature representation of the initially deduplicated tag data can be obtained through a cross-stage semantic transfer mechanism determined by the implicit semantic learning module. The output correction module can correct the feature representation based on the correction function to eliminate the bias caused by differences in data distribution.

[0115] In one possible implementation, the correction function The following formula 18 is satisfied: Formula 18 in, This is the corrected feature representation. This is the set of parameters for the correction function.

[0116] In one possible implementation, the correction function can employ batch normalization or layer normalization to standardize the feature distribution and eliminate biases between different stages and industries.

[0117] For example, the corrected feature representation set This serves as a cross-stage semantic representation of bidding information data. It integrates semantic information from different stages and industries within the bidding information data.

[0118] In one possible implementation, the computing device determines the initial high-order feature representation of the pre-deduplicated tag data based on a cross-stage semantic transfer network. ,in, For bidding data The initial high-order feature vector, is the dimension of the feature vector.

[0119] In one possible implementation, the computing device can compute a field-level association matrix, the field-level association matrix... elements Representation field and fields In the The degree of correlation between layers.

[0120] In one possible implementation, the elements in the field-level association matrix The following formula 19 is satisfied: Formula 19 in, For bidding data Middle field The value of , For the first Layered network bidding information data Feature representation, The partial derivative of the feature representation with respect to the input field reflects the degree of influence of the field on the feature representation.

[0121] In one possible implementation, the computing device can compute the feature gradient matrix. , characteristic gradient matrix elements Representation field For feature dimensions The impact.

[0122] In one possible implementation, the feature gradient matrix The following formula 20 is satisfied: Formula 20 in, For the first Layered Bidding Information Data In feature dimension The value on.

[0123] In one possible implementation, the computing device utilizes the computed field-level association matrix. and the characteristic gradient matrix Iterative corrections are made to the alias mapping and labeling strategies in the hierarchical alignment transformation.

[0124] In one possible implementation, the update rule could be: if, ,and, Then and Mapped to the same field ,in, The correlation threshold, The feature gradient difference threshold. and fields respectively and The feature gradient vector, i.e. The row and number OK.

[0125] In one possible implementation, for newly added or evolved fields, the feature gradient matrix is ​​used. The field influence in the data is reassessed, and its labeling is updated according to the following rules: If Then the field If promoted to a higher level Then the field Downgrading or merging, among which, and The threshold parameter is used to control the adjustment of the field hierarchy; the hierarchy of the field is dynamically adjusted according to the influence of the field on the feature representation, thereby optimizing the hierarchical representation structure.

[0126] In one possible implementation, the computing device reconstructs the hierarchical representation model based on the aforementioned update results. And update the initially aligned beacon data. New data indicates It will be used as input for the next iteration.

[0127] In one possible implementation, the following convergence condition is met: Complete the hierarchical representation model Reconstruction. Among them, and The first Second and third The higher-order feature representation matrix of the next iteration Denotes the Frobenius norm. This is a preset convergence threshold. When the change in the higher-order feature representation is less than the threshold... At that time, it is considered that the iterative process has converged, and the result of the hierarchical alignment transformation is consistent with the cross-stage semantics learned by the network.

[0128] In one possible implementation, the final high-order feature representation is output after the iterative process converges. ,in For bidding data The final higher-order feature vector.

[0129] In one possible implementation, the process by which the computing device determines the cross-stage semantic transfer network model through training an initial model is as follows: The computing device acquires historical tagging data. The computing device adds labels to the fields of the historical tagging data, the labels indicating whether the fields are duplicates, and trains an initial model based on the labeled historical tagging data and its feature vectors to determine the cross-stage semantic transfer network model.

[0130] In one possible implementation, the computing device is based on the obtained set of high-order feature vectors. and the corresponding beacon dataset Construct a supervised training dataset. For each marker data point... Based on business needs and expert knowledge, the fields are labeled to generate a tag set. ,in, 1 indicates that the field is a duplicate field, and 1 indicates that the field is a newly added field.

[0131] In one possible implementation, a weighted loss function is designed to fully utilize the importance of fields in different industries and the dynamic evolution of multi-stage tender documents. This guides the model's learning process.

[0132] In one possible implementation, the weighted loss function The following formula 21 is satisfied: Formula 21 in, For the first The field weight coefficients of each data point reflect the importance of the field and the time decay factor. The loss function is based on cross-picking. For the output correction module of the cross-stage semantic transfer network, with parameters For higher-order eigenvectors The prediction results.

[0133] In one possible implementation, the field weight coefficient The following formula 22 is satisfied: Formula 22 in, For fields In the relevant industry Importance coefficient in, through the field Importance weight To obtain formalization, the following conditions must be met. , For the current time and field The difference in the last occurrence time reflects the time decay of the field. This is the time decay coefficient, which controls the degree of influence of time factors on the weights.

[0134] In one possible implementation, based on a weighted loss function Supervised training is performed on the output correction module of the cross-stage semantic transfer network to optimize the model parameters. .

[0135] In one possible implementation, the optimized model parameters The following formula 23 is satisfied: Formula 23 in, These are the optimized model parameters.

[0136] In one possible implementation, an optimized model is used to evaluate the semantic similarity and derivation relationships between fields. For any two fields... and Calculate the semantic similarity between their higher-order feature vectors. .

[0137] In one possible implementation, semantic similarity satisfies the following formula 24: Formula 24 in, For vector dot product, Let L be the L2 norm of the vector. For vectors and The angle between them.

[0138] In one possible implementation, when similarity When the value is close to 1, it indicates that the fields have a high degree of semantic similarity, are duplicate fields, or have a derived relationship.

[0139] In one possible implementation, a new value index is added. The following formula 25 is satisfied: Formula 25 in, This represents the importance coefficient of the field in the industry. For fields semantic sparsity, This represents the time-based activity coefficient.

[0140] In one possible implementation, semantic sparsity The following formula 26 is satisfied: Formula 26 in, For fields The number of fields with semantic relationships. For fields Related fields Semantic similarity.

[0141] In one possible implementation, the time activity coefficient The following formula 27 is satisfied: Formula 27 in, The time decay coefficient is the factor that is used to determine the time decay rate. Class control time impact, For fields The time difference of the last occurrence.

[0142] In one possible implementation, the computing device mines candidate deduplication clues based on the new value index and semantic similarity, and outputs potential duplicate and mergeable field pairs based on the deduplication clues.

[0143] For example, potentially duplicate field pairs: for similarity Exceeding the threshold field pairs And the newly added value index and The values ​​are all low, identified as potential duplicate field pairs, and field pairs can be merged: For field pairs with derivative relationships and high semantic similarity, if a new value index is added... or Below the threshold Then the field and Merge the candidate deduplication clues. Organize the deduplication clues into a list. Each element contains field pairs and their corresponding similarity and new value index information.

[0144] In one possible implementation, the computing device can also construct an optimization model by comprehensively considering the importance of fields, semantic similarity, stage differences, and business rules to achieve the optimal field merging strategy. Fields determined to be new and having stage-specific business value are retained and highlighted, and high-quality multi-stage tagging data with invalid duplicate information removed and key new fields retained is output.

[0145] In one possible implementation, the computing device can also be based on a set of deduplicated threads. To meet the business needs of different stages of bidding data, further match similar or duplicate fields, and handle conflicts between similar or duplicate fields.

[0146] By comprehensively considering the importance of fields, semantic similarity, stage differences, and business rules, an optimization model can be built to optimize the field merging strategy, thereby improving the accuracy of field merging.

[0147] In one possible implementation, the computing device can construct a weighted bipartite graph based on deduplicated threads. ,in: Belongs to stage For the stage A collection of fields; Belongs to stage For the stage A collection of fields; Let be the set of edges consisting of candidate field pairs; where the weights of the edges are... It is determined by both the similarity of the fields and business requirements.

[0148] In one possible implementation, the edge weights The following formula 28 is satisfied: Formula 28 in, For fields and semantic similarity, This involves assigning weights to fields based on business requirements, reflecting the degree of importance of each field in the business context. The stage difference measure is the reasonableness of field matching across stages. Let be the weighting coefficient, satisfying .

[0149] In one possible implementation, the field matching weights satisfy the following formula 29: Formula 29 in, and The business importance weight of the field, and The range of values ​​is within Between these values, the larger the value, the higher the business importance of that field in the corresponding stage.

[0150] In one possible implementation, the stage difference degree satisfies the following formula 30: Formula 30 in, and fields respectively and The timestamp of the stage to which it belongs The time scale parameter controls the impact of stage differences on matching weights.

[0151] In one possible implementation, the computing device can determine a set of matching criteria using conflict constraints, constraint conditions, and field matching weights. An optimized model for field merging is constructed, in which constraints include: unique matching constraints, conflict handling constraints, and business rule constraints.

[0152] In one possible implementation, the conflict constraint satisfies ,in, Match weights to fields.

[0153] In one possible implementation, the unique matching constraint satisfies as well as ,in, In satisfying The value is 1 in the following case. In dissatisfaction The value is 0 in the following case.

[0154] In one possible implementation, conflict handling constraints are used to prevent matching of field pairs that conflict simultaneously. Conflict handling constraints can be... ,in, This is a collection of conflicting field pairs.

[0155] In one possible implementation, business rule constraints are used to add hard constraints based on business rules for field pairs that must match or must not match. For example, for field pairs that must match... satisfy For fields that are prohibited from matching satisfy .

[0156] In one possible implementation, the computing device can transform the model optimization into an integer linear programming problem, that is, solve the following expression to determine the optimal field matching set. .

[0157] In one possible implementation, the integer linear programming expression satisfies the following formula 31: Formula 31 Here, "st" is an abbreviation for "subjectto", meaning "to meet the following conditions".

[0158] In one possible implementation, the computing device can determine the optimal matching set. To merge similar or duplicate fields, the merging process can be as follows: the computing device can perform matching field pairing... Adopt appropriate merger strategies Generate new field values .

[0159] In one possible implementation, the new field value satisfies the following formula 32: Formula 32 in, and These are the field pairs to be merged.

[0160] In one possible implementation, for numeric field pairs to be merged, the computing device can take a weighted average or the latest value. For text field pairs, the computing device can merge them by string concatenation or by selecting the value with more information. For date field pairs, the computing device can merge them by taking the latest date or covering a wider date range. For field pairs that fail to match and conflict, the computing device can process them according to business rules and field importance, prioritizing the retention of only the more important field.

[0161] In one possible implementation, the computing device can calculate a business value index. For newly added fields that have phased business value Highlight key points and include them in the business value index. Exceeding the threshold In the case of, the field These will be retained and highlighted.

[0162] In one possible implementation, the business value index The following formula 33 is satisfied: Formula 33 in, For fields The importance weight of the business, This is the field's time activity coefficient. The stage adaptability of a field represents the business value of the field at the current stage.

[0163] In one possible implementation, the stage adaptability of the field The following formula 34 is satisfied: Formula 34 in, For fields timestamp, The time center for the current business phase, This refers to the time scale parameter for each stage.

[0164] In one possible implementation, the computing device can output merged, high-quality, multi-stage tag data. The computing device denotes the tag dataset after field matching, merging, and adding new field annotations as follows: Each bidding data Includes the following: merged field values: based on the best matching set Merger and Acquisition Strategy Generated field values; Retained new field: Business Value Index New fields are highlighted; Phased business information: Based on business needs, retain important field information specific to each phase.

[0165] In one possible implementation, the computing device can also verify the merged tag data. Verification and evaluation are conducted. The verification and evaluation process involves the computing device determining the rate of reduction in duplicate fields based on the number of duplicate segments before and after merging, and assessing data validity based on the coverage of key information in the bidding data. Furthermore, the computing device can enhance the business value of the bidding data by retaining and highlighting newly added and important fields.

[0166] In one possible implementation, the rate of reduction of duplicate fields satisfies the following formula 35: Formula 35 in, To reduce the rate of duplicate fields, This represents the number of repeated segments before merging. This represents the number of repeated segments after merging.

[0167] For example, the merged final dataset may include the following integrated fields: Project Name: A unified project name across all phases. Tender Number: A standardized tender number. Budget Amount: The verified budget amount. Tender Deadline: A standardized and confirmed deadline. Tenderer: The unified name of the tenderer. Successful Bidder: Confirmed information on the successful bidder. Evaluation Criteria: An important new field that is retained, providing detailed evaluation criteria. Evaluation Result Announcement: The merged announcement of the evaluation results. Contract Amount: The final contract amount. Contract Signing Date: The confirmed contract signing date. Contract Terms: Detailed contract terms that are retained and highlighted.

[0168] For example, such as Figure 4The diagram shows a flowchart of another method for deduplicating bidding data provided in this application embodiment. An electronic device acquires the original bidding data and performs normalization processing on it. The electronic device extracts fields from the original bidding data and calculates the importance weights of the fields. The electronic device hierarchically divides the fields according to their importance weights and constructs a mapping tree. Based on the mapping tree, the electronic device generates a hierarchical model. The electronic device performs preliminary deduplication on the original bidding data based on the hierarchical model, determining the preliminary deduplicated bidding data. The electronic device inputs the preliminary deduplicated bidding data into a cross-stage semantic transfer network model to determine the high-order semantic features of the preliminary deduplicated bidding data. The electronic device corrects the high-order semantic features and, based on the corrected high-order semantic features, determines the relevance of fields in the bidding data. The electronic device corrects the hierarchical alignment strategy and mapping strategy. Based on the relevance of fields, the corrected alignment strategy, and the corrected mapping strategy, the electronic device merges duplicate field pairs in the bidding data. The electronic device outputs high-quality deduplicated bidding data. The electronic device can also verify and evaluate the deduplication effect of the high-quality bidding data.

[0169] To address the problems existing in the prior art, this application proposes a method for deduplicating bidding data. The method initially deduplicates the original bidding data by assigning importance weights to each field, determining the initially merged bidding data. Further deduplication is then performed based on the feature vectors of each field and a cross-stage semantic transfer network model. This two-stage deduplication process effectively filters out duplicate fields in the original bidding data, improving deduplication accuracy. Furthermore, the cross-stage semantic transfer network model provided in this application has a high accuracy rate in identifying duplicate field pairs, thereby further improving the deduplication accuracy of the bidding data.

[0170] It is understood that the above-mentioned method for deduplicating bidding data can be implemented by a bidding data deduplication device. To achieve the above functions, the bidding data deduplication device includes hardware structures and / or software modules corresponding to the execution of each function. Those skilled in the art should readily recognize that, based on the modules and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein, the embodiments disclosed in this application can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed in hardware or by computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the embodiments disclosed in this application.

[0171] The embodiments disclosed in this application can divide functional modules according to the deduplication device for the tag data generated by the above method example. For example, each function can be divided into its own functional module, or two or more functions can be integrated into one processing module. The integrated module can be implemented in hardware or as a software functional module. It should be noted that the module division in the embodiments disclosed in this application is illustrative and is only a logical functional division. In actual implementation, there may be other division methods.

[0172] Figure 5 This is a schematic diagram of a deduplication device for tag data provided in an embodiment of the present invention. Figure 5 As shown, the deduplication device 50 for bidding data can be used to perform... Figure 3 The illustrated method for deduplicating tag data. The tag data deduplication device 50 includes a communication unit 501 and a processing unit 502.

[0173] The communication unit 501 is used to acquire the original tag information data, which includes multiple first fields; the processing unit 502 is used to deduplicate the original tag information data based on the importance weight of each of the multiple first fields, and determine the initially deduplicated tag information data, which includes multiple second fields; the processing unit 502 is also used to deduplicate the initially deduplicated tag information data based on the feature vector of each of the multiple second fields and the cross-stage semantic transfer network model.

[0174] In one possible implementation, the processing unit 502 is specifically used to: construct a hierarchical representation model of the original bidding data based on the importance weights of each field in the original bidding data, the hierarchical representation model being used to determine the importance level of each field in the original bidding data; determine the comprehensive similarity between multiple fields of the first importance level in the hierarchical representation model based on a comprehensive similarity formula, the first importance level being one of the multiple importance levels in the hierarchical representation model; and merge the first field in the original bidding data with a comprehensive similarity greater than a first threshold into one field to determine the bidding data after preliminary deduplication.

[0175] In one possible implementation, the processing unit 502 is specifically used to: determine the correlation between the third field and the fourth field based on the number of times the third field appears in multiple first fields and the number of times the fourth field appears in multiple first fields, wherein the third field and the fourth field are two fields at different importance levels in the hierarchical representation model; and determine whether to merge the first field and the second field into the same field based on the correlation and similarity between the first field and the second field.

[0176] In one possible implementation, the processing unit 502 is specifically used to: determine the semantic similarity between the fifth field and the sixth field based on the feature vector of the fifth field and the feature vector of the sixth field, and calculate the new value index of the fifth field and the new value index of the sixth field, wherein the fifth field is one of multiple second fields and the sixth field is one of multiple second fields; if the semantic similarity between the third field and the fourth field is greater than a second threshold and the new value index of the third field and the new value index of the fourth field are both less than a third threshold, the third field and the fourth field are identified as a duplicate field pair; and the duplicate field pairs in the initially deduplicated tag data are merged into one field.

[0177] In one possible implementation, the communication unit 501 is further configured to acquire historical tag data; the processing unit 502 is further configured to add labels to the fields of the historical tag data, the labels being used to indicate whether the fields of the historical tag data are duplicate fields; the processing unit 502 is further configured to train an initial model based on the labeled historical tag data and the feature vectors of the historical tag data, and determine the cross-stage semantic transfer network model.

[0178] In one possible implementation, the comprehensive similarity formula satisfies the following formula:

[0179] in, For fields and The overall similarity between them This represents the total number of similarity measurement methods. For the first Weights of various similarity metrics For the first Fields under various measurement methods and Similarity score, and These are two different fields.

[0180] In one possible implementation, the field's new value index satisfies the following formula:

[0181] in, This represents the importance coefficient of the field in the industry. For fields semantic sparsity, This represents the time-based activity coefficient.

[0182] Through the above description of the embodiments, those skilled in the art will clearly understand that, for the sake of convenience and brevity, only the division of the above functional modules is used as an example. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. The specific working process of the system, device, and unit described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0183] This disclosure also provides a computer-readable storage medium storing instructions that, when executed by a processor of an electronic device, enable the electronic device to perform the tag data deduplication method provided in the embodiments of this disclosure described above.

[0184] This disclosure also provides a computer program product containing instructions that, when run on an electronic device, cause the electronic device to execute the tag data deduplication method provided in the above-described embodiments of this disclosure.

[0185] The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of computer-readable storage media (a non-exhaustive list) include: an electrical connection having one or more wires; a portable computer disk drive; a hard disk drive; a random access memory (RAM); a read-only memory (ROM); an erasable programmable read-only memory (EPROM); a register; a hard disk drive; an optical fiber; a portable compact disc read-only memory (CD-ROM); an optical storage device; a magnetic storage device; or any suitable combination thereof; or any other form of computer-readable storage medium known in the art. An exemplary storage medium is coupled to a processor, enabling the processor to read information from and write information to the storage medium. Of course, the storage medium may also be a component of the processor. The processor and the storage medium may reside in an application-specific integrated circuit (ASIC). In the embodiments of this application, the computer-readable storage medium can be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0186] The above are merely specific embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method for deduplicating tag information data, characterized in that, include: Obtain the original bidding information data, which includes multiple first fields; Based on the importance weight of each of the plurality of first fields, the original bidding information data is deduplicated to determine the preliminary deduplicated bidding information data, which includes a plurality of second fields; Based on the feature vector of each of the multiple second fields and the cross-stage semantic transfer network model, the initially deduplicated tag information data is deduplicated.

2. The method according to claim 1, characterized in that, The step of removing duplicates from the original tag information data based on the importance weight of each of the plurality of first fields, and determining the initially deduplicated tag information data, includes: Based on the importance weights of each field in the original bidding data, a hierarchical representation model of the original bidding data is constructed. The hierarchical representation model is used to determine the importance level of each field in the original bidding data. Based on the comprehensive similarity formula, the comprehensive similarity between multiple fields of the first importance level in the hierarchical representation model is determined, where the first importance level is one of the multiple importance levels in the hierarchical representation model. The first field in the original bidding data whose overall similarity is greater than the first threshold is merged into one field to determine the preliminary deduplicated bidding data.

3. The method according to claim 2, characterized in that, The method further includes: Based on the number of times the third field appears in the plurality of first fields and the number of times the fourth field appears in the plurality of first fields, the correlation between the third field and the fourth field is determined, wherein the third field and the fourth field are two fields located at different importance levels in the hierarchical representation model; Based on the correlation and similarity between the first field and the second field, determine whether to merge the first field and the second field into the same field.

4. The method according to claim 1, characterized in that, The process of deduplicating the initially deduplicated tag data based on the feature vector of each of the plurality of second fields and the cross-stage semantic transfer network model includes: Based on the feature vectors of the fifth field and the sixth field, the semantic similarity between the fifth field and the sixth field is determined, and the new value index of the fifth field and the new value index of the sixth field are calculated. The fifth field is one of the plurality of second fields, and the sixth field is one of the plurality of second fields. If the semantic similarity between the third field and the fourth field is greater than the second threshold and the new value index of the third field and the new value index of the fourth field are both less than the third threshold, the third field and the fourth field are identified as a duplicate field pair. The duplicate fields in the initially deduplicated tag data are merged into one field.

5. The method according to claim 1, characterized in that, The method further includes: Obtain historical bidding data; Add tags to the fields of the historical tag data; the tags are used to indicate whether the fields of the historical tag data are duplicate fields. An initial model is trained based on historical tag data with the aforementioned labels and the feature vectors of the historical tag data, thereby determining the cross-stage semantic transfer network model.

6. The method according to claim 2, characterized in that, The comprehensive similarity formula satisfies the following formula: in, For fields and The overall similarity between them This represents the total number of similarity measurement methods. For the first The weights of various similarity measurement methods satisfy the following conditions: , For the first Fields under various measurement methods and The similarity score.

7. The method according to claim 2 or 3, characterized in that, The newly added value index of the field satisfies the following formula: in, This represents the importance coefficient of the field in the industry. For fields semantic sparsity, This represents the time-based activity coefficient.

8. A device for deduplicating tag data, characterized in that, The deduplication device for tag data includes: a communication unit and a processing unit; The communication unit is used to acquire raw flag data, the raw flag data including multiple first fields; The processing unit is used to remove duplicates from the original tag information data based on the importance weight of each of the plurality of first fields, and determine the tag information data after preliminary deduplication. The tag information data after preliminary deduplication includes a plurality of second fields. The processing unit is further configured to deduplicate the initially deduplicated tag data based on the feature vector of each of the plurality of second fields and the cross-stage semantic transfer network model.

9. A device for deduplicating tag data, characterized in that, include: A processor and a communication interface; the communication interface is coupled to the processor, the processor being used to run computer programs or instructions to implement the tag data deduplication method as described in any one of claims 1-7.

10. A computer-readable storage medium storing instructions, characterized in that, When the computer executes the instruction, the computer performs the deduplication method for tag data as described in any one of claims 1-7.

Images