A person entity recognition and fusion method, device and equipment based on multi-modal evidence fusion

Abstract

The present application provides a kind of based on multi-modal evidence fusion, character entity recognition and fusion method, device and equipment, carry out extraction and standardization to multi-source character data, generate to be matched record and retrieve candidate record set;Extract the multi-modal feature of each record, filter relationship event according to the effective time interval of biography, align and encode relationship path and biography track;Calculate each modal similarity, generate consistency coefficient, conflict penalty factor and hard conflict mark, adjust modal weight in combination with evidence quality and modal availability, get fusion matching score by weighting;Inhibit hard conflict record automatic fusion, carry out attribute fusion to the record without hard conflict and scoring reaches preset threshold, generate unified character entity and write into character knowledge base.The present application improves the accuracy of character entity recognition by the time alignment and consistency of biography and relationship event, reduces the incidence of false fusion.

Description

Technical Field

[0001] This invention relates to the field of multimodal data processing and knowledge graph technology, and in particular to a method, apparatus and equipment for person entity recognition and fusion based on multimodal evidence fusion. Background Technology

[0002] In the process of constructing a biographical knowledge base and aggregating multi-source data, information about the same person is often scattered across public web pages, institutional websites, public databases, and image resources. Different data sources differ in name descriptions, field formats, record times, and information completeness, resulting in multiple records for the same person, and different people potentially having the same name or similar descriptions. Therefore, it is necessary to determine the correspondence between different records through biographical entity identification and fusion, and to integrate data belonging to the same person into a unified biographical entity.

[0003] Existing methods for person entity recognition typically match based on name, attributes, or textual semantic similarity, with some methods also incorporating image and relational information to aid in the judgment. However, the completeness and reliability of various data types are inconsistent, and contradictory information may exist between different modalities. If the features of each modality are encoded separately and then directly summed for similarity, the high similarity of a local modality may mask inconsistencies in other modalities, leading to the incorrect association of different persons. For records with missing modalities, failure to distinguish between missing evidence and conflicting evidence may also cause the fusion results to be interfered with by incomplete data.

[0004] Furthermore, an individual's employer, position, and related events exhibit temporal changes. Similar employer, position, or relationship descriptions may correspond to different time periods, making inaccurate matching easy when comparing their content in isolation. Inconsistent attribute values ​​recorded from different sources may also stem from normal job changes. If these are not processed in conjunction with the valid timeframe of the relevant data, historical attribute changes may be misjudged as conflicts, or contradictory data within the same time period may be merged into the same individual entity.

[0005] In view of the above, this application is hereby submitted. Summary of the Invention

[0006] This invention discloses a method, apparatus, and device for person entity recognition and fusion based on multimodal evidence fusion, aiming to solve the problem of insufficient accuracy in person entity matching and incorrect fusion of records of different persons in multi-source person data due to differences in the quality of modal evidence and temporal conflicts in resume and relationship information.

[0007] The first embodiment of the present invention provides a method for person entity recognition and fusion based on multimodal evidence fusion, comprising: Extract and standardize publicly available personal data from multiple sources to generate records to be matched, and retrieve a set of candidate records accordingly; Text, attributes, face images, and resume features are extracted from the records to be matched and the records in the candidate record set. Relationship events are filtered according to the effective time interval of the resume. The relationship path is aligned with the resume trajectory. The relationship path is encoded to obtain multimodal features and alignment results. Based on the multimodal features, the similarity of each modality is calculated. A consistency coefficient, a conflict penalty factor, and a hard conflict marker are generated according to the resume trajectory and the alignment result. The weight of each modality is determined by combining the quality of evidence and modality availability. The text, attribute, and image modal weights are adjusted together using the consistency coefficient and the conflict penalty factor. The adjusted weights are normalized and the similarity of each modality is weighted and summed to obtain the fusion matching score. Automatic fusion of hard conflict records is prohibited based on the hard conflict marker. For records without hard conflict and whose fusion matching score reaches a preset threshold, attribute fusion is performed to generate a unified character entity and write it into the character knowledge base.

[0008] The second embodiment of the present invention provides a person entity recognition and fusion device based on multimodal evidence fusion, comprising: The record generation unit is used to extract and standardize publicly available data on persons from multiple sources, generate records to be matched, and retrieve a set of candidate records accordingly. The feature construction unit is used to extract text, attributes, face images and resume features from the record to be matched and the record in the candidate record set, filter relationship events according to the effective time interval of the resume, align the relationship path with the resume trajectory, encode the relationship path, and obtain multimodal features and alignment results. The fusion computing unit is used to calculate the similarity of each modality based on the multimodal features, generate a consistency coefficient, a conflict penalty factor, and a hard conflict marker according to the history trajectory and the alignment result, determine the weight of each modality by combining evidence quality and modality availability, adjust the text, attribute, and image modal weights together using the consistency coefficient and the conflict penalty factor, normalize the adjusted weights and sum the weights of each modality similarity to obtain the fusion matching score; The entity generation unit is used to prevent the automatic fusion of hard conflict records according to the hard conflict marker, perform attribute fusion on records without hard conflict and whose fusion matching score reaches a preset threshold, generate a unified person entity, and write it into the person knowledge base.

[0009] The third embodiment of the present invention provides a person entity recognition and fusion device based on multimodal evidence fusion, including a memory and a processor. The memory stores a computer program, which can be executed by the processor to implement a person entity recognition and fusion method based on multimodal evidence fusion as described in any of the above embodiments.

[0010] The fourth embodiment of the present invention provides a computer-readable storage medium storing a computer program, which can be executed by the processor of the device where the computer-readable storage medium is located, to implement a person entity recognition and fusion method based on multimodal evidence fusion as described in any of the above embodiments.

[0011] Beneficial effects: By extracting and standardizing multi-source person data, a multimodal matching basis is established between the records to be matched and the candidate records. Relationship events are filtered according to the effective time interval of the resume, and the relationship path is aligned with the resume trajectory. This allows the person entity matching to consider the similarity of multimodal features and the temporal correspondence between relationship information and resume information, reducing the matching bias caused by the mixing of relationship information from different periods and improving the ability to distinguish between people with the same name or similar information.

[0012] By combining evidence quality and modal availability to determine the weights of each modality, and using consistency coefficients and conflict penalty factors to jointly adjust the modal weights of text, attributes, and images, the verification results of resume and relationship information are included in the calculation of the fusion matching score, reducing the adverse effects of low-quality and contradictory evidence on the matching results. At the same time, hard conflict markers are used to prohibit the automatic fusion of records with hard conflicts, preventing such records from being incorrectly merged due to high fusion matching scores, thereby improving the accuracy of generating unified person entities and the data consistency of the person knowledge base. Attached Figure Description

[0013] Figure 1 This is a flowchart illustrating a method for human entity recognition and fusion based on multimodal evidence fusion provided in the first embodiment of the present invention. Figure 2 This is a schematic diagram of a human entity recognition and fusion device based on multimodal evidence fusion provided in the second embodiment of the present invention. Detailed Implementation

[0014] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0015] To better understand the technical solution of the present invention, the embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0016] This invention discloses a method, apparatus, and device for person entity recognition and fusion based on multimodal evidence fusion, aiming to solve the problem of insufficient accuracy in person entity matching and incorrect fusion of records of different persons in multi-source person data due to differences in the quality of modal evidence and temporal conflicts in resume and relationship information.

[0017] Please see Figure 1 The first embodiment of this invention provides a method for person entity recognition and fusion based on multimodal evidence fusion, in which a computer processes publicly available person data from multiple sources. The computer reads text, structured fields, and image data to establish standardized person records. Through candidate retrieval, multimodal feature construction, alignment of resume and relationship information, and fusion scoring, it determines whether records from different sources correspond to the same person, and writes the confirmed attributes, resume, and relationship information into a person knowledge base. During processing, the original records, source identifiers, and time information are retained, enabling the features, scores, and database entry results to be associated with corresponding evidence. The encoding and face detection can employ existing models, and their input and output uses are determined according to the following process.

[0018] S1, extract and standardize the publicly available data of people from multiple sources to generate records to be matched, and retrieve a set of candidate records accordingly; The computer retrieves data related to individuals from publicly available reports, databases, institutional web pages, encyclopedia entries, and publicly available image resources, reading individual names, text descriptions, attribute fields, images, resumes, and related events. For content involving multiple individuals from the same source, records are organized separately based on the position of the individual's name in the text, contextual references, and field relationships. For content with the same name from different sources, each record and source association are retained. Each record is associated with the original content and source identifier, enabling subsequent tracing of the origin of fields, images, and events.

[0019] For text-based data, word segmentation, noise reduction, and entity recognition are performed on character profiles, job descriptions, and event descriptions to extract character names, company names, job titles, time information, and related events, while retaining text fragments that support the extraction results. For structured data, based on the mapping relationship between source fields and preset standard fields, the field values ​​corresponding to name, alias, date of birth, employer, job title, and education are read. For job descriptions, education, company changes, and collaboration descriptions, event type, participants, associated companies, and occurrence time are extracted, and the event is associated with the character records in its source. The extraction results are saved with corresponding confidence scores, which are used to evaluate the reliability of the correspondence between the extraction results and the source content; the confidence scores output by the information extraction model can be used.

[0020] Name information is normalized, organizing Chinese names, foreign names, native language names, transliterated names, and aliases into standard names, alias sets, and cross-language mapping results. Name normalization can be achieved using alias dictionaries, transliteration rules, edit distance matching, cross-language semantic encoding models, and entity linking models. For names with known correspondences, a mapping is established between the standard name and the original name; for names that only have name similarity but whose correspondence has not yet been confirmed, their original expression and candidate mapping status are retained as the basis for subsequent candidate retrieval. When names are similar or have possible transliteration correspondences, it is still necessary to combine other modalities to determine the identity of the person, avoiding premature merging of different records during the name processing stage.

[0021] The attribute fields are normalized in terms of field name, value range, and unit. Fields with the same semantics from different sources are mapped to unified standard fields, and the expression of field values ​​is standardized. The standardization of unit names and job titles can be achieved using a pre-defined terminology dictionary, a synonym rule base, abbreviation expansion rules, and text matching models, mapping full names, abbreviations, and foreign language expressions with definite correspondences to consistent standard terms. Standardized attributes are stored as key-value pairs, associated with the original field name, original field value, and field mapping result. For values ​​with different meanings or whose equivalence cannot yet be confirmed, their content and source are retained for subsequent matching and attribute conflict handling.

[0022] For image data, the system reads the image, its associated description, and available image time information, and performs source tagging and resolution verification. Source tagging establishes the correspondence between the image and its source page, associated text, and person records; resolution verification reads the image size to determine if it meets the input requirements for subsequent face feature extraction. Verification results are saved along with the image. If no image is provided, the missing state is retained; if an image exists but its quality is insufficient, the quality state is retained; whether the image contains a usable face is determined by subsequent face detection. If image association cannot be determined, the pending confirmation state is retained.

[0023] The date format and timeline of resume information are standardized, converting the time expressions in descriptions of employment, education, and unit changes into a unified format and associating them with the corresponding units, positions, and events. When start and end times are clearly recorded, the corresponding valid time range is saved; when only the year or month is recorded, the original time precision is preserved; when the end time is not specified, the state of "end time unknown" is retained. Timeline completion is based on verifiable time information and the chronological relationship of events from the source, organizing known resume content in chronological order, and not adding appointment dates without supporting evidence. The source publication time and event occurrence time are saved separately, ensuring that the actual occurrence time of historical experience is not changed by re-recording it from a later source.

[0024] For relationship events, event type classification and participant standardization are performed. Semantically consistent descriptions of employment, education, unit changes, and collaborations are mapped to corresponding standard event types, and participant names, unit names, and job titles are standardized. Each event retains its occurrence time or valid time range, source identifier, extraction confidence level, and interpersonal relationships. The names of interpersons in relationship events retain their corresponding source record identifiers. After the above processing, standardized names, text, attributes, images, resumes, and relationship events are organized into records to be matched, while the extraction time, standardization rule version, and field mapping results are saved.

[0025] Name retrieval features are generated based on the standard names, alias sets, and cross-language mapping results in the records to be matched. Text retrieval features are generated based on the encoded text descriptions. Attribute retrieval features are generated based on standardized attribute key-value pairs. Resume retrieval features are generated based on the resume timeline and job sequence. Existing records in the personnel knowledge base are indexed according to the same field standards using rule indexes, and vector indexes are created using corresponding encoding methods. Name retrieval features are used to find records corresponding to names or mapping relationships, text retrieval features are used to find records with similar descriptive semantics, and attribute and resume retrieval features are used to find records that match known fields or experience content.

[0026] Candidate retrieval employs a combination of rule-based recall and vector-based recall. Rule-based recall accesses the rule index based on standard names, alias mappings, and standardized attributes, and returns candidate records in conjunction with known unit, job title, and time information. Vector-based recall utilizes text retrieval features and vector representations of attribute and resume retrieval features to access corresponding vector indexes, returning candidate records based on retrieval similarity, ensuring that records with different names but similar descriptions or experiences are included in the candidate range. The results returned from each recall channel are aggregated, and duplicate records are removed based on the knowledge base record identifier, resulting in a candidate record set. Existing records with the same name but different record identifiers are retained separately; the recall results are only used to determine the subsequent comparison range.

[0027] When some attributes, images, or resumes are missing, candidate retrieval uses the currently available name, text, and known attribute information, retaining the state of no data provided. The missing state is passed to subsequent feature construction processes. When the candidate set is empty, the state of the record to be matched and the state of no candidates retrieved are saved for subsequent processing as unmatched records; at this time, there are no record pairs available for scoring, and no score indicating a successful identity match is generated. For cases where candidates are recalled, the record to be matched and each candidate record constitute subsequent comparison objects.

[0028] In a specific instance, an existing record has the name Michael Anderson, the position of Chief Executive Officer, and the organization of Company A, indicating that he assumed this position in 2022. A newly acquired record has the name Mike Anderson, the position of CEO, and the organization of Company A, with a resume describing him as the CEO of Company A in 2022. Using a job terminology dictionary and abbreviation expansion rules, the corresponding job title is standardized to "CEO." Given the known alias mapping support, a correspondence is established between the two names. Subsequently, a recall process is performed combining the name, organization, position, year of appointment, and textual semantics to include existing records in the candidate record set. The output of the record to be matched and the candidate record set serves as input for step S2.

[0029] S2, extract text, attributes, face images and resume features from the records to be matched and the records in the candidate record set, filter relationship events according to the effective time interval of the resume, align the relationship path with the resume trajectory, encode the relationship path, and obtain multimodal features and alignment results; The computer reads the standardized text, attribute key-value pairs, images, history, and relational events of each record, preserving the independent comparison relationship between the record to be matched and each candidate record. Feature construction uses an encoding model corresponding to the corresponding data type. Two records in the same modality use the same encoding method and comparable feature dimensions; different modalities are not required to have the same dimensions. Retrieval encoding results obtained using the same model and the same input content can be reused; when the retrieval information is insufficient to support a complete match, the corresponding content in the standardized record is read and re-encoded. Retrieval similarity and fusion score are saved separately.

[0030] For text features, a text encoding model is used to encode the biographical information, job descriptions, and descriptions of public events, resulting in text feature vectors that reflect the semantics of the descriptions. The encoding preserves the context of the person, organization, position, and time, avoiding the direct substitution of extracted names for complete descriptions. For descriptions in different languages, a cross-language text encoding model can be used to create comparable vector representations. Text features are associated with the original description and source identifier, and text similarity is subsequently calculated using the text feature vectors of the two records. If the text is empty or a valid vector cannot be obtained, the record is considered unavailable, and invalid vectors are not used in the similarity calculation.

[0031] For attribute features, standardized attribute key-value pairs are encoded using a unified field identifier, ensuring a consistent correspondence for the same field across different records. The encoding results retain the field name, value, and value status, enabling subsequent comparison of attribute content by field. For time-varying attributes such as employing unit and position, the valid time range is retained, associating attribute consistency judgments with the time range; for missing fields, the missing status is retained, and field values ​​from candidate records are not used to fill in the records to be matched. Attribute vectors are used to represent field content, with standardized key-value pairs and time information retained simultaneously, used for subsequent calculation of attribute set consistency scores and interpretation of differences.

[0032] For image features, based on source association and resolution verification results, a face detection model is used to determine the face location, a target region is extracted using an object detection model or segmentation model, and then the image features are output by a face representation encoding model. The target region should have evidence of association with the current record; if multiple faces exist in the image but the target person cannot be identified, the state of uncertain association is retained. Image features are stored together with image source information, image time information, and image-associated text information. If the image is missing, no valid face is detected, or the target association cannot be determined, an unusable state is output; if features can be extracted but the image quality is insufficient, the quality evaluation result is retained for subsequent weight attenuation.

[0033] Extract time nodes, employer nodes, position nodes, and event nodes from work experience, education experience, and employer change records. Organize nodes belonging to the same experience into resume segments with valid time intervals. Time nodes represent the time of event occurrence or the start and end dates of experience; employer nodes represent the employing unit or educational institution; position nodes represent the position held; and event nodes represent employment, education, or employer change events. Each node uses standardized results and is linked to its original source. When there is no complete resume text but there are clearly defined employment or education events, the corresponding resume segments can be organized based on the extracted event information to form actual, observable partial experiences, without supplementing unknown other experiences.

[0034] Resume segments are connected chronologically to form a resume trajectory, preserving the chronological relationship between segments, changes in employers and positions, and their respective valid time intervals. For overlapping segments, the overlap relationship and events that may explain the relationship are retained, and no segment is forcibly deleted during sequence connection. For records with year or month precision, the original precision is retained, and the uncertain state is preserved for parts whose chronological order cannot be determined. The node content, time information, and known order are encoded using a time-series coding model to obtain resume features, while preserving their correspondence with resume segments.

[0035] The target time window is determined based on the effective time intervals corresponding to employment, education, and employer changes in the current record pair's resume trajectory. For each window, events whose occurrence times fall within that window, as well as events whose effective time intervals intersect with the window, are filtered from the related events of the two records. Multiple discontinuous resume stages retain their corresponding windows and filtering results, ensuring that relationship paths can be mapped to specific experiences. When comparing two records from the same stage, the same target time window is used to avoid comparing relationships from different periods as contemporaneous relationships when each party selects different time ranges.

[0036] When one record lacks a complete resume while another has a definite resume time range, that definite time range can be used as the target time window for this comparison. This filters out the relationship events accessible to both records, while retaining the state where the first record lacks sufficient resume information. If neither record has a definite resume or employment event time range, the state where time constraints cannot be established is retained, and the unknown time range is not treated as the full time range. Events whose timeframes cannot be determined to fit within the window are retained as evidence to be verified and do not directly participate in path matching that already meets the time conditions. Modal availability is subsequently determined based on the actual comparable content.

[0037] A heterogeneous relationship graph with time and source tags is constructed using the filtered events. The graph includes nodes for people, organizations, positions, and events, connected by roles, education, participation in events, and collaborations. Relationship edges store the relationship type, occurrence time, or valid time interval, and are associated with the source providing the relationship. The person nodes corresponding to the records to be matched and the candidate records are kept distinct, and node correspondences are established using standardized organization, position, and event information. For the same relationship supported by multiple sources, the associations of each source are preserved; for events where the same event cannot yet be confirmed, different nodes are retained.

[0038] The relationship paths associated with the current person's node are extracted from the heterogeneous relationship graph. The unit nodes, job nodes, event nodes, and time sequence within these paths are compared to their corresponding resume trajectories. Node comparison uses standard names, unit aliases, and job mappings to determine matching; event comparison combines event type, participants, and time to determine correspondence; and time comparison verifies whether the event's time matches the valid time of the resume segment. For multi-event paths, the consistency between the event sequence and the resume order is further verified. Each record's path is first verified to correspond with its resume, and then the path relationship between the two records is compared using common nodes and corresponding times to form the alignment result for the current record pair.

[0039] The alignment results record node matching, time sequence consistency, and path conflict, and save the relevant resume fragments, relationship paths, and source evidence. When a relationship and resume both point to the same unit, position, and matching time, the matching relationship is recorded; when discrepancies occur, the specific nodes and time ranges are retained; if a determination cannot be made solely due to missing data or insufficient time precision, an unverifiable status is recorded. For discrepancies, events that might explain the discrepancy, such as unit changes, position changes, concurrent appointments, or proxy appointments, are also retained. Step S2 outputs verifiable correspondence and discrepancy information; the nature of a hard conflict is further determined in step S3 by combining verified events.

[0040] A relational path encoding model is used to encode the node types, node attributes, relation types, and known event sequences in the selected paths, resulting in relational event features. These features retain the target time window and source association, and are used to compare the relational structure within the corresponding history stages. Text, attribute, image, history, and relational event features are organized into multimodal features according to modality, with each feature maintaining its own data structure and usability. Subsequently, the similarity of the corresponding modalities is calculated, enabling quality assessment and weight adjustment to be located at the specific modality.

[0041] The resume features and corresponding trajectories are used together in step S3 to calculate the resume similarity, and the calculation method is as follows:

[0042] in, Indicates the similarity of resumes; The degree of trajectory overlap is calculated based on the ratio of the sum of the intersection durations to the sum of the union durations of the effective time intervals corresponding to the matching nodes. The matching nodes are identical or compatible unit nodes, job nodes, and event nodes. It represents the continuity of time and is calculated based on the number and length of unreasonable time intervals and the degree of violation of the order of events; The similarity of unit paths is calculated by weighted sequence matching based on unit level, unit alias, unit transfer edge, and job transfer edge. , and These are the preset non-negative coefficients corresponding to trajectory overlap, temporal continuity, and unit path similarity, respectively, and satisfying the following:

[0043] When calculating trajectory overlap, the node correspondence is first determined, and then the corresponding valid time intervals are compared. The same resume fragment recorded from the same source is processed according to standardized correspondence to avoid repeated accumulation of the same duration. Temporal continuity is evaluated in conjunction with confirmed job change events; unexplained intervals or increased degree of sequence deviations result in a lower score. Unit path similarity is evaluated by combining node matching and the order of unit and position changes. If only the event occurrence time is available without a comparable duration, the event is still used for node and sequence verification; if a valid duration comparison cannot be formed, the duration ratio calculation with a denominator of zero is not performed. If a valid resume similarity cannot be formed, the scoring status of that modality is marked as unavailable, but known event evidence is still retained.

[0044] Relationship event features and paths are used for relationship similarity calculation in step S3. Within the target time window, the consistency of common units, common events, common paths, and path order are compared and weighted according to preset weights. Common events are verified by type, participants, and time; common paths are verified by node correspondence and relationship type; and path order is compared based on determinable sequential relationships. Similarity is used to express the degree of matching between two records, and alignment results further provide evidence that their relationship matches the history, differs, or cannot be verified. Step S2 correlates and outputs multimodal features, history trajectory, relationship path, alignment results, and available status, enabling step S3 to complete scoring and constraint calculations on the same record pair.

[0045] S3, calculate the similarity of each modality based on the multimodal features, generate a consistency coefficient, a conflict penalty factor, and a hard conflict marker according to the resume trajectory and the alignment result, determine the weight of each modality by combining evidence quality and modality availability, adjust the text, attribute, and image modal weights together using the consistency coefficient and the conflict penalty factor, normalize the adjusted weights and sum the weights of each modality similarity to obtain the fusion matching score; The computer processes each record to be matched and each candidate record separately, reading the corresponding features, available status, and alignment results, and associating them with the source and extraction information saved in step S1. The quality evaluation and weight calculation of the current record pair are performed based on the evidence actually involved in the comparison, and the scoring results of other candidate record pairs cannot be directly applied. For modalities that can obtain effective corresponding features, their similarity is calculated; for modalities that cannot obtain effective similarity, the reason and unavailable status are retained. Hard conflict judgments and available modal scores are saved separately to avoid losing existing conflict evidence due to the absence of individual modalities.

[0046] Text similarity This is the cosine similarity between the feature vectors of two record texts. Attribute similarity. The consistency score of the standardized attribute set corresponding to the attribute feature is determined by verifying the standardized values ​​according to the common comparable fields; time-varying fields are compared in conjunction with the corresponding valid time intervals. Image similarity. Image consistency scores are calculated by combining facial representation features, image source information, image time information, and image-related text information. Image source and associated text are used to verify the correspondence between faces and person records, while image time is used to preserve its temporal context and avoid using images with incorrect associations simply based on appearance similarity as a basis for identification.

[0047] Resume Similarity The score for matching the resume sequence corresponding to the resume features is calculated using a weighted approach based on the aforementioned trajectory overlap, temporal continuity, and unit path similarity. Relationship Similarity The similarity score is calculated based on the consistency of common units, common events, common paths, and path order within the target time window, corresponding to the characteristics of the related events and filtered through the valid time intervals of the records. The similarity score is correlated with the fields, trajectories, or paths that produced the result. For records with only partially observable information, the results are formed based on the comparable portions, and their completeness is reflected in subsequent quality assessments. Records that cannot be effectively compared remain unavailable.

[0048] For each modality, source reliability, data integrity, cross-source consistency, and extraction confidence are calculated separately. Source reliability is determined by calling a preset source score based on the source identifier; data integrity is evaluated based on the information required for this comparison and the actual information provided; cross-source consistency is evaluated based on multi-source verification of the corresponding attributes, events, or relationships; and extraction confidence is determined based on the extraction results saved by the fields or events involved in the comparison. These scores are normalized to 0 to 1 according to preset rules, with higher values ​​indicating a stronger likelihood of matching for the corresponding quality factor. The scores are based on the modality evidence actually used for the current record.

[0049] For each mode, the basic weights are calculated according to the following formula:

[0050] in, For modal indexing, , , , , and These represent text modality, attribute modality, image modality, resume modality, and relational event modality, respectively. Representing modes The basic weights; , , and Representing modes The reliability of the source, the integrity of the data, cross-source consistency, and the confidence level of extraction; , , and These are the preset non-negative coefficients corresponding to source reliability, data integrity, cross-source consistency, and extraction confidence, respectively, and they satisfy the following:

[0051] When multiple sources provide mutually corroborating information about the attribute, event, or relationship to be verified, the corresponding cross-source consistency score is improved. When there are conflicts in attribute values, event times, or relationship paths between sources, the corresponding score is reduced. Cross-source comparisons combine field correspondences and valid times to prevent treating legitimate changes from different historical periods as concurrent value conflicts. The base weight reflects the quality of modal evidence, while usability adjustment further reflects whether the modality can be used for the actual calculation of the current record pair. Its calculation method is as follows:

[0052] in, Representing modes The availability indicator, with a value ranging from 0 to 1; Representing modes The base weights are adjusted for availability. If any record lacks the data required to complete the modality comparison or cannot form a valid corresponding similarity, then... Set to 0. When a modality is available for comparison but the data quality is below a preset quality threshold, it will be... Set to an attenuation value greater than 0 and less than 1; when the mode is available and the quality requirements are met, it can be set to 1.

[0053] The resume consistency coefficient is determined based on the consistency of unit path, time continuity, and job sequence in the two resume records. The relationship consistency coefficient is determined based on the proportion of common events in the relationship events, the similarity of the relationship paths, and the consistency of common associated nodes. The above evaluation is conducted in conjunction with the node correspondence, event sequence, and time verification results of step S2. When the resume and relationship both point to the same unit, position, and time, and source evidence can be obtained for verification, the corresponding consistency coefficient is increased. and The values ​​range from 0 to 1, with larger values ​​indicating a higher degree of consistency.

[0054] Determine the penalty factor for resume conflict based on the degree of resume conflict. Determine the relationship conflict penalty factor based on the degree of relationship path conflict. .in, and Values ​​range from 0 to 1, with higher values ​​indicating a greater degree of conflict. The evaluation considers time overlap, the correspondence between organizations and positions, event sequence, and verified events. For inconsistencies in resumes or relationship paths that do not constitute a hard conflict, increasing the degree of inconsistency raises the corresponding conflict penalty factor or lowers the corresponding consistency coefficient. Similarity serves as the input for the weighted summation, while the consistency coefficient and conflict penalty factor serve as the inputs for weight adjustment.

[0055] When verification is impossible solely due to missing resume data, and All are set to 0; this is only set when verification is impossible due to missing relation event data. and All are set to 0. When partial evidence exists, evaluation is based on the observable timeframe; unknown parts are not directly penalized for conflict. The unavailability of a modality score does not mean all its original information is invalid; verifiable events can still be retained for verification of other correspondences. Evidence of actual inconsistency is used for conflict severity evaluation; missing, soft, and hard conflicts are stored separately.

[0056] A hard conflict marker is generated when two resume trajectories correspond to mutually exclusive positions or organizations within overlapping valid time intervals, and no verified concurrent, acting, or job change events can explain the discrepancy. Mutual exclusion relationships are identified based on preset mutual exclusion correspondences and verified using standardized nodes and time ranges; surface overlaps caused by different organization names or year precision are insufficient to independently determine a mutual exclusion conflict. A hard conflict marker is generated when a relationship path conflicts with an organization, position, or time in the resume that cannot be explained by verified events. Verified events should have traceable origins, and their participants and times should correspond to relevant resume segments.

[0057] Hard conflict markers are associated with conflict nodes, time ranges, historical fragments or relationship paths, and source evidence, and are passed to step S4 as independent results. Hard conflict records are not included in the automatic fusion determination, and high text, attribute, or image similarity does not eliminate the corresponding markers. For record pairs without hard conflicts, weights are adjusted based on evidence quality, usability, and consistency, and the adjusted weights for text, attributes, images, historical records, and relationship event modalities are calculated according to the following formulas:

[0058]

[0059]

[0060]

[0061]

[0062] in, , , , and These represent the adjustment weights for the text modality, attribute modality, image modality, resume modality, and relational event modality, respectively. , , , and These represent the base weights after availability adjustment for the corresponding modalities; and They represent and Enhancement coefficients in text, attribute, and image modality weighting; and These represent the modal weighting adjustments in the resume. In the adjustment of enhancement coefficients and relational event modal weights The enhancement coefficients are all preset non-negative coefficients and none are greater than the preset upper limit of the enhancement coefficient.

[0063] History and relationship consistency jointly participate in the calculation of adjustment weights for text, attribute, and image modalities, with corresponding conflict penalties affecting the adjustment weights of these three modalities. History and relationship / event modalities are adjusted according to their respective coefficients. The adjustment weights are normalized to determine the final contribution ratio of each available modality. The base weight of a missing modality is adjusted to 0 based on availability, and its adjustment weight remains 0. An upper limit on the enhancement coefficient restricts the magnitude of the enhancement; its specific value is used as a preset parameter and remains consistent within the same comparison.

[0064] When there are no hard conflicts and the sum of all adjustment weights is greater than 0, normalization is performed according to the following formula:

[0065] in, Representing modes Adjusting the weights, Representing modes Normalized fusion weights.

[0066] , , , and These represent the normalized fusion weights for text modality, attribute modality, image modality, resume modality, and relationship event modality, respectively. The sum of the effective weights after normalization is 1. If there is no hard conflict but all weights are adjusted to 0, the state where no effective score can be formed is retained, and normalization with a denominator of 0 is not performed. For record pairs that can be normalized, the fusion matching score is calculated according to the following formula:

[0067] in, Indicates the fusion matching score. , , , and These represent the aforementioned text similarity, attribute similarity, image similarity, resume similarity, and relationship similarity, respectively. Weighted terms corresponding to missing modalities are not included in the summation and are not multiplied by zero weight using undefined similarity. Similarity is calculated using its respective defined method, with text similarity maintaining the definition of cosine similarity; subsequent thresholds are consistent with the fusion scoring definition used.

[0068] In a numerical example, assume that two records have comparable complete modal data and there are no hard conflicts. , , , and The values ​​are 0.96, 0.97, 0.92, 1.00, and 0.91, respectively, obtained through the above calculations. , , , and If the values ​​are 0.25, 0.25, 0.15, 0.25, and 0.10 respectively, then... The value is 0.9615. The above weights are a set of exemplary normalized results; the actual weights are dynamically determined based on the current record. Step S3 outputs the fusion matching score or the state where a valid score cannot be formed, while simultaneously saving the hard conflict marker, modal similarity, weights, and evidence for step S4 to determine fusion, independent retention, or review.

[0069] S4. Based on the hard conflict marker, the automatic fusion of hard conflict records is prohibited. For records without hard conflict and whose fusion matching score reaches a preset threshold, attribute fusion is performed to generate a unified character entity and write it into the character knowledge base.

[0070] The computer receives the fusion matching score, hard conflict markers, and source evidence for the current record pair. It first checks for hard conflicts in resumes or relationships. If either marker exists, automatic fusion is prohibited, and the corresponding node, time, resume fragment, and relationship path are retained. If the individuals can be identified as different people based on existing evidence, they are retained as independent entities; if verification of source, event time, or job change is required, a record awaiting review is generated. For record pairs without hard conflicts but unable to form a valid score, the evidence is insufficient, and the process proceeds to review, but not to the branch where the score reaches the threshold.

[0071] For record pairs without hard conflicts and with valid scores, a first preset threshold and a second preset threshold are used for judgment, with the first preset threshold being greater than the second preset threshold. When the fusion matching score is greater than or equal to the first preset threshold, they are determined to be the same person entity and attribute fusion is performed; when the score is greater than or equal to the second preset threshold but less than the first preset threshold, a record awaiting review is generated; when the score is less than the second preset threshold, it is retained as an independent entity. Each threshold is a preset parameter that corresponds to the scoring calculation method, and the comparison uses the aforementioned clear boundaries to ensure that the critical score has a definite processing result. Candidate retrieval similarity is not involved in this threshold judgment.

[0072] The records pending review store the identifier of the record to be matched, the identifier of the candidate record, the available score, the modal evidence, the conflict marker, and the content to be verified. The computer receives and associates the review results. If the record is confirmed to be the same person, attribute fusion is performed; if the record is confirmed to be different people, the records are retained independently; if the identity cannot be clearly determined, the record remains in the pending review state. If a hard conflict exists but the review confirms that the record is the same person, the original conflict evidence and its interpretation or revised conclusion are retained, and the review confirmation result is used for subsequent processing. This process is not considered automatic fusion based on scoring to overcome hard conflict restrictions. The review conclusion is associated with the corresponding record pair.

[0073] Attribute fusion only aggregates records that have been confirmed to belong to the same person. For each standard attribute field, candidate values ​​from different sources are collected, and the original value, standardized value, source identifier, source publication time, valid time interval, source reliability, information freshness, cross-source consistency, extraction confidence, and historical stability are saved. The same standardized value can be aggregated to support a source, but the time basis recorded by each source is retained separately; different values ​​continue to retain their respective evidence. When multiple candidates simultaneously match a record to be matched, existing conflicting information is checked before aggregation. Records that cannot yet be confirmed to be compatible are further reviewed to avoid unverified expansion of the scope of records for the same person.

[0074] The valid time interval indicates the time range within which the attribute's described state is valid, and the source publication time indicates the time the information was published. The intervals for employing unit and position are determined based on the start and end dates of employment, unit changes, and position changes; when only the year, month, or end time is recorded, the corresponding precision and state are retained. The static attribute of birth date applies to the entire lifecycle of the individual, meaning that this attribute does not change with the query time and does not require obtaining the complete lifecycle dates of the individual first. Candidate values ​​with uncertain times retain their evidence and cannot be considered valid attributes solely because of their recent publication time.

[0075] The system filters candidate time-varying attributes within a valid time range based on the time of the query. The time of the query is the target time at which the attribute status of an individual needs to be determined, and can be provided by the query request; when generating the current attribute status, the corresponding time generated by the current entity can be used. Values ​​within other time ranges are retained as historical attributes. If no candidate value can be supported by evidence to be valid for that time, the state of that field at the target time, before it is determined, is retained, and the most recent historical appointment record is not directly carried over to the current one. Images and events retain their original time context.

[0076] When the valid time intervals of different attribute candidate values ​​do not overlap, the values ​​are saved according to their respective intervals, forming attribute records for different stages. When there are verified unit changes, job changes, or end-of-term events that can explain the changes, the events are associated with the values ​​before and after the changes, so that historical changes can be mapped to specific resumes. For confirmed concurrent or agency situations, the corresponding values ​​are retained according to the job, unit, and time range supported by the event. When standard fields allow expressing multiple concurrent employment relationships that are established simultaneously, each relationship carries independent time and event basis, avoiding compressing well-founded concurrent employment relationships into mutually exclusive single values.

[0077] When the valid time intervals of different candidate values ​​overlap, the positions or units belong to mutually exclusive sets, and there is no verified explanation for concurrent appointments, acting appointments, or changes in employment, an attribute conflict marker is generated, and the historical stability of the corresponding candidate value is reduced. The marker stores the conflicting field, value, overlapping interval, and source. Attribute conflicts and hard conflicts in the identity matching stage are stored separately. For cases where attribute values ​​differ but their mutual exclusivity or concurrent validity cannot yet be confirmed, the differences and time uncertainty are retained, and the determination is made later based on credibility and verification evidence; one of the values ​​is not directly deleted.

[0078] The reliability of attribute candidate values ​​is determined based on their source scores; cross-source consistency is determined based on the verification of values ​​and applicable timeframes from different sources; information freshness is evaluated based on the source's publication time and update status; extraction confidence is based on the corresponding extraction results; and historical stability is determined based on the compatibility of the value with the historical record of the same person, the effective time interval, and confirmed events. Higher historical compatibility increases historical stability, while the presence of unexplained overlaps or mutually exclusive values ​​decreases historical stability. Each evaluation is specific to a particular candidate value.

[0079] Attribute credibility is calculated using the following formula:

[0080] in, This indicates the credibility of an attribute candidate value. Indicates the reliability of the source. Indicates cross-source consistency. Indicates the freshness of information. This indicates the extraction confidence level. This indicates historical stability. All five scores used in the calculation are normalized to a range of 0 to 1, and the resulting attribute credibility also ranges from 0 to 1. This credibility evaluates the reliability of the attribute candidate values, while the matching scores are used to evaluate the identity correspondence of the record pairs. The two types of scores are stored separately.

[0081] When multiple valid candidate values ​​exist for the same queried time, the candidate value with the highest ranking is selected as the field fusion result based on a comprehensive ranking of source reliability, the degree of matching between the valid time interval and the queried time, and attribute credibility. The degree of time matching is determined based on the support of valid time evidence for the target time; candidate values ​​with clearly defined time boundaries and those with only partially defined time boundaries are given different evaluation statuses. The above evaluation results can be used as a sorting key, and comparisons can be made according to a preset priority, which remains consistent throughout the same processing cycle. Selection is performed for differences that can be resolved based on time and source evidence, while unselected candidate values ​​and their supporting evidence are saved for subsequent updates and verification.

[0082] For valid time interval overlap conflicts that cannot be resolved, evidence should be preserved and a review record for the corresponding field should be generated. Conflicts should not be considered resolved solely based on sorting results. After receiving the attribute review results, the fusion result is determined based on the confirmed values, time intervals, and events. If the values ​​apply to different stages, they are saved according to the confirmed interval. If a value or time record from a certain source is confirmed as incorrect, the verification record is retained and the confirmed candidate value is used. Fields that remain unclear are kept in a review pending state. Other confirmed fields can be saved, but unresolved conflict values ​​are not output as definitive attributes.

[0083] A unified person entity is generated based on the confirmed standard name, alias, basic attributes, available avatars, resume, educational background, events, and relationship information. Name information retains cross-language and alias mappings; resumes and time-varying attributes are organized by valid time intervals; and relationship information retains confirmed associations between persons and organizations, between persons, and between persons and events. Entities are mapped to the original records and source evidence participating in the fusion process, and fusion scores, attribute credibility, and verification conclusions are saved. When an entity is confirmed to be associated with an existing entity in the knowledge base, its information is updated and its history is retained; records identified as independent persons or for which no candidates were found are saved as independent or unmatched, avoiding the misrepresentation of a lack of candidates as a verified difference from all persons.

[0084] Based on the aforementioned name examples, assuming further comparable images and resume information with clearly defined valid times are obtained, step S3 yields a fusion matching score of 0.9615 with no hard conflicts. The first preset threshold is 0.90, so the two records proceed to attribute fusion. Michael Anderson is retained as the standard name, and Mike Anderson is stored in the alias set. The job title is standardized as CEO, and the company is standardized as CompanyA. The 2022 employment history and events are saved with the original time precision. The information for this year supports the recording of the corresponding employment events; whether the individual is still in office is determined by subsequent evidence supporting the target time. When new job titles or companies change subsequently, updates are made according to the valid timeframe and verified events, while retaining the original experience.

[0085] After the knowledge base of individuals is written, record identifiers, fusion matching scores, hard conflict handling results, attribute selection results, and review conclusions are saved, forming a traceable fusion log. When new public evidence is subsequently obtained, the corresponding matching and attribute processing can be re-executed based on the record relationships. Review results, fusion logs, and revision information can be used to update subsequent candidate recall rules, modal weights, attribute conflict rules, and threshold parameters. Updated configurations are used for subsequent calculations, while the original results retain evidence and processing basis.

[0086] Please see Figure 2 The second embodiment of the present invention provides a person entity recognition and fusion device based on multimodal evidence fusion, comprising: The record generation unit 201 is used to extract and standardize multi-source publicly available personal data, generate records to be matched, and retrieve a set of candidate records accordingly. The feature construction unit 202 is used to extract text, attributes, face images and resume features from the record to be matched and the records in the candidate record set, filter relationship events according to the effective time interval of the resume, align the relationship path with the resume trajectory, encode the relationship path, and obtain multimodal features and alignment results. The fusion calculation unit 203 is used to calculate the similarity of each modality based on the multimodal features, generate a consistency coefficient, a conflict penalty factor and a hard conflict marker according to the history trajectory and the alignment result, determine the weight of each modality by combining evidence quality and modality availability, adjust the text, attribute and image modal weights together using the consistency coefficient and the conflict penalty factor, normalize the adjusted weights and sum the weights of each modality similarity to obtain the fusion matching score; The entity generation unit 204 is used to prohibit the automatic fusion of hard conflict records according to the hard conflict mark, perform attribute fusion on records without hard conflict and whose fusion matching score reaches a preset threshold, generate a unified person entity and write it into the person knowledge base.

[0087] The third embodiment of the present invention provides a person entity recognition and fusion device based on multimodal evidence fusion, including a memory and a processor. The memory stores a computer program, which can be executed by the processor to implement a person entity recognition and fusion method based on multimodal evidence fusion as described in any of the above embodiments.

[0088] The fourth embodiment of the present invention provides a computer-readable storage medium storing a computer program, which can be executed by the processor of the device where the computer-readable storage medium is located, to implement a person entity recognition and fusion method based on multimodal evidence fusion as described in any of the above embodiments.

[0089] Exemplary examples show that the computer program described in the third and fourth embodiments of the present invention can be divided into one or more modules, which are stored in the memory and executed by the processor to complete the present invention. The one or more modules can be a series of computer program instruction segments capable of performing specific functions, which describe the execution process of the computer program in implementing a person entity recognition and fusion device based on multimodal evidence fusion. For example, the apparatus described in the second embodiment of the present invention.

[0090] The processor referred to can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor. The processor is the control center of the aforementioned method for person entity recognition and fusion based on multimodal evidence fusion, connecting various parts of the method through various interfaces and lines.

[0091] The memory can be used to store the computer program and / or modules. The processor, by running or executing the computer program and / or modules stored in the memory, and by calling the data stored in the memory, implements various functions of a person entity recognition and fusion method based on multimodal evidence fusion. The memory may mainly include a program storage area and a data storage area. The program storage area may store the operating system, at least one application program required for a function (such as sound playback function, text conversion function, etc.), etc.; the data storage area may store data created based on the use of the mobile phone (such as audio data, text message data, etc.). In addition, the memory may include high-speed random access memory, and may also include non-volatile memory, such as hard disk, memory, plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, at least one disk storage device, flash memory device, or other volatile solid-state storage device.

[0092] If the implemented module is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the above embodiments of the present invention can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include: any entity or device capable of carrying the computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content included in the computer-readable medium can be appropriately added or removed according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electrical carrier signals and telecommunication signals.

[0093] It should be noted that the device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Furthermore, in the accompanying drawings of the device embodiments provided by this invention, the connection relationships between modules indicate that they have communication connections, which can be specifically implemented as one or more communication buses or signal lines. Those skilled in the art can understand and implement this without any creative effort.

[0094] The above description is merely a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method for person entity recognition and fusion based on multimodal evidence fusion, characterized in that, include: Extract and standardize publicly available personal data from multiple sources to generate records to be matched, and retrieve a set of candidate records accordingly; Text, attributes, face images, and resume features are extracted from the records to be matched and the records in the candidate record set. Relationship events are filtered according to the effective time interval of the resume. The relationship path is aligned with the resume trajectory. The relationship path is encoded to obtain multimodal features and alignment results. Based on the multimodal features, the similarity of each modality is calculated. A consistency coefficient, a conflict penalty factor, and a hard conflict marker are generated according to the resume trajectory and the alignment result. The weight of each modality is determined by combining the quality of evidence and modality availability. The text, attribute, and image modal weights are adjusted together using the consistency coefficient and the conflict penalty factor. The adjusted weights are normalized and the similarity of each modality is weighted and summed to obtain the fusion matching score. Automatic fusion of hard conflict records is prohibited based on the hard conflict marker. For records without hard conflict and whose fusion matching score reaches a preset threshold, attribute fusion is performed to generate a unified character entity and write it into the character knowledge base.

2. The method for person entity recognition and fusion based on multimodal evidence fusion according to claim 1, characterized in that, The process of extracting and standardizing publicly available personal data from multiple sources to generate records to be matched, and retrieving a candidate record set based on this data, includes: The system extracts the person's name, text description, attribute fields, image, resume, and relationship events from the multi-source publicly available person data. The extraction results are then normalized in terms of name, mapped in terms of attribute fields, marked in terms of image source, verified in terms of image resolution, standardized in terms of resume time format, and standardized in terms of relationship event participants. This process generates the record to be matched, which includes the source identifier and extraction confidence level. Name retrieval features are generated based on the standard name, alias set, and cross-language mapping results in the records to be matched; text retrieval features are generated based on the encoding results of the text description; attribute retrieval features are generated based on the standardized attribute key-value pairs; and resume retrieval features are generated based on the resume timeline and job sequence. Using the aforementioned name retrieval features, text retrieval features, attribute retrieval features, and resume retrieval features, the knowledge base of individuals is retrieved through a combination of rule-based recall and vector-based recall to obtain the candidate record set.

3. The method for person entity recognition and fusion based on multimodal evidence fusion according to claim 1, characterized in that, The multimodal features include text features, attribute features, image features, resume features, and relationship event features; The text features are obtained by encoding the text description through a text encoding model; the attribute features are obtained by encoding standardized attribute key-value pairs; and the image features are obtained by encoding the image through face detection, target region extraction, and face representation. Time nodes, unit nodes, position nodes, and event nodes are extracted from work experience, education experience, and employer change records to form resume segments with valid time intervals. The resume segments are connected in chronological order to form a resume trajectory. The resume trajectory is encoded using a time-series coding model to obtain the resume features. Encode the relationship paths formed by relationship events filtered through the valid time interval of the resume to obtain the relationship event features; The text features, attribute features, image features, history features, and relationship event features corresponding to each record are combined into the multimodal features, which are used to calculate the corresponding modal similarity between the record to be matched and each record in the candidate record set; The resume similarity corresponding to the resume features is calculated according to the following formula: in, Indicates the similarity of resumes; The degree of trajectory overlap is calculated based on the ratio of the sum of the intersection durations to the sum of the union durations of the effective time intervals corresponding to the matching nodes. The matching nodes are identical or compatible unit nodes, job nodes, and event nodes. It represents the continuity of time and is calculated based on the number and length of unreasonable time intervals and the degree of violation of the order of events; The similarity of unit paths is calculated by weighted sequence matching based on unit level, unit alias, unit transfer edge, and job transfer edge. , and These are the preset non-negative coefficients corresponding to trajectory overlap, temporal continuity, and unit path similarity, respectively, and satisfying the following: .

4. The method for person entity recognition and fusion based on multimodal evidence fusion according to claim 3, characterized in that, The process of filtering relationship events by valid time interval of the resume and aligning the relationship path with the resume trajectory includes: The target time window is determined based on the effective time intervals corresponding to the employment, education, and unit changes in the resume trajectory. Relationship events whose occurrence time falls within the target time window or whose effective time intervals intersect with the target time window are filtered out. The filtered relationship events are used to construct a heterogeneous relationship graph with time stamps and source stamps. The unit nodes, job nodes, event nodes, and time sequence of the relationship path in the heterogeneous relationship graph are compared with the resume trajectory to generate the alignment results of record node matching, time sequence consistency, and path conflict. The similarity of relationships is calculated by weighting the degree of consistency of common units, common events, common paths, and path order within the target time window, and the alignment results are used to determine the consistency coefficient, conflict penalty factor, and hard conflict marker.

5. The method for person entity recognition and fusion based on multimodal evidence fusion according to claim 1, characterized in that, The determination of modality weights by combining evidence quality and modality availability includes: For each modality, calculate the source reliability, data integrity, cross-source consistency, and extraction confidence, and calculate the base weights according to the following formula: The base weights are adjusted based on the modal availability indicator: in, For modal indexing, , , , , and These represent text modality, attribute modality, image modality, resume modality, and relational event modality, respectively. Representing modes The basic weights; , , and Representing modes The reliability of the source, the integrity of the data, cross-source consistency, and the confidence level of extraction; , , and These are the preset non-negative coefficients corresponding to source reliability, data integrity, cross-source consistency, and extraction confidence, respectively, and they satisfy the following: Representing modes The availability indicator, whose value ranges from 0 to 1; Representing modes The base weight adjusted for availability; When the corresponding mode is missing, Set to 0; when the corresponding modality is not missing but the data quality is lower than the preset quality threshold, Set to an attenuation value greater than 0 and less than 1; When multiple sources provide mutually verifying information regarding the same person's attributes, events, or relationships, the corresponding [status / condition] is improved. When there are attribute value conflicts, event time conflicts, or relationship path conflicts among multiple sources, reduce the corresponding... This yields the availability-adjusted base weights used for subsequent consistency adjustments and conflict penalties.

6. The method for person entity recognition and fusion based on multimodal evidence fusion according to claim 5, characterized in that, The consistency coefficient includes the resume consistency coefficient. Consistency coefficient of relationship The conflict penalty factor includes a resume conflict penalty factor. and relationship conflict punishment factor ; Determined based on the consistency of unit paths, time continuity, and job sequence in the aforementioned resume trajectory. The determination is based on the proportion of common events in relational events, the similarity of relational paths, and the consistency of commonly associated nodes. Determined based on the degree of conflict in the resume Determined based on the degree of conflict in the relationship path ; For cases where verification is impossible solely due to missing resume data or relationship event data, both the corresponding consistency coefficient and conflict penalty factor are set to 0. The adjustment weights for each mode are calculated using the following formula: in, , , , and These represent the adjustment weights for the text modality, attribute modality, image modality, resume modality, and relational event modality, respectively. , , , and These represent the base weights after availability adjustment for the corresponding modalities; and They represent and Enhancement coefficients in text, attribute, and image modal weighting adjustments; and These represent the modal weighting adjustments in the resume. In the adjustment of enhancement coefficients and relational event modal weights The enhancement coefficient; When there are no hard conflicts and the sum of the adjusted weights is greater than 0, normalization is performed according to the following formula: in, Representing modes Adjusting the weights, Representing modes Normalized fusion weights; , , , and These represent the normalized fusion weights for text modality, attribute modality, image modality, resume modality, and relationship event modality, respectively. The fusion matching score is obtained by weighting and summing the corresponding modal similarities using the normalized fusion weights. in, Indicates the fusion matching score; Indicates text similarity; Indicates attribute similarity; Indicates image similarity; Indicates the similarity of resumes; Indicates the similarity of relationships.

7. The method for person entity recognition and fusion based on multimodal evidence fusion according to claim 6, characterized in that, The step of generating a consistency coefficient, a conflict penalty factor, and a hard conflict marker based on the history trajectory and the alignment result includes: When the resume trajectories of the record to be matched and the candidate record correspond to mutually exclusive positions or mutually exclusive units within the overlapping valid time interval, and there are no verified concurrent, acting, or job change events that can explain the differences in positions or units between the resume trajectories of the record to be matched and the candidate record, a hard conflict marker is generated. When there is a conflict between the relationship path and the unit, position or time information in the resume trajectory that cannot be explained by verified events, a hard conflict marker is generated. For inconsistencies in resumes or relationship paths that do not constitute a hard conflict, the corresponding conflict penalty factor and consistency coefficient are determined based on the degree of inconsistency. When the degree of inconsistency increases, the corresponding conflict penalty factor is increased or the corresponding consistency coefficient is decreased.

8. The method for person entity recognition and fusion based on multimodal evidence fusion according to claim 1, characterized in that, The step of prohibiting automatic fusion of hard-conflict records based on the hard-conflict marker, and performing attribute fusion on records without hard conflicts and whose fusion matching score reaches a preset threshold, includes: When the hard conflict markers of the resume or the hard conflict markers of the relationship exist, the corresponding records are prohibited from being automatically merged, and the corresponding records are retained as independent entities or generated as records to be reviewed. The preset threshold is a first preset threshold; when there is no hard conflict, if the fusion matching score is greater than or equal to the first preset threshold, the corresponding record is determined to be the same person entity and attribute fusion is performed; if the fusion matching score is greater than or equal to the second preset threshold and less than the first preset threshold, a record to be reviewed is generated; if the fusion matching score is less than the second preset threshold, the corresponding record is retained as an independent entity, wherein the first preset threshold is greater than the second preset threshold; For the record to be reviewed, the review result is received. If the review result confirms that the record is the same person entity, attribute fusion is performed. If the review result confirms that the record is a different person entity, the record is retained as an independent entity.

9. A person entity recognition and fusion device based on multimodal evidence fusion, characterized in that, include: The record generation unit is used to extract and standardize publicly available data on persons from multiple sources, generate records to be matched, and retrieve a set of candidate records accordingly. The feature construction unit is used to extract text, attributes, face images and resume features from the record to be matched and the record in the candidate record set, filter relationship events according to the effective time interval of the resume, align the relationship path with the resume trajectory, encode the relationship path, and obtain multimodal features and alignment results. The fusion computing unit is used to calculate the similarity of each modality based on the multimodal features, generate a consistency coefficient, a conflict penalty factor, and a hard conflict marker according to the history trajectory and the alignment result, determine the weight of each modality by combining evidence quality and modality availability, adjust the text, attribute, and image modal weights together using the consistency coefficient and the conflict penalty factor, normalize the adjusted weights and sum the weights of each modality similarity to obtain the fusion matching score; The entity generation unit is used to prevent the automatic fusion of hard conflict records according to the hard conflict marker, perform attribute fusion on records without hard conflict and whose fusion matching score reaches a preset threshold, generate a unified person entity, and write it into the person knowledge base.

10. A person entity recognition and fusion device based on multimodal evidence fusion, characterized in that, The system includes a memory and a processor. The memory stores a computer program that can be executed by the processor to implement a person entity recognition and fusion method based on multimodal evidence fusion as described in any one of claims 1 to 8.

Images