A marine-related enterprise operation monitoring data collection method

Abstract

The application relates to the technical field of data processing, and discloses a kind of marine-related enterprise operation monitoring data collection methods, comprising: configuring data source interface, incremental cursor and pulling cycle, establishing marine-related enterprise main body directory and identification change chain;According to cursor incremental pulling, failure breakpoint continues to pull;Calculate field fingerprint and deduplicate;Extract multiple field evidence, calculate and normalize degree of distinction weight according to appearance frequency of directory. Block recall candidate, calculate similarity degree weighting score field by field, merge, review or newly build according to threshold value;When merging, record old identification to new identification change edge, migrate historical monitoring data to current main body, and recalculate affected cycle summary.The application improves the accuracy and processing efficiency of subject identification in cross-source collection.

Description

Technical Field

[0001] This invention belongs to the field of data processing technology, specifically a method for collecting operational monitoring data of marine enterprises. Background Technology

[0002] The operational monitoring data of marine-related enterprises are scattered across the business systems of multiple departments, including registration, tax declaration, import and export customs declaration, vessel dynamics, fisheries management, and insurance payment. When conducting marine economic operation monitoring, data needs to be collected periodically from these data sources and aggregated by enterprise entity. During this process, records of the same enterprise in different data sources must be identified as belonging to the same entity; otherwise, data for the same entity will be split into multiple records, or data from different entities will be incorrectly merged, distorting the monitoring results aggregated by entity. The identification of the same entity is inconsistent across different data sources. Some data sources lack or incorrectly enter the unified social credit code; some data sources still use obsolete organization codes; enterprises may also undergo name changes, relocations, and restructuring; branch offices and parent companies often have highly similar names; and in fisheries management data sources, the owner of a fishing vessel and the actual operator are often not the same entity. When using exact name matching for aggregation, the same entity is split due to subtle differences in name; when using fuzzy matching, different entities with highly similar names are incorrectly merged.

[0003] Patent application CN113641826A discloses an entity alignment method, apparatus, and system for multi-source knowledge graph fusion. This method extracts entity features from the knowledge graph and generates an entity embedding matrix to obtain entity representations. It then enhances the entity representation based on the relationship information between the entity and its neighbors to obtain a complete entity representation. The final entity embedding matrix is ​​then obtained, and a loss function is calculated. A bidirectional global filtering strategy combining attributes is then used to process the loss function and entity attribute information to generate iterative positive and negative sample sets. The neural network model is iteratively trained using these sample sets, enabling the trained model to align and fuse multiple knowledge graphs. The similarity is obtained by weighting the distance between entity embedding vectors, the attribute similarity matrix, and the attribute value similarity matrix using hyperparameters. Global alignment is then used to select target entities from the candidate set without replacement to output aligned entity pairs.

[0004] However, the weighting coefficient for similarity is a global hyperparameter, treating different values ​​of the same field equally. In enterprise entity aggregation, common values ​​for fields such as the legal representative's name are frequently repeated in the directory. Their similarity does not necessarily indicate the same entity, while similarities for less common values ​​are highly indicative. Weighting with a global coefficient can lead to erroneous merging based on common values. Furthermore, it requires pre-aligned seed data to train the neural network model, which is difficult to obtain manually labeled alignment seeds in cross-departmental business database aggregation scenarios, and the model's performance depends on hyperparameter tuning. The similarity matrix is ​​a full matrix of two entity sets, and the computational complexity increases quadratically with the size of the entity list. Once the entity list reaches tens of thousands, it becomes difficult to complete within the specified timeframe in a single cycle. Summary of the Invention

[0005] The purpose of this invention is to provide a method for collecting operational monitoring data of marine enterprises, in order to solve the following problems when collecting operational monitoring data of marine enterprises from cross-data sources: the same subject is split due to inconsistent identification or different subjects are incorrectly merged; the field weights and the distinguishing ability of the field values ​​are mismatched; the model is trained by relying on pre-aligned seed data; uncertain record pairs cannot be isolated and handed over to manual review; the comparison calculation volume increases quadratically as the directory size increases; duplicate data is formed by repeated retrieval; and the time series is broken due to the failure to synchronize and reattach historical data after the subject is merged.

[0006] To achieve the above objectives, the present invention provides a method for collecting operational monitoring data of marine enterprises, the method comprising: S1. Configure the access interface, incremental cursor field and retrieval period of each data source, establish a list of marine-related enterprise entities and a chain list of entity identifier changes, and configure the field evidence set, automatic merging threshold and the threshold to be reviewed that is less than the automatic merging threshold.

[0007] S2. In each retrieval cycle, retrieve change records using the incremental cursor of each data source. Calculate the content fingerprint for each retrieved change record. If the content fingerprint is the same as the content fingerprint of an already aggregated record, discard the change record; otherwise, write it into the queue to be aggregated.

[0008] S3. For each change record in the queue to be aggregated, extract the values ​​of each field in the field evidence set, calculate the distinguishability that monotonically decreases as the number of occurrences of each field value in the subject directory table increases, and obtain the weight of the field after normalization.

[0009] S4. Recall candidate entities from the entity directory table by pressing the block key, calculate the field similarity between the change record and each candidate entity, and sum them according to the weight to obtain the matching score; when the maximum matching score is not less than the automatic merging threshold, merge the change record into the candidate entity with the highest matching score, i.e., the target entity; when it is between the threshold to be reviewed and the automatic merging threshold, write it into the queue to be reviewed; when it is less than the threshold to be reviewed, create a new entity record.

[0010] S5. During the merging process, if the subject identifier in the change record is inconsistent with the current identifier of the target subject, a new directed edge is added to the subject identifier change chain, pointing from the subject identifier to the current identifier. The chain is then traversed forward to reattach the historical marine enterprise operation monitoring data attached to the starting identifier of the directed edge to the target subject and the summary value of the affected period is recalculated.

[0011] Furthermore, the data source includes at least two of the following: registration data source, tax declaration data source, import and export customs declaration data source, ship dynamic data source, fisheries management data source, and insurance payment data source; the field evidence set includes a standardized name string, unified social credit code, legal representative's name, standardized registered address string, contact number, and a set of business scope characteristic words; the entity directory table stores the current identifier and field evidence values ​​of each entity, and the entity identifier change chain stores the directed edges pointing from the old identifier to the new identifier, as well as their change types and change times; the method of retrieving change records using incremental cursors from each data source includes: The incremental cursor field is taken from the record update timestamp field in the business table of each data source, and the initial value of the incremental cursor is taken from the creation time of the main directory table; The lower bound of the time window for each fetch is the incremental cursor, and the upper bound is the time when the fetch is initiated minus the preset write latency margin. The write latency margin is not less than the maximum transaction commit time of the corresponding data source. The data is fetched in batches according to the preset number of records per batch. The next batch is fetched only after each batch is successfully fetched and written. The incremental cursor is advanced after all batches are completed. If any batch fails to fetch or write, the write operation of the current batch is rolled back while the incremental cursor remains unchanged. When writing to the queue to be aggregated, the content fingerprint is used as the unique key. When the same content fingerprint is written repeatedly, only one record is retained, so that repeated retrieval in the same batch will not generate duplicate records.

[0012] Furthermore, the method for calculating the content fingerprint includes: The values ​​of each field are normalized according to a preset field sequence and then concatenated. The preset field sequence is arranged in lexicographical order of field names, so that the storage order of fields in the data source does not affect the content fingerprint; The normalization of values ​​for each field includes removing leading and trailing whitespace characters, converting full-width characters to half-width characters, converting English letters to lowercase, and uniformly setting empty values ​​to preset empty value placeholders; The normalized values ​​of each field are concatenated into a fingerprint input string using a preset separator. A secure hash value is calculated on the fingerprint input string, and the first 16 bytes are taken as the content fingerprint. The first two bytes of the fingerprint are stored in a fingerprint table in buckets, and the comparison is performed only within the corresponding bucket.

[0013] Furthermore, the method for generating the name normalization string includes: Convert full-width characters in the company name to half-width characters, traditional Chinese characters to simplified Chinese characters, and remove spaces and punctuation marks; Identify and extract the organizational form suffix from the end of the company name according to the preset organizational form suffix table, and map the extracted organizational form suffix as an organizational form code and store it separately. Identify and extract the administrative division names in parentheses and at the beginning of the enterprise name according to the preset administrative division name table, and map the extracted administrative division names as regional tags and store them separately. The remaining string after stripping is replaced with synonyms according to a preset alias dictionary to obtain the normalized name string.

[0014] Furthermore, the method for calculating the discrimination index and normalizing it to obtain the weights includes: Calculate the discriminant of the m-th field value: ;in, This indicates the total number of entities in the entity directory table. This indicates the number of times the value of the m-th field appears in the main directory table, where the number of occurrences is 1 when it is zero. Normalize the discriminative power of each field value to obtain the weight of the m-th field in this change record: Where M represents the number of fields in the change record that have non-null values; Fields with empty values ​​in this change record are not included in the normalization and their weights are zero, so that each change record calculates its weight according to the fields it actually has.

[0015] Furthermore, the method for calculating the field similarity between the change record and each candidate subject field by field includes: Unified Social Credit Code field: First, verify the Unified Social Credit Code in the change record according to the verification bit rules. If the verification passes and is completely the same as the Unified Social Credit Code of the candidate subject, the similarity is 1. If the verification passes but is not the same, the similarity is 0. If the verification fails, the field is regarded as empty, and the weights of the remaining non-empty fields are recalculated according to the normalization method. Name normalization string field: Calculate similarity: ;in, This represents the edit distance between two name normalization strings. and These represent the lengths of the two name normalization strings, This represents the geographic label consistency coefficient, indicating whether the geographic labels of the two are the same or both are space-time. Take 1, not at the same time Take a preset penalty coefficient that is less than 1; Normalized string field for registered address: The normalized string for registered address is divided into five levels: provincial, municipal, district / county, township / street and house number. The comparison is performed level by level. The preset level weights corresponding to the same level are accumulated as the similarity. The sum of the preset level weights is 1 and the level weight of township / street is greater than that of the other levels. For the fields of legal representative's name and contact number: the similarity is 1 if the values ​​are exactly the same, otherwise it is 0; Business scope feature word set field: The similarity is the ratio of the number of elements in the intersection of two feature word sets to the number of elements in the union.

[0016] Furthermore, the method for recalling candidate subjects from the subject directory table by block key includes: Configure at least three sets of block keys, wherein the block key includes the first few characters of the name normalization string concatenated with the code of the district / county where the registration is located, the last few digits of the contact number, and the name of the legal representative; An inverted index is created for the main entity directory table according to the block key of each group. The index key is the value of the block key, and the index value is the set of entity identifiers with that block key value. For each change record, calculate the group key value for each group, retrieve the corresponding subject identifier set from each inverted index and find the union of the sets, which will be used as the candidate subject set for that change record; When the candidate entity set is empty, the maximum matching score of the change record is recorded as zero, and a new entity record is created in the entity directory table according to the condition that it is less than the threshold to be reviewed.

[0017] Furthermore, the method for determining the automatic merging threshold and the threshold to be reviewed includes: Select a set of labeled sample record pairs, which includes positive sample pairs labeled as the same subject and negative sample pairs labeled as different subjects; For each pair in the set of sample records, a matching score is calculated based on the weight and the field similarity. The false merging rate and the missed merging rate are calculated separately at the candidate threshold. The false merging rate is the proportion of negative sample pairs with a matching score not less than the candidate threshold to all negative sample pairs. The missed merging rate is the proportion of positive sample pairs with a matching score less than the candidate threshold to all positive sample pairs. The smallest candidate threshold that makes the false merging rate no greater than the first allowable value is taken as the automatic merging threshold, and the largest candidate threshold that makes the missed merging rate no greater than the second allowable value is taken as the threshold to be reviewed.

[0018] Furthermore, the entity identifier is taken from the unified social credit code; if the unified social credit code is missing or fails the verification bit rule, it is taken as a combination of the normalized name string and the regional tag; the method of forward traversing the entity identifier change chain and reattaching the historical marine enterprise operation monitoring data to the target entity includes: Starting from the starting point of the directed edge, traverse forward step by step along the directed edge in the main identifier change list until the identifier with no outgoing edge is reached. All identifiers on the traversal path except the identifier with no outgoing edge are determined as the set of identifiers to be reattached. During the forward traversal, visited identifiers are recorded. When a visited identifier is visited again, the traversal of that branch is terminated and a loop alarm is output, so that loops in the main identifier change list do not cause the traversal to not terminate. Query the operational monitoring data of marine enterprises under each identifier in the set of identifiers to be reattached, update their main identifier field to the current identifier of the target entity, and retain the original main identifier and reattachment time in the reattachment record; After deduplicating the data according to the period to which it belongs, a list of affected periods is obtained. The summative value of each period in the list of affected periods is recalculated, and the summative values ​​before and after the recalculation are saved together.

[0019] Furthermore, the method also includes: For the record pairs in the queue to be reviewed, a manual review conclusion is received. If the manual review conclusion is that the record pairs belong to the same entity, the records are merged according to step S4 and the identifier change chain maintenance and historical data re-attachment are performed according to step S5. If the manual review conclusion is that the record pairs belong to different entities, a new entity record is created in the entity directory table. Records automatically merged within the pre-set evaluation cycle are randomly selected and sent for manual review. The proportion of records that are determined to be from different entities after review is counted as the measured false merging rate. When the measured false merging rate is greater than the first allowable value, the automatic merging threshold is increased by a preset step size, and the increase in a single evaluation cycle does not exceed a preset adjustment limit; when the measured false merging rate is not greater than the first allowable value and the proportion of the number of record pairs in the queue to be reviewed to the total number of change records in this evaluation cycle is greater than a preset review load limit, the threshold to be reviewed is decreased by the preset step size, and the decrease in a single evaluation cycle does not exceed the adjustment limit. The timing of each threshold adjustment, the threshold values ​​before and after the adjustment, and the measured indicators that triggered the adjustment are all saved.

[0020] Compared with the prior art, the beneficial effects of this invention are: This invention calculates the distinguishability of each field value based on its frequency of occurrence in the subject directory table and assigns weights accordingly. This prevents common values ​​from being assigned the same evidentiary strength as rare values, deriving weights from the data itself without manual setting. It uses an automatic merging threshold and a pending review threshold to form a three-branch decision, isolating uncertain record pairs in the pending review queue without directly affecting the aggregation results. It uses a block key inverted index to recall candidate subjects, reducing the number of comparisons from being proportional to the square of the directory size to being proportional to the directory size itself, allowing the process to be completed within the retrieval cycle even as the directory size increases. It uses incremental cursors and content fingerprints to suppress breakpoint re-retrieval and repeated writing, ensuring that repeated retrieval does not generate duplicate data. It uses a subject identifier change linked list to record identifier changes and traverses forward along the chain to reattach historical data and recalculate the sum of affected periods, ensuring that the time series of the same subject does not break at the merging point and that periods before and after are comparable. Attached Figure Description

[0021] Figure 1 This is a flowchart of a method for collecting operational monitoring data of marine enterprises according to the present invention; Figure 2 This is a schematic diagram illustrating the relinking of the main identifier change chain with historical data in an embodiment of the present invention. Detailed Implementation

[0022] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention are described clearly and completely below. Obviously, the described embodiments are only a part of the embodiments of this invention, not all of them. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this invention.

[0023] This invention is applicable to scenarios where operational monitoring data of marine enterprises are collected periodically from business systems of multiple departments. It is particularly suitable for collection processes where enterprise entity identifiers are inconsistent between data sources, the entity directory is large, and cross-period time series comparability needs to be maintained.

[0024] Example 1 like Figure 1 As shown in the figure, this embodiment provides a method for collecting operational monitoring data of marine enterprises, the method including: S1. Configure the access interface, incremental cursor field and retrieval period of each data source, establish a list of marine-related enterprise entities and a chain list of entity identifier changes, and configure the field evidence set, automatic merging threshold and the threshold to be reviewed that is less than the automatic merging threshold.

[0025] This embodiment was implemented in a coastal prefecture-level city, and accessed 6 data sources, namely, registration data source, tax declaration data source, import and export customs declaration data source, ship dynamic data source, fishery management data source, and insurance payment data source. All access interfaces were read-only view interfaces; the retrieval cycle was set to 1 day.

[0026] The total number of entities, N, at the time of creation of the entity directory table was 12860; the field evidence set includes six fields: normalized name string, unified social credit code, legal representative's name, normalized registered address string, contact number, and business scope characteristic word set. The automatic merging threshold is set to 0.85, and the threshold for pending review is set to 0.60.

[0027] S2. In each fetch cycle, fetch change records using the incremental cursors of each data source. Calculate the content fingerprint for each fetched change record. If the content fingerprint matches that of an already aggregated record, discard the change record; otherwise, write it into the pending aggregation queue. The incremental cursor field is taken from the record update timestamp field in the business table of each data source; the lower bound of the time window for each pull is the incremental cursor, and the upper bound is the time of initiation of this pull minus the preset write latency margin; the pull is performed in batches according to the preset number of records per batch, and the incremental cursor is advanced after all batches are completed; if any batch fails, the write of this batch is rolled back and the incremental cursor remains unchanged.

[0028] The write latency margin is set to 5 minutes, and the batch size is set to 500 records. The purpose of setting the write latency margin is as follows: Before a transaction on the data source side is committed, its record update timestamp has already been generated. If the upper bound of the fetch window is the current time, records in the commit process will not be actually read because their timestamps fall within the window, and the lower bound of the window in the next cycle has already been pushed forward to after that timestamp, so these records will be permanently missed. By rolling back the upper bound by 5 minutes, such records will fall into the window of the next cycle.

[0029] For each retrieved change record, the values ​​of each field are normalized according to a preset field sequence, then concatenated and a content fingerprint is calculated; if the content fingerprint is the same as the content fingerprint of the already collected records, the change record is discarded.

[0030] The secure hash value uses SHA-256, and the first 16 bytes are taken as the content fingerprint. The content fingerprint is then divided into 65,536 buckets based on its first two bytes. For example, on a certain day, 3,842 change records are retrieved from the tax return data source. Of these, 1,196 records (31.13%) have the same content fingerprint as already aggregated records and are discarded. The remaining 2,646 records are written to the pending aggregation queue. Even when the company's declarations have not changed, the records in this data source will still have their timestamps updated due to batch processing. If fingerprint comparison is not performed, the aforementioned 1,196 records will be repeatedly entered into the database and included in the aggregation.

[0031] S3. For each change record in the queue to be aggregated, extract the values ​​of each field in the field evidence set, calculate the distinguishability that monotonically decreases as the number of occurrences of each field value in the subject directory table increases, and obtain the weight of the field after normalization.

[0032] Count the number of times each field value appears in the main directory table, calculate the discrimination index using the following formula, and normalize it to obtain the weight: ; For example, in step S3, the Unified Social Credit Code field in a change record retrieved from the fisheries management data source is empty, while the values ​​of the other five fields are not empty. The frequency of each field value in the entity directory table is as follows: normalized name string 2 times, legal representative's name 23 times, normalized registered address string 3 times, contact number 1 time, and the rarest feature word in the business scope feature word set 47 times. The distinguishability of each field calculated using the above formula is 8.7689, 6.3282, 8.3635, 9.4620, and 5.6154, respectively, with a sum of 38.5379. After normalization, the weights of each field are 0.2275, 0.1642, 0.2170, 0.2455, and 0.1457, respectively. It is evident that the legal representative's name appears 23 times in the directory, and its distinguishability and weight are significantly lower than the contact number, which appears only once, consistent with the actual indicative power of this field value.

[0033] S4. Recall candidate entities from the entity directory table by pressing the block key, calculate the field similarity between the change record and each candidate entity, and sum them according to the weight to obtain the matching score; when the maximum matching score is not less than the automatic merging threshold, merge the change record into the candidate entity with the highest matching score, i.e., the target entity; when it is between the threshold to be reviewed and the automatic merging threshold, write it into the queue to be reviewed; when it is less than the threshold to be reviewed, create a new entity record.

[0034] In step S4, this embodiment configures three sets of block keys. The first set is the concatenation of the first two characters of the normalized name string and the district / county code of the registered location. The second set is the last six digits of the contact phone number. The third set is the name of the legal representative. With a directory size of 12,860, if each change record is compared with all entities one by one, each change record would require 12,860 comparisons. Using the above three-set block keys, the average number of candidate entities per change record is 7.3, meaning the number of comparisons per change record is reduced to 7.3, a 99.94% decrease compared to a full comparison.

[0035] In step S4, the field similarity of each candidate entity is calculated field by field and summed according to the weights. The calculation method of the field similarity includes: the unified social credit code field is compared after verification according to the check digit rule; the normalized name string field is calculated according to the edit distance and the consistency coefficient of the regional marker; the normalized registered address string field is compared step by step according to five levels and the level weights are accumulated; the legal representative name field and the contact number field are judged to be completely identical; the business scope feature word set field is calculated according to the ratio of the number of intersection and union elements.

[0036] In step S4, the regional label consistency coefficient When the geographical markers are different, the weight is 0.85; the weights of the five levels of the registered address are as follows: provincial level 0.10, municipal level 0.15, district / county level 0.20, township / street level 0.40, and house number 0.15, with a sum of 1.00.

[0037] For example, in step S4, based on the aforementioned change record, the similarity of each field between it and the candidate subject with the highest matching score is calculated as follows.

[0038] Name normalization string: The normalized name string of this change record is "×× Blue Ocean Distant Water Fisheries", with a length of 8. The normalized name string of the candidate entity is "×× Blue Ocean Distant Water Fisheries Development", with a length of 10. The edit distance between the two is 2. The geographic tags of both are empty. If we take 1, the similarity is 1 minus 2 divided by 10, which is 0.80.

[0039] Legal representative's name: Both are exactly the same, with a similarity of 1.00.

[0040] The standardized string of registered addresses is the same at the provincial, municipal, district / county, and township / street levels, but different at the house number. The similarity is 0.10 + 0.15 + 0.20 + 0.40, which equals 0.85.

[0041] Contact number: Both are exactly the same, with a similarity of 1.00.

[0042] Business scope feature word set: The feature word set of this change record contains 37 elements, the feature word set of the candidate subject contains 41 elements, the intersection contains 29 elements, the union contains 49 elements, and the similarity is 29 divided by 49, which is 0.5918.

[0043] The weighted summation is performed based on the weights of the fields, which are 0.18200, 0.16420, 0.18445, 0.24550 and 0.08623 respectively. The matching score is 0.8624, which is not less than the automatic merging threshold of 0.85. Therefore, this change record is merged into the candidate subject.

[0044] In contrast, another change record shares the same similarity with the candidate entity that matches it most highly in terms of name normalization string, registered address normalization string, and business scope feature word set as the previous example. However, the legal representative's name is the same, but the contact number is different. Its matching score is 0.18200 + 0.16420 + 0.18445 + 0 + 0.08623, or 0.6169, which is between the pending review threshold of 0.60 and the automatic merging threshold of 0.85. Therefore, it is added to the pending review queue and not merged yet. Another change record has a different legal representative's name and contact number. Its matching score is 0.4527, which is less than the pending review threshold of 0.60. Therefore, a new entity record is created in the entity directory table.

[0045] In step S1, the method for determining the automatic merging threshold and the threshold to be reviewed includes: selecting a set of labeled sample record pairs; calculating the matching score for each pair; calculating the false merging rate and the missed merging rate on the candidate thresholds respectively; taking the smallest candidate threshold that makes the false merging rate no greater than the first allowable value as the automatic merging threshold, and taking the largest candidate threshold that makes the missed merging rate no greater than the second allowable value as the threshold to be reviewed.

[0046] In this embodiment, 2000 pairs of labeled sample records were selected, including 1000 positive sample pairs and 1000 negative sample pairs. The first tolerance value was set to 0.5%, and the second tolerance value was set to 1.0%. At the candidate threshold of 0.85, the false merging rate was 0.4%, and the false merging rate was 6.2%; at the candidate threshold of 0.60, the false merging rate was 0.9%, and the false merging rate was 8.7%. Therefore, the automatic merging threshold was set to 0.85, and the threshold for review was set to 0.60. The proportion of sample pairs falling into the review interval was 6.8% of all sample pairs.

[0047] S5. During the merging process, if the subject identifier in the change record is inconsistent with the current identifier of the target subject, a new directed edge is added to the subject identifier change chain, pointing from the subject identifier to the current identifier. The chain is then traversed forward to reattach the historical marine enterprise operation monitoring data attached to the starting identifier of the directed edge to the target subject and the summary value of the affected period is recalculated.

[0048] The main identifier is the unified social credit code. If the unified social credit code is missing or fails the verification rule, it is a combination of the normalized name string and the regional mark.

[0049] like Figure 2 As shown in this embodiment, a marine-related enterprise underwent two identity changes: its original enterprise name corresponded to the old identity K1, and after changing its name in June 2025, it corresponded to the old identity K2. Subsequently, a unified social credit code was added to the registration data source, corresponding to the current entity identity K3. The entity identity change chain list contains two directed edges: one from K1 to K2 and the other from K2 to K3, with the change types being name change and code addition, respectively.

[0050] Starting from the starting point of the directed edge, traverse forward to the point where there is no outgoing edge. All the marks on the traversal path except for the mark with no outgoing edge are identified as the set of marks to be reattached. During the forward traversal, the marks that have been visited are recorded. When a mark that has been visited is visited again, the traversal of that branch is terminated and a loop alarm is output.

[0051] The traversal path starting from K1 is K1, K2, K3, and the set of identifiers to be reattached is {K1, K2}. The purpose of setting up loop detection is that occasional errors in identifier backfilling in the data source will generate a reverse edge from K3 to K1. If visited identifiers are not recorded, the traversal will loop between K1, K2, and K3 without terminating.

[0052] Query the operation monitoring data under each identifier in the set of identifiers to be re-attached, update the main identifier field of the main identifier to the current identifier of the target main identifier; after deduplicating the data to be re-attached according to the period to which it belongs, obtain the list of affected periods, and recalculate the summary value for each period.

[0053] A total of 1247 historical monitoring data entries were attached to K1 and K2, covering the period from January 2023 to May 2025. After deduplication, the list of affected periods included 29 monthly periods. After the reattachment was completed, the summary value of each of these 29 periods was recalculated, and the summaries before and after the recalculation were saved together. The total number of reattached records (1247) and the list of affected periods were then output.

[0054] The method further includes: receiving manual review conclusions for records in the queue to be reviewed; extracting automatically merged records within the current evaluation period according to a preset evaluation period and sending them for manual review, and calculating the measured false merging rate; when the measured false merging rate is greater than the first allowable value, increasing the automatic merging threshold by a preset step size.

[0055] In this embodiment, the evaluation period is set to one quarter, the preset step size is set to 0.01, the adjustment limit is set to 0.05, and the review load limit is set to 3%. For example, in a certain quarter, 500 pairs of automatically merged records are extracted and sent for manual review. Four pairs are determined to be from different entities, resulting in a measured false merging rate of 0.8%, which is greater than the first allowable value of 0.5%. Therefore, the automatic merging threshold is increased from 0.85 to 0.86, and the timing of this adjustment, the threshold values ​​before and after the adjustment, and the measured false merging rate of 0.8% are all recorded. In the following quarter, the measured false merging rate drops to 0.4%, and no further adjustment is made.

[0056] Example 2 This embodiment provides a method for collecting operational monitoring data of marine enterprises when the data source does not provide an incremental query interface. The difference from Embodiment 1 is the method of pulling the fishery management data source in step S2. The method of pulling other data sources and the remaining steps are the same as in Embodiment 1.

[0057] The fisheries management data source only provides full snapshot files and does not provide an incremental interface for querying by record update timestamps. To address this, the full snapshot files are divided into a predetermined number of data blocks based on the hash value of the subject identifier. Records within each data block are sorted by content fingerprint, and a block fingerprint is calculated. The block fingerprints are then merged pairwise at each level to obtain the root fingerprint. The root fingerprint of the current snapshot is compared with the root fingerprint of the previous snapshot. If they match, it is determined that there has been no change in the current snapshot, and the snapshot content is not parsed further. If they differ, the block fingerprints are compared downwards at each level. Only records in data blocks whose block fingerprints have changed are parsed, and the content fingerprint is calculated again as in Example 1 and written into the queue to be aggregated.

[0058] The number of data blocks is set to 1024. For example, a full snapshot contains 186,420 records. After comparison, only the block fingerprints of 3 data blocks change. Parsing these 3 data blocks yields 547 records, a 99.71% reduction in parsing volume compared to parsing all records. Among these, 112 records are written to the collection queue after content fingerprint comparison.

[0059] This embodiment also provides a handling method when the fishing vessel owner and the actual operator are inconsistent. The records of the fisheries management data source contain both the fishing vessel owner's identifier and the actual operator's identifier. When the two are inconsistent, the actual operator's identifier is used as the collection subject for the operational monitoring data in steps S3 to S5, and the correspondence between the fishing vessel owner's identifier and the actual operator's identifier is written into the associated subject table. If the actual operator of the same fishing vessel changes in different periods, the operational monitoring data of the fishing vessel is split according to the time of change. Data before the time of change is collected to the actual operator before the change, and data after the time of change is collected to the actual operator after the change, so as to avoid the output of the same fishing vessel being counted twice on two subjects.

[0060] It should be noted that the above embodiments use the collection of operational monitoring data of marine enterprises as an example for illustration, but the concept of the present invention is also applicable to other scenarios that require data collection across multiple data sources by subject and where the subject identifiers are inconsistent. In such scenarios, the set of field evidence can be replaced with a set of fields with discriminative capabilities in the corresponding scenario, while the remaining steps remain the same.

[0061] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for collecting operational monitoring data of marine-related enterprises, characterized in that, The method includes: S1. Configure the access interface, incremental cursor field and retrieval period of each data source, establish a list of marine-related enterprise entities and a chain list of entity identifier changes, and configure the field evidence set, automatic merging threshold and the threshold to be reviewed that is less than the automatic merging threshold. S2. In each retrieval cycle, retrieve change records using the incremental cursor of each data source. Calculate the content fingerprint for each retrieved change record. If the content fingerprint is the same as the content fingerprint of the already collected record, discard the change record; otherwise, write it into the queue to be collected. S3. Extract the values ​​of each field in the field evidence set for each change record in the queue to be collected, calculate the distinguishability that monotonically decreases as the number of occurrences of each field value in the main entity directory table increases, and obtain the weight of the field after normalization. S4. Recall candidate entities from the entity directory table by pressing the block key, calculate the field similarity between the change record and each candidate entity, and sum them according to the weight to obtain the matching score; when the maximum matching score is not less than the automatic merging threshold, merge the change record into the candidate entity with the highest matching score, i.e., the target entity; when the score is between the threshold to be reviewed and the automatic merging threshold, write it into the queue to be reviewed; when the score is less than the threshold to be reviewed, create a new entity record. S5. During the merging process, if the subject identifier in the change record is inconsistent with the current identifier of the target subject, a new directed edge is added to the subject identifier change chain, pointing from the subject identifier to the current identifier. The chain is then traversed forward to reattach the historical marine enterprise operation monitoring data attached to the starting identifier of the directed edge to the target subject and the summary value of the affected period is recalculated.

2. The method for collecting operational monitoring data of marine enterprises according to claim 1, characterized in that, The data source includes at least two of the following: registration data source, tax declaration data source, import and export customs declaration data source, ship dynamic data source, fisheries management data source, and insurance payment data source; the field evidence set includes a standardized name string, unified social credit code, legal representative's name, standardized registered address string, contact number, and a set of business scope characteristic words; the entity directory table stores the current identifier and field evidence values ​​of each entity, and the entity identifier change chain stores the directed edges pointing from the old identifier to the new identifier, as well as their change types and change times; The method for retrieving change records using incremental cursors from each data source includes: The incremental cursor field is taken from the record update timestamp field in the business table of each data source, and the initial value of the incremental cursor is taken from the creation time of the main directory table; The lower bound of the time window for each fetch is the incremental cursor, and the upper bound is the time when the fetch is initiated minus the preset write latency margin. The write latency margin is not less than the maximum transaction commit time of the corresponding data source. The data is fetched in batches according to the preset number of records per batch. The next batch is fetched only after each batch is successfully fetched and written. The incremental cursor is advanced after all batches are completed. If any batch fails to fetch or write, the write operation of the current batch is rolled back while the incremental cursor remains unchanged. When writing to the queue to be aggregated, the content fingerprint is used as the unique key. When the same content fingerprint is written repeatedly, only one record is retained, so that repeated retrieval in the same batch will not generate duplicate records.

3. The method for collecting operational monitoring data of marine enterprises according to claim 2, characterized in that, The method for calculating the content fingerprint includes: The values ​​of each field are normalized according to a preset field sequence and then concatenated. The preset field sequence is arranged in lexicographical order of field names, so that the storage order of fields in the data source does not affect the content fingerprint; The normalization of values ​​for each field includes removing leading and trailing whitespace characters, converting full-width characters to half-width characters, converting English letters to lowercase, and uniformly setting empty values ​​to preset empty value placeholders; The normalized values ​​of each field are concatenated into a fingerprint input string using a preset separator. A secure hash value is calculated on the fingerprint input string, and the first 16 bytes are taken as the content fingerprint. The first two bytes of the fingerprint are stored in a fingerprint table in buckets, and the comparison is performed only within the corresponding bucket.

4. The method for collecting operational monitoring data of marine enterprises according to claim 3, characterized in that, The method for generating the name normalization string includes: Convert full-width characters in the company name to half-width characters, traditional Chinese characters to simplified Chinese characters, and remove spaces and punctuation marks; Identify and extract the organizational form suffix from the end of the company name according to the preset organizational form suffix table, and map the extracted organizational form suffix as an organizational form code and store it separately. Identify and extract the administrative division names in parentheses and at the beginning of the enterprise name according to the preset administrative division name table, and map the extracted administrative division names as regional tags and store them separately. The remaining string after stripping is replaced with synonyms according to a preset alias dictionary to obtain the normalized name string.

5. The method for collecting operational monitoring data of marine enterprises according to claim 4, characterized in that, The method for calculating the discrimination index and normalizing it to obtain the weights includes: Calculate the discriminant of the m-th field value: ;in, This indicates the total number of entities in the entity directory table. This indicates the number of times the value of the m-th field appears in the main directory table, where the number of occurrences is 1 when it is zero. Normalize the discriminative power of each field value to obtain the weight of the m-th field in this change record: Where M represents the number of fields in the change record that have non-null values; Fields with empty values ​​in this change record are not included in the normalization and their weights are zero, so that each change record calculates its weight according to the fields it actually has.

6. The method for collecting operational monitoring data of marine enterprises according to claim 5, characterized in that, The method for calculating the field similarity between the change record and each candidate subject, field by field, includes: Unified Social Credit Code field: First, verify the Unified Social Credit Code in the change record according to the verification bit rules. If the verification passes and is completely the same as the Unified Social Credit Code of the candidate subject, the similarity is 1. If the verification passes but is not the same, the similarity is 0. If the verification fails, the field is regarded as empty, and the weights of the remaining non-empty fields are recalculated according to the normalization method. Name normalization string field: Calculate similarity: ;in, This represents the edit distance between two name normalization strings. and These represent the lengths of the two name normalization strings, This represents the geographic label consistency coefficient, indicating whether the geographic labels of the two are the same or both are space-time. Take 1, not at the same time Take a preset penalty coefficient that is less than 1; Normalized string field for registered address: The normalized string for registered address is divided into five levels: provincial, municipal, district / county, township / street and house number. The comparison is performed level by level. The preset level weights corresponding to the same level are accumulated as the similarity. The sum of the preset level weights is 1 and the level weight of township / street is greater than that of the other levels. For the fields of legal representative's name and contact number: the similarity is 1 if the values ​​are exactly the same, otherwise it is 0; Business scope feature word set field: The similarity is the ratio of the number of elements in the intersection of two feature word sets to the number of elements in the union.

7. The method for collecting operational monitoring data of marine enterprises according to claim 6, characterized in that, The method for recalling candidate subjects from the subject directory table by using a block key includes: Configure at least three sets of block keys, wherein the block key includes the first few characters of the name normalization string concatenated with the code of the district / county where the registration is located, the last few digits of the contact number, and the name of the legal representative; An inverted index is created for the main entity directory table according to the block key of each group. The index key is the value of the block key, and the index value is the set of entity identifiers with that block key value. For each change record, calculate the group key value for each group, retrieve the corresponding subject identifier set from each inverted index and find the union of the sets, which will be used as the candidate subject set for that change record; When the candidate entity set is empty, the maximum matching score of the change record is recorded as zero, and a new entity record is created in the entity directory table according to the condition that it is less than the threshold to be reviewed.

8. The method for collecting operational monitoring data of marine enterprises according to claim 7, characterized in that, The methods for determining the automatic merging threshold and the threshold to be reviewed include: Select a set of labeled sample record pairs, which includes positive sample pairs labeled as the same subject and negative sample pairs labeled as different subjects; For each pair in the set of sample records, a matching score is calculated based on the weight and the field similarity. The false merging rate and the missed merging rate are calculated separately at the candidate threshold. The false merging rate is the proportion of negative sample pairs with a matching score not less than the candidate threshold to all negative sample pairs. The missed merging rate is the proportion of positive sample pairs with a matching score less than the candidate threshold to all positive sample pairs. The smallest candidate threshold that makes the false merging rate no greater than the first allowable value is taken as the automatic merging threshold, and the largest candidate threshold that makes the missed merging rate no greater than the second allowable value is taken as the threshold to be reviewed.

9. The method for collecting operational monitoring data of marine enterprises according to claim 8, characterized in that, The main identifier is taken from the unified social credit code. If the unified social credit code is missing or fails the verification bit rule, it is taken as a combination of the name normalization string and the regional mark. The method of forward traversing the linked list along the main entity identifier and reattaching historical marine enterprise operation monitoring data to the target entity includes: Starting from the starting point of the directed edge, traverse forward step by step along the directed edge in the main identifier change list until the identifier with no outgoing edge is reached. All identifiers on the traversal path except the identifier with no outgoing edge are determined as the set of identifiers to be reattached. During the forward traversal, visited identifiers are recorded. When a visited identifier is visited again, the traversal of that branch is terminated and a loop alarm is output, so that loops in the main identifier change list do not cause the traversal to not terminate. Query the operational monitoring data of marine enterprises under each identifier in the set of identifiers to be reattached, update their main identifier field to the current identifier of the target entity, and retain the original main identifier and reattachment time in the reattachment record; After deduplicating the data according to the period to which it belongs, a list of affected periods is obtained. The summative value of each period in the list of affected periods is recalculated, and the summative values ​​before and after the recalculation are saved together.

10. The method for collecting operational monitoring data of marine enterprises according to claim 9, characterized in that, The method further includes: For the record pairs in the queue to be reviewed, a manual review conclusion is received. If the manual review conclusion is that the record pairs belong to the same entity, the records are merged according to step S4 and the identifier change chain maintenance and historical data re-attachment are performed according to step S5. If the manual review conclusion is that the record pairs belong to different entities, a new entity record is created in the entity directory table. Records automatically merged within the pre-set evaluation cycle are randomly selected and sent for manual review. The proportion of records that are determined to be from different entities after review is counted as the measured false merging rate. When the measured false merging rate is greater than the first allowable value, the automatic merging threshold is increased by a preset step size, and the increase in a single evaluation cycle does not exceed a preset adjustment limit; when the measured false merging rate is not greater than the first allowable value and the proportion of the number of record pairs in the queue to be reviewed to the total number of change records in this evaluation cycle is greater than a preset review load limit, the threshold to be reviewed is decreased by the preset step size, and the decrease in a single evaluation cycle does not exceed the adjustment limit. The timing of each threshold adjustment, the threshold values ​​before and after the adjustment, and the measured indicators that triggered the adjustment are all saved.

Images