Letter text and seal analysis system and method based on multi-modal dynamic association

Abstract

The invention discloses a letter text and seal analysis system and method based on multi-modal dynamic association, and relates to the technical field of image recognition and letter text analysis, the system comprises a multi-modal data fusion module, a space-time association analysis module, a semantic visual cross validation module and a dynamic risk scoring module, and the modules are connected; the multi-modal data fusion module is used for feature extraction and alignment of texts and seals; the time-space correlation analysis module is used for establishing a seal using behavior model based on historical data; the semantic visual cross validation module is used for calculating the matching degree of the text and the seal; and the dynamic risk scoring module is used for comprehensively evaluating the credibility of the letter. The method is used for solving the problem of intelligent detection of counterfeiting means based on'real seal + tampering content '.

Description

Technical Field

[0001] This invention relates to the field of image recognition and letter text analysis technology, and more specifically, to a letter text and seal analysis system and method based on multimodal dynamic association. Background Technology

[0002] In key sectors such as finance, law, and government affairs, correspondence, as legally binding official documents, directly impacts the security of major economic transactions and the validity of legal documents through its authenticity and completeness. With the acceleration of digitalization and the continuous upgrading of forgery technology, traditional anti-counterfeiting methods face unprecedented challenges. Currently, forgers have developed advanced forgery techniques based on "genuine seals + altered content." These techniques retain the visual characteristics of genuine seals while meticulously modifying key textual information, rendering traditional single-modality detection methods completely ineffective.

[0003] Existing technological systems have significant limitations: First, most systems employ fragmented verification methods, either relying solely on computer vision to detect the authenticity of seals or solely on natural language processing to analyze text content. This approach of processing text and seal information independently completely ignores the inherent connection between the two, making it impossible for the system to detect forgeries where the seal remains authentic but the content has been altered. Second, traditional verification methods generally employ static analysis, focusing only on the characteristics of the document itself while completely ignoring the spatiotemporal behavioral data generated during the use of the seal. This limitation prevents the system from recognizing seals used outside of working hours, in unusual locations, or under other abnormal circumstances, even if the seal itself is genuine.

[0004] To address the above problems, this invention proposes a solution. Summary of the Invention

[0005] To overcome the aforementioned deficiencies of the prior art, embodiments of the present invention provide a system and method for analyzing correspondence text and seals based on multimodal dynamic association. By integrating natural language processing and computer vision technologies, the system achieves dynamic association verification between text and seals, solving the problem of intelligent detection of forgery methods based on "genuine seal + altered content".

[0006] To achieve the above objectives, the present invention provides the following technical solution: A multimodal dynamic association-based letter text and seal analysis system includes: a multimodal data fusion module, a spatiotemporal association analysis module, a semantic-visual cross-validation module, and a dynamic risk scoring module, with connections between the modules. The multimodal data fusion module extracts key information from the letter text through semantic analysis and obtains the visual features of the seal, which are obtained through seal detection, extraction, and authenticity identification. The spatiotemporal association analysis module analyzes the rationality of current seal usage based on a spatiotemporal compliance verification formula using historical seal usage data. The semantic-visual cross-validation module establishes a dynamic text-seal association model and outputs the matching degree between the text content and seal features. The dynamic risk scoring module outputs a comprehensive risk score for the letter based on the matching degree between the text content and seal features.

[0007] In a preferred embodiment, the multimodal data fusion module includes a key information extraction step for the letter text, specifically: the BERT model encodes the letter text and outputs a semantic feature vector; Key fields in the email are extracted using named entity recognition technology. The extracted fields are then matched and verified against a preset template. If a field is missing or its format is incorrect, it is marked as "incomplete information".

[0008] In a preferred embodiment, the multimodal data fusion module includes acquiring the visual features of the seal, specifically by performing convolution processing on the seal image using a ResNet model to output a visual feature vector.

[0009] In a preferred embodiment, the spatiotemporal correlation analysis module performs spatiotemporal compliance checks based on the current stamping time, historical average stamping time, current GPS coordinates, historical frequently used locations, geofence radius, and time decay coefficient.

[0010] In a preferred embodiment, the semantic visual cross-validation module specifically involves: aligning key information of the letter text with the features of the seal area through a cross-modal attention mechanism, and assigning attention weights to key entities in the text and corresponding areas in the seal; the multimodal fusion result is obtained by calculating the text feature vector, the seal feature vector, the attention weights, and the number of aligned feature pairs.

[0011] In a preferred embodiment, the dynamic risk scoring module will employ a multi-factor fusion algorithm.

[0012] In a preferred embodiment, if the matching verification fails in the key information extraction step of the letter text, the weight adjustment mechanism of the dynamic risk scoring module is triggered, specifically: the weight coefficient of the multi-factor fusion algorithm is increased from the first set of weight coefficients to the second set of weight coefficients; the spatiotemporal anomaly probability weight and the forgery probability weight are simultaneously reduced to the first preset lower limit and the second preset lower limit.

[0013] In a preferred embodiment, the semantic visual cross-validation module further includes an abnormal matching alarm function, specifically: when the matching degree between the text content and the seal features... When the value falls below the preset second threshold, an alarm signal is generated and the letter is marked as pending review and automatically pushed to the manual review queue; mismatched text-stamp areas are highlighted; historical matching degree data of similar letters are retrieved, and the distance between the current matching degree and historical data is calculated using the K-nearest neighbor algorithm. If the distance exceeds the preset reference standard deviation, it is determined to be an abnormal outlier and the alarm level is upgraded to high risk.

[0014] A method for analyzing letter text and seals based on multimodal dynamic association includes the following steps: Key information in the letter text is extracted through semantic analysis, and visual features of the seal are obtained. These visual features are obtained by detecting, extracting, and identifying the authenticity of the seal in the letter. The rationality of current seal usage behavior is analyzed by combining historical seal usage data with a spatiotemporal compliance verification formula. Establish a dynamic association model between text and seal, and output the matching degree between text content and seal features; Based on the degree of matching between the text content and the seal features, the overall risk score of the letter is output.

[0015] The technical effects and advantages of the letter text and seal analysis system and method based on multimodal dynamic association of the present invention are as follows: This invention's multimodal fusion analysis method significantly improves the system's anti-counterfeiting capabilities, effectively identifying various complex forgery methods and overcoming the limitations of traditional single-modal detection, achieving cross-modal collaborative verification. The system employs a BERT-ResNet dual-channel neural network architecture. The text processing channel extracts deep semantic features of the letter text through a pre-trained language model, while the seal processing channel utilizes a deep convolutional network to capture the microscopic visual features of the seal. The two channels achieve precise alignment of the feature space through a designed cross-modal attention mechanism, enabling the system to detect "genuine seal + tampered content" forgeries that are difficult to detect using traditional single-modal methods.

[0016] The introduction of spatiotemporal behavior analysis enables the system to detect abnormal behavior, breaking through the constraints of static verification and introducing a dynamic behavior analysis dimension. By constructing a knowledge graph of seal usage behavior, the system can intelligently analyze multi-dimensional features such as seal usage time, location, and frequency. In specific implementation, the system adopts an adaptive spatiotemporal modeling algorithm to create personalized profiles for seal usage behavior of different departments and levels; the dynamic risk assessment mechanism provides more refined judgment criteria and supports differentiated processing strategies. This effectively solves the complex forgery methods of "genuine seal + tampered content" currently faced in the field of anti-counterfeiting of correspondence. Attached Figure Description

[0017] Figure 1 This is a schematic diagram of the structure of a letter text and seal analysis system based on multimodal dynamic association according to the present invention.

[0018] Figure 2 This is a flowchart illustrating a method for analyzing letter text and seals based on multimodal dynamic association according to the present invention. Detailed Implementation

[0019] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0020] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising..." does not exclude the presence of additional identical elements in the process, method, article, or apparatus that includes said element.

[0021] Example 1, Figure 1 The present invention provides a system for analyzing letter text and seals based on multimodal dynamic association, comprising: a multimodal data fusion module, a spatiotemporal association analysis module, a semantic visual cross-validation module, and a dynamic risk scoring module, wherein the modules are interconnected; The multimodal data fusion module is used to extract key information from the letter text through semantic analysis and obtain the visual features of the seal. The visual features of the seal are obtained by detecting, extracting and identifying the authenticity of the seal in the letter. The spatiotemporal correlation analysis module is used to analyze the rationality of current seal usage behavior based on spatiotemporal compliance verification formulas by combining historical seal usage data. The semantic-visual cross-validation module is used to establish a dynamic association model between text and seal, and output the matching degree between text content and seal features. The dynamic risk scoring module is used to output a comprehensive risk score for the letter based on the multimodal fusion results.

[0022] In this embodiment, the multimodal data fusion module is used to extract key information of the letter text through semantic analysis and obtain the visual features of the seal. The visual features of the seal are obtained by detecting, extracting and identifying the authenticity of the seal in the letter. In this embodiment, the multimodal data fusion module includes a step for extracting key information from letter text, specifically: The BERT model encodes the email text and outputs semantic feature vectors; Key fields in the email were extracted using named entity recognition technology. The extracted fields are matched and validated against the preset template. If a field is missing or the format is incorrect, it is marked as "incomplete information". The ResNet model is used to perform convolution processing on the seal image to output a visual feature vector.

[0023] In this embodiment, the spatiotemporal correlation analysis module is used to analyze the rationality of the current seal usage behavior based on the spatiotemporal compliance verification formula by combining historical seal usage data; In this embodiment, the spatiotemporal correlation analysis module will perform a spatiotemporal compliance check, calculated using the following formula:

[0024] in For spatiotemporal anomaly probability, This is the current stamping time. This represents the historical average printing time. Current GPS coordinates This is a commonly used historical location. The radius of the geofence. and This is the time decay coefficient.

[0025] In this embodiment, the semantic visual cross-validation module is used to establish a dynamic association model between text and seal, and output the matching degree between text content and seal features.

[0026] In this embodiment, the semantic visual cross-validation module specifically comprises: By using a cross-modal attention mechanism, key information in the letter text is aligned with the features of the seal area, and attention weights are assigned to key entities in the text and corresponding areas in the seal. The formula for calculating semantic-visual matching degree is:

[0027] in, The degree of matching between the text content and the seal features. For text feature vectors, For the seal feature vector, For attention weights, Number of feature pairs for alignment.

[0028] In this embodiment, the dynamic risk scoring module is used to output a comprehensive risk score for the letter based on the multimodal fusion results.

[0029] In this embodiment, the dynamic risk scoring module will employ a multi-factor fusion algorithm:

[0030] in , , The first set of weight coefficients and , To determine the probability of seal forgery, is the steepness coefficient of the Sigmoid function, used to map the forgery probability to a non-linear risk contribution.

[0031] In this embodiment, the probability of seal forgery is... The specific calculation formula is as follows:

[0032] in The features of the seal to be tested, To verify the characteristics of a genuine seal, To preset the first threshold, It is a structural similarity index. , For the true seal of history The distribution parameters.

[0033] It should be noted that the preset first threshold is the Euclidean distance threshold used to determine seal forgery; anything exceeding this threshold is considered abnormal.

[0034] In this embodiment, if the matching verification fails in the key information extraction step of the letter text, the weight adjustment mechanism of the dynamic risk scoring module is triggered, specifically as follows: The weight coefficients of the multi-factor fusion algorithm are increased from the first group of weight coefficients to the second group of weight coefficients; It should be noted that the first set of weight coefficients are the initially set weight values, used to balance the contributions of matching degree, spatiotemporal anomaly probability and forgery probability in risk scoring; the second set of weight coefficients are higher weights after adjustment when the verification of key text information fails, in order to strengthen the scoring influence of text matching items. Simultaneously reduce the spatiotemporal anomaly probability weight and the forgery probability weight to the first preset lower limit and the second preset lower limit; It should be noted that the first preset lower limit is the spatiotemporal anomaly probability weight when text verification fails. The minimum value that is reduced; the second preset lower limit is the forgery probability weight when text verification fails. The minimum value that was reduced.

[0035] In this embodiment, the semantic visual cross-validation module further includes an abnormal matching alarm function, specifically: When the text content matches the seal features When the value is below the preset second threshold, an alarm signal is generated and the letter is marked as pending review, and it is automatically pushed to the manual review queue. It should be noted that the preset second threshold is the degree of matching between the text content and the seal features. The threshold value; if it falls below this value, an alarm is triggered and a high-risk flag is displayed. And highlight the mismatched text-stamp area; The system retrieves matching data from historical similar emails and calculates the distance between the current matching degree and historical data using the K-nearest neighbor algorithm. If the distance exceeds the preset reference standard deviation, it is identified as an outlier and the alarm level is upgraded to high risk. It should be noted that the matching data of historical similar letters refers to the matching results of past letter texts and seal features stored in the system, including matching score, alignment of key fields, and corresponding risk level.

[0036] In this embodiment, the method of a letter text and seal analysis system based on multimodal dynamic association extracts key information of the letter text through semantic analysis and obtains the visual features of the seal. The visual features of the seal are obtained by detecting, extracting and identifying the authenticity of the letter seal. The rationality of current seal usage behavior is analyzed by combining historical seal usage data with a spatiotemporal compliance verification formula. Establish a dynamic association model between text and seal, and output the matching degree between text content and seal features; Based on the degree of matching between the text content and the seal features, the overall risk score of the letter is output.

[0037] The above formulas are all dimensionless calculations. The formulas are derived from software simulations based on a large amount of collected data to obtain the most recent real-world results. The preset parameters in the formulas are set by those skilled in the art according to the actual situation.

[0038] The above embodiments can be implemented, in whole or in part, by software, hardware, firmware, or any other combination thereof. When implemented using software, the above embodiments can be implemented, in whole or in part, in the form of a computer program product.

[0039] Those skilled in the art will recognize that the modules and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0040] In addition, the functional modules in the various embodiments of this application can be integrated into one processing module, or each module can exist physically separately, or two or more modules can be integrated into one module.

[0041] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0042] In conclusion, the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A system for analyzing letter text and seals based on multimodal dynamic association, characterized in that, include: The multimodal data fusion module is used to extract key information from the letter text through semantic analysis and obtain the visual features of the seal. The visual features of the seal are obtained by detecting, extracting and identifying the authenticity of the seal in the letter. The spatiotemporal correlation analysis module is used to combine historical seal usage data and analyze the rationality of current seal usage behavior based on spatiotemporal compliance verification formulas. The semantic-visual cross-validation module is used to establish a dynamic association model between text and seal, and output the matching degree between text content and seal features. The dynamic risk scoring module is used to output a comprehensive risk score for a letter based on the matching degree between the text content and the seal features.

2. The letter text and seal analysis system based on multimodal dynamic association according to claim 1, characterized in that, The multimodal data fusion module includes a key information extraction step for letter text, specifically: The BERT model encodes the email text and outputs semantic feature vectors; Key fields in the email were extracted using named entity recognition technology. The extracted key fields are matched and validated against the preset template. If a field is missing or the format is incorrect, it is marked as incomplete information.

3. The letter text and seal analysis system based on multimodal dynamic association according to claim 2, characterized in that, The multimodal data fusion module includes acquiring the visual features of the seal, specifically: The ResNet model is used to perform convolution processing on the seal image to output a visual feature vector.

4. The letter text and seal analysis system based on multimodal dynamic association according to claim 3, characterized in that, The spatiotemporal compliance check uses the following calculation formula: in For spatiotemporal anomaly probability, This is the current stamping time. This represents the historical average printing time. Current GPS coordinates This is a commonly used historical location. The radius of the geofence. and This is the time decay coefficient.

5. The letter text and seal analysis system based on multimodal dynamic association according to claim 4, characterized in that, The semantic visual cross-validation module includes: By aligning key information in the letter text with features in the seal area through a cross-modal attention mechanism, attention weights are assigned to key entities in the text and corresponding areas in the seal, and semantic-visual matching degree is calculated. The formula for calculating the semantic-visual matching degree is as follows: in, The degree of matching between the text content and the seal features. For text feature vectors, For the seal feature vector, For attention weights, Number of feature pairs for alignment.

6. The letter text and seal analysis system based on multimodal dynamic association according to claim 5, characterized in that, The comprehensive risk score of the output function adopts a multi-factor fusion algorithm, and the specific formula is as follows: in, , , These are the first set of weighting coefficients. To determine the probability of seal forgery, is the steepness coefficient of the Sigmoid function, used to map the forgery probability to a non-linear risk contribution.

7. The letter text and seal analysis system based on multimodal dynamic association according to claim 6, characterized in that, The probability of seal forgery The specific calculation formula is as follows: in, The features of the seal to be tested, To verify the characteristics of a genuine seal, To preset the first threshold, It is a structural similarity index. , For the true seal of history The distribution parameters.

8. The letter text and seal analysis system based on multimodal dynamic association according to claim 7, characterized in that, The key information extraction step of the letter text also includes: if the matching verification fails, triggering the weight adjustment mechanism of the dynamic risk scoring module, specifically: The weight coefficients of the multi-factor fusion algorithm are increased from the first group of weight coefficients to the second group of weight coefficients; Simultaneously reduce the spatiotemporal anomaly probability weight and the forgery probability weight to the first preset lower limit and the second preset lower limit.

9. The letter text and seal analysis system based on multimodal dynamic association according to claim 8, characterized in that, The semantic visual cross-validation module also includes an abnormal matching alarm function, specifically: When the text content matches the seal features When the value is below the preset second threshold, an alarm signal is generated and the letter is marked as pending review, and it is automatically pushed to the manual review queue. And highlight the mismatched text-stamp area; The system retrieves matching data from historical similar emails and calculates the distance between the current matching degree and historical data using the K-nearest neighbor algorithm. If the distance exceeds the preset reference standard deviation, it is identified as an outlier and the alarm level is upgraded to high risk.

10. A method for analyzing letter text and seals using the multimodal dynamic association-based system as described in any one of claims 1-9, characterized in that, Includes the following steps: Key information in the letter text is extracted through semantic analysis, and visual features of the seal are obtained. These visual features are obtained by detecting, extracting, and identifying the authenticity of the seal in the letter. The rationality of current seal usage behavior is analyzed by combining historical seal usage data with a spatiotemporal compliance verification formula. Establish a dynamic association model between text and seal, and output the matching degree between text content and seal features; Based on the degree of matching between the text content and the seal features, the overall risk score of the letter is output.

Images